Multi-literature mixed character recognition method and system based on deep learning

Abstract

The invention discloses a multi-literature mixed character recognition method and system based on deep learning. The method comprises the steps of text image collection, primary data processing, text detection model construction, text recognition model construction and multi-literature mixed character recognition. According to the invention, original image data is obtained through text image acquisition; a data primary processing method of color normalization, random visual confusion, geometric space deformation, multi-scale image generation and data set segmentation is adopted; a multi-granularity deep learning model is adopted as a text detection model, features of different granularities are captured through an adaptive receptive field mechanism, and sub-tasks are mutually enhanced by utilizing the synergistic effect between the tasks; a two-way context network model is adopted as a text recognition model, feature expressions for different texts are fused through a dynamic routing mechanism, two-way time sequence modeling and visual semantic alignment are combined, context dependence of character sequences is considered, and the method is anchored to visual evidences.

Description

Technical Field

[0001] This invention relates to the field of intelligent hybrid character recognition technology, specifically to a multilingual hybrid character recognition method and system based on deep learning. Background Technology

[0002] Multilingual text recognition is a technology that combines computer vision and natural language processing to identify and accurately process mixed text that spans multiple languages ​​and character sets. This technology can handle complex text scenarios that simultaneously contain different languages ​​and writing systems, significantly improving the efficiency of cross-language and cross-cultural text processing and promoting seamless conversion and retrieval of multilingual information.

[0003] However, traditional multilingual mixed text recognition methods rely on feature extraction networks with a single scale or fixed structure, making it difficult to flexibly adapt to the unique structures and varied appearances of different writing systems. Furthermore, treating text classification and text detection as isolated tasks lacks effective cross-task information interaction, leading to technical problems such as missed detections and false detections when dealing with mixed texts with large scale spans, complex layouts, or geometric deformations. Traditional multilingual mixed text recognition methods also ignore the fundamental differences between different writing systems in character sets, writing rules, and visual patterns, resulting in weak feature extraction targeting. Moreover, when utilizing contextual information, it is often difficult to balance the relationship between visual features and language models, easily leading to errors due to over-reliance on semantic context or failure to correct visual ambiguities by ignoring context. Summary of the Invention

[0004] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a multi-language mixed text recognition method and system based on deep learning. Traditional multi-language mixed text recognition methods rely on feature extraction networks with single scales or fixed structures, making it difficult to flexibly adapt to the unique structures and varied appearances of different writing systems. Furthermore, treating text classification and text detection as isolated tasks lacks effective cross-task information interaction, leading to frequent false positives and false negatives when dealing with mixed texts with large scale spans, complex layouts, or geometric deformations. This solution creatively employs a multi-granularity deep learning model as the text detection model. Through an adaptive receptive field mechanism, it dynamically captures features of different granularities, from local strokes to global layout. It also utilizes inter-task collaborative attention to integrate text localization, text classification, and geometric regression. The sub-tasks reinforce each other, improving the overall perception and understanding of mixed text in the detection phase. Traditional multi-language mixed text recognition methods often neglect fundamental differences in character sets, writing rules, and visual patterns among different writing systems, resulting in weak feature extraction targeting. Furthermore, when utilizing contextual information, they often struggle to balance the relationship between visual features and language models, easily leading to errors due to over-reliance on semantic context or failing to correct visual ambiguities by ignoring context. This solution creatively employs a bidirectional contextual network model as the text recognition model. Through a dynamic routing mechanism, it integrates feature expressions for different scripts, achieving multi-script differentiation at the feature level. Combining bidirectional temporal modeling with visual semantic alignment, it fully considers the contextual dependencies of character sequences while strictly anchoring to visual evidence.

[0005] The technical solution adopted by this invention is as follows: The multilingual character recognition method based on deep learning provided by this invention includes the following steps:

[0006] Step S1: Text image acquisition;

[0007] Step S2: Preliminary data processing;

[0008] Step S3: Text detection model construction;

[0009] Step S4: Text recognition model construction;

[0010] Step S5: Multilingual mixed character recognition.

[0011] Further, in step S1, the text image acquisition is used to acquire the original image data required for multi-language mixed character recognition. Specifically, it is to obtain the original image dataset for character recognition through image acquisition. The original image dataset for character recognition specifically includes a historical original image dataset and a current original image dataset. Both the historical original image dataset and the current original image dataset contain text image data. The historical original image dataset also contains bounding box annotation data, text content annotation data, and text language annotation data.

[0012] Further, in step S2, the preliminary data processing is used to perform preliminary processing on the acquired raw image data, specifically including the following steps:

[0013] Step S21: Color normalization, used to eliminate the influence of illumination and color differences on recognition. Specifically, it involves adaptively learning the color distribution of the image through a shallow CNN network and dynamically adjusting the contrast and brightness to obtain a color-normalized image.

[0014] Step S22: Random visual obfuscation, used to simulate image degradation in real-world scenes, specifically by applying random block occlusion, Gaussian noise, and blurring operations to obtain an enhanced image;

[0015] Step S23: Geometric spatial deformation, used to improve the model's adaptability to deformed text. Specifically, by applying random elastic deformation and smooth geometric transformation, text bending and perspective changes are simulated to obtain the deformed image.

[0016] Step S24: Multi-scale image generation, used to adapt the model to text of different sizes, specifically by generating image versions at multiple scales to ensure that the model can handle text regions of various sizes, thus obtaining multi-scale images;

[0017] Step S25: Dataset segmentation, used to obtain training data and test data, specifically, to segment the original image dataset of the recognition history;

[0018] The current original image dataset is pre-processed using color normalization and multi-scale image generation to obtain an image dataset to be identified. The historical original image dataset is pre-processed using color normalization, random visual confusion, geometric spatial deformation, multi-scale image generation, and dataset segmentation to obtain a detection training set and a detection test set.

[0019] Further, in step S3, the text detection model construction is used to construct the model required for detecting text regions in an image and classifying the text regions by language. Specifically, it is achieved by constructing a multi-granularity deep learning model as the text detection model. The multi-granularity deep learning model outputs the text detection result through multi-granularity feature interaction and the design of a collaborative attention unit.

[0020] The construction of the text detection model specifically includes the following steps:

[0021] Step S31: Backbone feature extraction, used to extract multi-level features from the input image, includes the following steps:

[0022] Step S311: Convolutional feature extraction, used to capture local texture features, specifically by progressively downsampling the image through stacked 3×3 convolutions and batch normalization operations to obtain multi-scale preliminary features;

[0023] Step S312: Feature pyramid construction, used to fuse multi-scale information, specifically by combining preliminary features of different scales through upsampling and lateral connection to obtain multi-scale pyramid features;

[0024] Step S32: Multi-granularity feature interaction, used to dynamically optimize the feature receptive field and interact with multi-granularity features, includes the following steps:

[0025] Step S321: Design of multi-dilution rate convolution groups to capture contextual information of different ranges. Specifically, multi-scale pyramid features are processed by dilated convolutions with dilation rates of 2, 4 and 8 to obtain feature maps with different receptive fields at the same scale.

[0026] Step S322: Gating weight generation, used to dynamically select the best receptive field. Specifically, the weight of each spatial location is calculated through the sub-network to obtain the gating weights of different receptive fields at the same scale.

[0027] Step S323: Feature fusion, used for adaptive fusion of multi-granularity features, specifically, based on the gating weights of different receptive fields at the same scale, the features of different receptive fields at the same scale are weighted and summed to obtain multi-scale optimized pyramid features;

[0028] Step S33: Multi-task output design for collaborative text detection, language classification, and bounding box regression. The steps include:

[0029] Step S331: Multi-task-specific feature generation, used to provide customized features for different tasks. Specifically, multi-scale optimized pyramid features are processed through independent 3×3 convolutional layers to obtain multi-scale task-specific features.

[0030] Step S332: Collaborative attention unit design to enhance information flow between tasks. Specifically, firstly, text detection features at the same scale are used as queries, and language classification features are used as keys and values. A cross-attention mechanism is used to process these features to obtain multi-scale text-language attention features. Then, text detection features at the same scale are used as queries, and bounding box features are used as keys and values. A cross-attention mechanism is used to process these features to obtain multi-scale text-boundary attention features.

[0031] Then, text-text type attention features and text-boundary attention features at the same scale are concatenated, processed by a linear layer, and then residually connected with text detection features at the same scale to obtain multi-scale text detection collaborative features.

[0032] Finally, the multi-scale text classification features and multi-scale bounding box features are processed in the same way to obtain multi-scale text classification collaborative features and multi-scale bounding box collaborative features.

[0033] Step S333: Single-scale output acquisition, used to generate the prediction output of the single-scale model, specifically by processing the collaborative features of each task through a 1×1 convolutional layer and activation function to obtain the text probability map, language classification map and bounding box map;

[0034] Step S334: Multi-scale output fusion, used to fuse the model prediction output results of multiple scales. Specifically, for the multi-scale text probability map, multi-scale language classification map and multi-scale bounding box map, the prediction maps of all scales are upsampled to the size of the model input image and merged to obtain the final text probability map, the final language classification map and the final bounding box map.

[0035] Step S34: Construct and train the model. Specifically, the model is constructed by integrating the backbone feature extraction, multi-granularity feature interaction and multi-task output design. The model is trained and its performance is verified based on the detection training set and the detection test set to obtain the multi-granularity deep learning model as a text detection model.

[0036] Step S35: Dataset acquisition, used to obtain training and test data for the subsequent text recognition model. Specifically, the detection training set and the detection test set are used as inputs to the text detection model, and after processing, the recognition training set and the recognition test set are obtained.

[0037] Further, in step S4, the text recognition model construction is used to construct the model required for text recognition based on the text detection results. Specifically, it is to construct a bidirectional context network model as the text recognition model. The bidirectional context network model obtains the text recognition result by combining dynamic routing coding and bidirectional temporal context mechanism.

[0038] The text recognition model is constructed by the following steps:

[0039] Step S41: Dynamic routing encoding, used to adaptively extract features from texts of different languages, includes the following steps:

[0040] Step S411: Route weight calculation, used to obtain primary recognition features and determine the weight of expert paths. Specifically, it involves processing the text region image obtained by cropping the final text probability map through a convolutional neural network to obtain primary recognition features, and calculating the weight of each expert path through an independent multilayer perceptron to obtain the expert path route weight.

[0041] Step S412: Expert feature extraction and fusion, used to provide multi-language feature representation and fuse multi-expert path features. Specifically, multiple independent multilayer perceptrons are used as expert paths. Each expert path extracts a feature representation of a language based on the primary recognition features and fuses them based on the expert path routing weights to obtain path fusion features.

[0042] Step S413: Sequence feature generation, used to convert feature maps into sequences, specifically by converting path fusion features into time series through a reshaping operation to obtain encoded temporal features;

[0043] Step S42: Design of bidirectional temporal context mechanism to capture the contextual dependencies of text sequences and dynamically integrate bidirectional information. Specifically, a bidirectional long short-term memory network is used to capture the contextual features of the time series from left to right and from right to left, respectively, to obtain the forward hidden state and the backward hidden state, and then they are fused through a gating mechanism to obtain enhanced contextual features.

[0044] Step S43: Visual semantic alignment, used to prevent the decoding process from deviating from visual evidence, includes the following steps:

[0045] Step S431: Global visual feature extraction, used to preserve the visual information of the input image. Specifically, the final text probability map, the final language classification map and the final bounding box map are concatenated in the channel dimension, and a separate convolutional neural network is used for feature extraction to obtain fused visual features.

[0046] Step S432: Gated alignment fusion, used to balance and fuse enhanced contextual features and fused visual features. Specifically, by concatenating enhanced contextual features and fused visual features, linearly mapping them, the sigmoid activation function is used to calculate the gating weights, and the enhanced contextual features and the linearly mapped fused visual features are weighted and fused to obtain visual semantic fusion features.

[0047] Step S44: Text recognition output acquisition, used to obtain text recognition results. Specifically, the softmax activation function is used to process the visual semantic fusion features after linear mapping. The obtained results are then decoded by CTC to obtain the text recognition results.

[0048] Step S45: Model construction and training, specifically, through the dynamic routing encoding, the bidirectional temporal context mechanism design, the visual semantic alignment and the text recognition output acquisition, a bidirectional context network model is constructed. The model is trained and its performance is verified based on the recognition training set and the recognition test set to obtain the bidirectional context network model as the text recognition model.

[0049] Further, in step S5, the multi-language mixed text recognition specifically involves first using the image dataset to be recognized as the input of the text detection model, then using the obtained text detection result as the input of the text recognition model, outputting the text recognition result, and combining the text recognition result with the text detection result to obtain the final text recognition result. The text detection result specifically includes a final text probability map, a final language classification map, and a final boundary box map.

[0050] The multi-language hybrid character recognition system based on deep learning provided by this invention includes a text image acquisition module, a preliminary data processing module, a text detection model construction module, a text recognition model construction module, and a multi-language hybrid character recognition module.

[0051] The text image acquisition module is used to acquire raw data, obtain a raw image dataset for text recognition, and send the raw image dataset for text recognition to the preliminary data processing module.

[0052] The data preliminary processing module is used for preliminary data processing. Through preliminary data processing, a dataset of images to be identified, a detection training set, and a detection test set are obtained. The dataset of images to be identified is sent to the multilingual mixed text recognition module, and the detection training set and the detection test set are sent to the text detection model construction module.

[0053] The text detection model construction module is used to construct a text detection model. By constructing a multi-granularity deep learning model as a text detection model, it processes the detection training set and the detection test set to obtain the recognition training set and the recognition test set. The text detection model is then sent to the multi-language mixed text recognition module, and the recognition training set and the recognition test set are sent to the text recognition model construction module.

[0054] The text recognition model construction module is used to construct a text recognition model. It constructs a bidirectional context network model as a text recognition model and sends the text recognition model to the multi-language mixed text recognition module.

[0055] The multilingual mixed text recognition module is used for multilingual mixed text recognition. It obtains the final text recognition result by processing the current data using the text detection model and the text recognition model.

[0056] The beneficial effects achieved by the present invention using the above solution are as follows:

[0057] (1) Traditional multi-language mixed text recognition methods rely on feature extraction networks with a single scale or fixed structure, which are difficult to adapt flexibly to the unique structure and varied appearance of different writing systems. Furthermore, treating text classification and text detection as isolated tasks lacks effective cross-task information interaction, which leads to technical problems such as missed detection and false detection when facing mixed texts with large scale spans, complex layouts, or geometric deformations. This solution creatively adopts a multi-granularity deep learning model as the text detection model. Through an adaptive receptive field mechanism, it dynamically captures different granularity features from local strokes to global layout. By utilizing collaborative attention between tasks, the three sub-tasks of text localization, text classification, and geometric regression are mutually enhanced, thereby improving the overall perception and understanding of mixed texts in the detection stage.

[0058] (2) In view of the technical problems of traditional multi-language mixed text recognition methods, which ignore the fundamental differences between different writing systems in character sets, writing rules and visual patterns, resulting in weak feature extraction and difficulty in balancing the relationship between visual features and language models when using context information, it is easy to make errors due to over-reliance on semantic context or failure to correct visual ambiguity due to ignoring context. This solution creatively adopts a bidirectional context network model as the text recognition model. Through dynamic routing mechanism, it integrates feature expressions for different languages ​​to achieve the distinction of multiple languages ​​at the feature level. It also combines bidirectional temporal modeling and visual semantic alignment to fully consider the contextual dependence of character sequences and strictly anchor to visual evidence. Attached Figure Description

[0059] Figure 1 A flowchart illustrating the multilingual character recognition method based on deep learning provided by this invention;

[0060] Figure 2 A schematic diagram of the modules of the deep learning-based multilingual character recognition system provided by the present invention;

[0061] Figure 3 This is a flowchart illustrating the preliminary data processing in step S2.

[0062] Figure 4 A flowchart illustrating the process of building the text detection model in step S3;

[0063] Figure 5 A flowchart illustrating the process of building the text recognition model in step S4.

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0065] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0066] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0067] Example 1, see Figure 1 The present invention provides a multilingual character recognition method based on deep learning, which includes the following steps:

[0068] Step S1: Text image acquisition;

[0069] Step S2: Preliminary data processing;

[0070] Step S3: Text detection model construction;

[0071] Step S4: Text recognition model construction;

[0072] Step S5: Multilingual mixed character recognition.

[0073] Example 2, see Figure 1 and Figure 2This embodiment is based on the above embodiment. In step S1, the text image acquisition is used to acquire the original image data required for multi-language mixed text recognition. Specifically, it is to obtain the original image dataset for text recognition through image acquisition. The original image dataset for text recognition specifically includes the recognition history original image dataset and the recognition current original image dataset. Both the recognition history original image dataset and the recognition current original image dataset contain text image data. The recognition history original image dataset also contains bounding box annotation data, text content annotation data, and text language annotation data.

[0074] The text image data specifically refers to multilingual text image data collected from the ICDAR series datasets and the MLT multilingual text dataset.

[0075] Example 3, see Figure 1 , Figure 2 and Figure 3 This embodiment is based on the above embodiment. In step S2, the preliminary data processing is used to perform preliminary processing on the acquired raw image data, specifically including the following steps:

[0076] Step S21: Color normalization, used to eliminate the influence of illumination and color differences on recognition. Specifically, it involves adaptively learning the color distribution of the image through a shallow CNN network and dynamically adjusting the contrast and brightness to obtain a color-normalized image.

[0077] Step S22: Random visual obfuscation, used to simulate image degradation in real-world scenes, specifically by applying random block occlusion, Gaussian noise, and blurring operations to obtain an enhanced image;

[0078] Step S23: Geometric spatial deformation, used to improve the model's adaptability to deformed text. Specifically, by applying random elastic deformation and smooth geometric transformation, text bending and perspective changes are simulated to obtain the deformed image.

[0079] Step S24: Multi-scale image generation, used to adapt the model to text of different sizes, specifically by generating image versions at multiple scales to ensure that the model can handle text regions of various sizes, thus obtaining multi-scale images;

[0080] Step S25: Dataset segmentation, used to obtain training data and test data, specifically, to segment the original image dataset of the recognition history;

[0081] The current original image dataset is pre-processed using color normalization and multi-scale image generation to obtain an image dataset to be identified. The historical original image dataset is pre-processed using color normalization, random visual confusion, geometric spatial deformation, multi-scale image generation, and dataset segmentation to obtain a detection training set and a detection test set.

[0082] Example 4, see Figure 1 , Figure 2 and Figure 4 This embodiment is based on the above embodiment. In step S3, the text detection model is constructed to build a model required for detecting text regions in an image and classifying text regions by language. Specifically, a multi-granularity deep learning model is constructed as the text detection model. The multi-granularity deep learning model outputs the text detection result through multi-granularity feature interaction and the design of a collaborative attention unit.

[0083] The construction of the text detection model specifically includes the following steps:

[0084] Step S31: Backbone feature extraction, used to extract multi-level features from the input image, includes the following steps:

[0085] Step S311: Convolutional feature extraction, used to capture local texture features, specifically involves progressively downsampling the image through stacked 3×3 convolutions and batch normalization to obtain preliminary multi-scale features. The formula used is as follows:

[0086] ;

[0087] In the formula, This indicates the initial characteristics of the second scale. This represents a 3×3 convolution function. Represents the ReLU activation function. This represents the batch normalization function. This represents the input image to the model. This represents the initial features at the (a+1)th scale. Represents the downsampling function. Represents the residual block function. This represents the preliminary features at the a-th scale, where 'a' represents the scale index.

[0088] Step S312: Feature pyramid construction, used to fuse multi-scale information. Specifically, it combines preliminary features of different scales through upsampling and lateral connections to obtain multi-scale pyramid features. The formula used is as follows:

[0089] ;

[0090] In the formula, This represents the pyramid features at the fifth scale. This represents a 1×1 convolution function. This indicates the initial characteristics of the fifth scale. Let a represent the pyramid features at the a-th scale. Represents the upsampling function. This represents the pyramid feature at the (a+1)th scale;

[0091] Step S32: Multi-granularity feature interaction, used to dynamically optimize the feature receptive field and interact with multi-granularity features, includes the following steps:

[0092] Step S321: Design of multi-dilution rate convolutional groups to capture contextual information at different ranges. Specifically, multi-scale pyramid features are processed using dilated convolutions with dilation rates of 2, 4, and 8 to obtain feature maps with different receptive fields at the same scale. The formula used is as follows:

[0093] ;

[0094] In the formula, This represents the dilated convolution output feature with a dilation rate of c at the b-th scale. This represents a dilated convolution function with a dilation rate of c. Let b represent the pyramid feature at the b-th scale, where b represents the scale index and c represents the void ratio index.

[0095] Step S322: Gating weight generation, used to dynamically select the optimal receptive field. Specifically, the weight of each spatial location is calculated through the sub-network to obtain the gating weights of different receptive fields at the same scale. The formula used is as follows:

[0096] ;

[0097] In the formula, This represents the comprehensive gated weighted feature map at the b-th scale, with 3 channels corresponding to three different hole rates. This represents the global pooling function. This indicates a broadcast addition operation. This represents the gating weight for the porosity c at the b-th scale. This represents the sigmoid activation function. This represents the c-th channel of the integrated gated weight feature map at the b-th scale;

[0098] Step S323: Feature fusion, used for adaptive fusion of multi-granularity features. Specifically, based on the gating weights of different receptive fields at the same scale, the features of different receptive fields at the same scale are weighted and summed to obtain multi-scale optimized pyramid features. The formula used is as follows:

[0099] ;

[0100] In the formula, This represents the optimized pyramid feature at the b-th scale. This represents element-wise multiplication.

[0101] Step S33: Multi-task output design for collaborative text detection, language classification, and bounding box regression. The steps include:

[0102] Step S331: Multi-task-specific feature generation, used to provide customized features for different tasks. Specifically, multi-scale optimized pyramid features are processed through independent 3×3 convolutional layers to obtain multi-scale task-specific features. The formula used is as follows:

[0103] ;

[0104] In the formula, Let b represent the text detection features at the b-th scale. This represents the text classification feature at the b-th scale. This represents the bounding box feature at the b-th scale. This represents a 3×3 convolution function used to obtain text detection features. This represents a 3×3 convolution function used to obtain language classification features. This represents a 3×3 convolution function used to obtain bounding box features;

[0105] Step S332: Collaborative attention unit design to enhance information flow between tasks. Specifically, firstly, text detection features at the same scale are used as queries, and language classification features are used as keys and values. A cross-attention mechanism is used to process these features to obtain multi-scale text-language attention features. Then, text detection features at the same scale are used as queries, and bounding box features are used as keys and values. A cross-attention mechanism is used to process these features to obtain multi-scale text-boundary attention features.

[0106] Then, text-text type attention features and text-boundary attention features at the same scale are concatenated, processed by a linear layer, and then residually connected with text detection features at the same scale to obtain multi-scale text detection collaborative features.

[0107] Finally, the multi-scale text classification features and multi-scale bounding box features are processed in the same way to obtain multi-scale text classification collaborative features and multi-scale bounding box collaborative features.

[0108] Step S333: Single-scale output acquisition, used to generate the single-scale model prediction output, specifically by processing the collaborative features of each task through a 1×1 convolutional layer and activation function to obtain the text probability map, language classification map, and bounding box map. The formula used is as follows:

[0109] ;

[0110] In the formula, This represents the text probability graph at the b-th scale. This represents the language classification diagram at the b-th scale. This represents the boundary box diagram at the b-th scale. Let b represent the text detection collaborative features at the b-th scale. This represents the text classification synergy feature at the b-th scale. Represents the collaborative features of the bounding box at the b-th scale. This represents the softmax activation function;

[0111] Step S334: Multi-scale output fusion, used to fuse the model prediction output results of multiple scales. Specifically, for the multi-scale text probability map, multi-scale language classification map and multi-scale bounding box map, the prediction maps of all scales are upsampled to the size of the model input image and merged to obtain the final text probability map, the final language classification map and the final bounding box map.

[0112] Step S34: Construct and train the model. Specifically, the model is constructed by integrating the backbone feature extraction, multi-granularity feature interaction and multi-task output design. The model is trained and its performance is verified based on the detection training set and the detection test set to obtain the multi-granularity deep learning model as a text detection model.

[0113] Step S35: Dataset acquisition, used to obtain training and test data for the subsequent text recognition model. Specifically, the detection training set and the detection test set are used as inputs to the text detection model, and after processing, the recognition training set and the recognition test set are obtained.

[0114] By performing the above operations, this solution addresses the technical problems of traditional multi-language mixed text recognition methods, which rely on feature extraction networks with a single scale or fixed structure, making it difficult to flexibly adapt to the unique structures and varied appearances of different writing systems. Furthermore, treating text classification and text detection as isolated tasks lacks effective cross-task information interaction, leading to frequent missed and false detections when dealing with mixed texts with large scale spans, complex layouts, or geometric deformations. This solution creatively adopts a multi-granularity deep learning model as the text detection model. Through an adaptive receptive field mechanism, it dynamically captures features of different granularities, from local strokes to global layout. By utilizing collaborative attention between tasks, the three sub-tasks of text localization, text classification, and geometric regression mutually enhance each other, improving the overall perception and understanding of mixed texts in the detection stage.

[0115] Example 5, see Figure 1 , Figure 2 and Figure 5 This embodiment is based on the above embodiment. In step S4, the text recognition model is constructed to build a model required for text recognition based on text detection results. Specifically, a bidirectional context network model is constructed as the text recognition model. The bidirectional context network model obtains the text recognition result by combining dynamic routing coding and bidirectional temporal context mechanism.

[0116] The text recognition model is constructed by the following steps:

[0117] Step S41: Dynamic routing encoding, used to adaptively extract features from texts of different languages, includes the following steps:

[0118] Step S411: Route weight calculation, used to obtain primary recognition features and determine the weights of expert paths. Specifically, it involves processing the text region image obtained by cropping the final text probability map through a convolutional neural network to obtain primary recognition features, and then calculating the weight of each expert path through an independent multilayer perceptron to obtain the expert path route weights. The formula used is as follows:

[0119] ;

[0120] In the formula, Indicates primary identification features, This represents the convolutional neural network function used to obtain primary recognition features. This represents the text region image obtained by cropping based on the final text probability map, and Rw represents the expert path routing weight. This represents the multilayer perceptron function used to obtain the expert path routing weights;

[0121] Step S412: Expert feature extraction and fusion, used to provide multi-language feature representations and fuse multi-expert path features. Specifically, multiple independent multilayer perceptrons are used as expert paths. Each expert path extracts feature representations for one language based on primary recognition features, and the features are weighted and fused based on the expert path routing weights to obtain path fusion features. The formula used is as follows:

[0122] ;

[0123] In the formula, This represents the output feature of the k-th expert path. Let the multilayer perceptron function of the k-th expert path be represented. This represents the path fusion feature, where K represents the number of document types. This represents the expert path routing weight of the k-th expert path;

[0124] Step S413: Sequence feature generation, used to convert feature maps into sequences, specifically by converting path fusion features into time series through a reshaping operation to obtain encoded temporal features;

[0125] Step S42: Design of a bidirectional temporal context mechanism to capture the contextual dependencies of text sequences and dynamically integrate bidirectional information. Specifically, a bidirectional long short-term memory network is used to capture the contextual features of the time series from left to right and from right to left, respectively, to obtain the forward hidden state and the backward hidden state. These are then fused through a gating mechanism to obtain enhanced contextual features. The formula used is as follows:

[0126] ;

[0127] In the formula, This represents the forward hidden state at time step t. This represents the backward hidden state at time step t. This represents the forward long short-term memory network function. This represents the backward long short-term memory network function. This represents the encoded temporal characteristics at time step t. This represents the forward hidden state at time step t-1. This represents the backward hidden state at time step t-1. This represents the bidirectional temporal context gating weights at time step t. This represents the bidirectional temporal context gating weight mapping matrix. This represents the bias term of the bidirectional temporal context gating weight mapping. This represents the enhanced contextual features at time step t. This represents the multilayer perceptron function used to obtain enhanced contextual features;

[0128] Step S43: Visual semantic alignment, used to prevent the decoding process from deviating from visual evidence, includes the following steps:

[0129] Step S431: Global visual feature extraction, used to preserve the visual information of the input image. Specifically, the final text probability map, the final language classification map and the final bounding box map are concatenated in the channel dimension, and a separate convolutional neural network is used for feature extraction to obtain fused visual features.

[0130] Step S432: Gated alignment fusion, used to balance and fuse enhanced contextual features and fused visual features. Specifically, by concatenating enhanced contextual features and fused visual features, linearly mapping them, the sigmoid activation function is used to calculate the gating weights, and the enhanced contextual features and the linearly mapped fused visual features are weighted and fused to obtain visual semantic fusion features.

[0131] Step S44: Text recognition output acquisition, used to obtain text recognition results. Specifically, the softmax activation function is used to process the visual semantic fusion features after linear mapping. The obtained results are then decoded by CTC to obtain the text recognition results.

[0132] Step S45: Model construction and training, specifically, through the dynamic routing encoding, the bidirectional temporal context mechanism design, the visual semantic alignment and the text recognition output acquisition, a bidirectional context network model is constructed. The model is trained and its performance is verified based on the recognition training set and the recognition test set to obtain the bidirectional context network model as the text recognition model.

[0133] By performing the above operations, this solution addresses the technical problems of traditional multi-language mixed text recognition methods, which ignore the fundamental differences between different writing systems in character sets, writing rules, and visual patterns, resulting in weak feature extraction and difficulty in balancing the relationship between visual features and language models when utilizing contextual information. These problems often lead to errors due to over-reliance on semantic context or failure to correct visual ambiguities by ignoring context. This solution creatively adopts a bidirectional context network model as the text recognition model. Through a dynamic routing mechanism, it integrates feature expressions for different writing systems to achieve differentiation of multiple writing systems at the feature level. Furthermore, by combining bidirectional temporal modeling with visual semantic alignment, it fully considers the contextual dependence of character sequences while strictly anchoring to visual evidence.

[0134] Example 6, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In step S5, the multi-language mixed text recognition specifically involves first using the image dataset to be recognized as the input of the text detection model, then using the obtained text detection result as the input of the text recognition model, outputting the text recognition result, and combining the text recognition result with the text detection result to obtain the final text recognition result. The text detection result specifically includes the final text probability map, the final language classification map, and the final boundary box map.

[0135] Example 7, see Figure 1 and Figure 2 Based on the above embodiments, the multi-language mixed character recognition system based on deep learning provided by the present invention includes a text image acquisition module, a preliminary data processing module, a text detection model construction module, a text recognition model construction module, and a multi-language mixed character recognition module;

[0136] The text image acquisition module is used to acquire raw data, obtain a raw image dataset for text recognition, and send the raw image dataset for text recognition to the preliminary data processing module.

[0137] The data preliminary processing module is used for preliminary data processing. Through preliminary data processing, a dataset of images to be identified, a detection training set, and a detection test set are obtained. The dataset of images to be identified is sent to the multilingual mixed text recognition module, and the detection training set and the detection test set are sent to the text detection model construction module.

[0138] The text detection model construction module is used to construct a text detection model. By constructing a multi-granularity deep learning model as a text detection model, it processes the detection training set and the detection test set to obtain the recognition training set and the recognition test set. The text detection model is then sent to the multi-language mixed text recognition module, and the recognition training set and the recognition test set are sent to the text recognition model construction module.

[0139] The text recognition model construction module is used to construct a text recognition model. It constructs a bidirectional context network model as a text recognition model and sends the text recognition model to the multi-language mixed text recognition module.

[0140] The multilingual mixed text recognition module is used for multilingual mixed text recognition. It obtains the final text recognition result by processing the current data using the text detection model and the text recognition model.

[0141] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0142] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

[0143] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for recognizing multi-lingual mixed script based on deep learning, characterized in that: The method comprises the following steps: S1: text image acquisition, through image acquisition, obtain the original image data set of character recognition, the original image data set of character recognition specifically includes the original image data set of recognition history and the original image data set of recognition current; S2: data preliminary processing, the original image data collected is preliminarily processed, and the image data set to be recognized, the detection training set and the detection test set are obtained; S3: text detection model construction, for constructing the model required for image text region detection and text region classification, specifically, a multi-granularity deep learning model is constructed as a text detection model, and the detection training set and the detection test set are processed to obtain a recognition training set and a recognition test set, the multi-granularity deep learning model is output by multi-granularity feature interaction and design of collaborative attention unit, and text detection result is obtained; S4: text recognition model construction, for constructing the model required for text recognition based on text detection result, specifically, a bidirectional context network model is constructed as a text recognition model, the bidirectional context network model is obtained by combining dynamic routing encoding and bidirectional time sequence context mechanism, and text recognition result is obtained; S5: multi-language mixed character recognition, specifically, the image data set to be recognized is first input into the text detection model to obtain text detection result, the text detection result is input into the text recognition model to output text recognition result, the final text recognition result is obtained by combining the text recognition result and the text detection result, and the text detection result specifically includes final text probability graph, final script classification graph and final bounding box graph. 2.The deep learning-based multilingual mixed script recognition method of claim 1, wherein: The text detection model construction specifically comprises the following steps: Step S31: main feature extraction, for extracting multi-level features from input images, the steps comprising: Step S311: convolution feature extraction, for capturing local texture features, specifically, by stacking 3x3 convolution and batch normalization operation, the image is gradually down-sampled to obtain multi-scale preliminary features; Step S312: feature pyramid construction, for fusing multi-scale information, specifically, by up-sampling and horizontal connection, different scale preliminary features are combined to obtain multi-scale pyramid features; Step S32: multi-granularity feature interaction, for dynamically optimizing feature receptive field and interacting multi-granularity features, the steps comprising: Step S321: multi-hole rate convolution group design, for capturing context information of different ranges, specifically, by adopting 2, 4 and 8 hole rate convolution to process multi-scale pyramid features, features with different receptive fields at the same scale are obtained; Step S322: gate weight generation, for dynamically selecting the best receptive field, specifically, by subnetwork calculating the weight of each spatial position, the gate weight of different receptive fields at the same scale is obtained; Step S323: feature fusion, for adaptively fusing multi-granularity features, specifically, based on the gate weight of different receptive fields at the same scale, the features of different receptive fields at the same scale are weighted and summed to obtain multi-scale optimized pyramid features; Step S33: Multi-task output design is used for cooperatively completing text detection, script classification and bounding box regression, and the steps include: Step S331: Multi-task special feature generation is used for providing customized features for different tasks, and specifically, multi-scale task special features are obtained by processing multi-scale optimized pyramid features through an independent 3*3 convolutional layer; Step S332: Cooperative attention unit design is used for enhancing information flow among tasks, and specifically, multi-scale text-script attention features are obtained by first taking text detection features at the same scale as queries, script classification features as keys and values, and processing them through a cross-attention mechanism, and multi-scale text-boundary attention features are obtained by taking text detection features at the same scale as queries and boundary box features as keys and values, and processing them through a cross-attention mechanism; Then, the text-script attention features and the text-boundary attention features at the same scale are spliced, processed through a linear layer, and then connected in residual with the text detection features at the same scale to obtain multi-scale text detection cooperative features; Finally, the multi-scale script classification features and the multi-scale boundary box features are processed in the same way to obtain multi-scale script classification cooperative features and multi-scale boundary box cooperative features; Step S333: Single-scale output acquisition is used for generating single-scale model prediction output results, and specifically, text probability maps, script classification maps and boundary box maps are obtained by processing task cooperative features through a 1*1 convolutional layer and an activation function; Step S334: Multi-scale output fusion is used for fusing multi-scale model prediction output results, and specifically, final text probability maps, final script classification maps and final boundary box maps are obtained by upsampling all-scale prediction maps to the size of the model input image and merging them; Step S34: Model construction and training, and specifically, the multi-granularity deep learning model is constructed by integrating the backbone feature extraction, the multi-granularity feature interaction and the multi-task output design, the model is trained based on the detection training set and the detection test set and the performance is verified, and the multi-granularity deep learning model is obtained as a text detection model; Step S35: Data set acquisition is used for acquiring training data and test data for subsequent text recognition models, and specifically, the detection training set and the detection test set are taken as inputs of the text detection model, and recognition training sets and recognition test sets are obtained after processing. 3.The deep learning-based multilingual mixed script recognition method of claim 1, wherein: The text recognition model construction specifically includes the following steps: Step S41: Dynamic routing encoding is used for adaptively extracting features of different script texts, and the steps include: Step S411: Routing weight calculation is used for obtaining primary recognition features and determining the weights of expert paths, and specifically, the primary recognition features are obtained by processing text region images obtained by cropping based on the final text probability map through a convolutional neural network, and the weights of the expert paths are obtained by calculating the weights of each expert path through an independent multi-layer perceptron to obtain expert path routing weights; Step S412: expert feature extraction and fusion, for providing multi-language feature representation and fusing multi-expert path features, specifically, a plurality of independent multi-layer perceptrons are used as expert paths, each expert path extracts a language-specific feature representation based on the primary recognition features, and the path fusion features are obtained by weighted fusion based on the expert path routing weights; Step S413: sequence feature generation, for converting the feature map into a sequence, specifically, the path fusion features are converted into a time sequence through a reshaping operation to obtain the encoded time sequence features; Step S42: bidirectional time sequence context mechanism design, for capturing the context dependence of the text sequence and dynamically integrating bidirectional information, specifically, a bidirectional long short-term memory network is used to capture the context features from left to right and right to left of the time sequence respectively, to obtain the forward hidden state and the backward hidden state, and the enhanced context features are obtained by fusing through the gating mechanism; Step S43: visual semantic alignment, for preventing the decoding process from deviating from the visual evidence, the steps include: Step S431: global visual feature extraction, for preserving the visual information of the input image, specifically, the final text probability map, the final language classification map and the final bounding box map are spliced in the channel dimension, and an independent convolutional neural network is used for feature extraction to obtain the fusion visual features; Step S432: gated alignment fusion, for balancing and fusing the enhanced context features and the fusion visual features, specifically, the enhanced context features and the fusion visual features are spliced, after linear mapping, the gating weight is calculated by using the sigmoid activation function, and the enhanced context features and the fusion visual features after linear mapping are weighted and fused to obtain the visual semantic fusion features; Step S44: text recognition output acquisition, for obtaining the text recognition result, specifically, the visual semantic fusion features after linear mapping are processed by using the softmax activation function, and the text recognition result is obtained after the result is decoded by CTC; Step S45: model construction and training, specifically, the bidirectional context network model is constructed by integrating the dynamic routing encoding, the bidirectional time sequence context mechanism design, the visual semantic alignment and the text recognition output acquisition, the model is trained based on the recognition training set and the recognition test set and the performance is verified, and the bidirectional context network model is obtained as the text recognition model. 4.The deep learning-based multilingual mixed script recognition method of claim 1, wherein: The recognition historical original image data set and the recognition current original image data set both contain text image data, and the recognition historical original image data set further contains bounding box annotation data, text content annotation data and text language annotation data. 5.The deep learning based multi-script mixed script recognition method of claim 1, wherein: The data preliminary processing specifically includes the following steps: Step S21: color normalization, for eliminating the influence of light and color difference on recognition, specifically, the image color distribution is adaptively learned by a shallow CNN network and the contrast and brightness are dynamically adjusted to obtain the color normalized image; Step S22: random visual confusion, for simulating image degradation in real scenes, specifically, enhanced images are obtained by applying random block occlusion, Gaussian noise and blur operations; Step S23: geometric space deformation, for improving the adaptability of the model to the deformed text, specifically, by applying random elastic deformation and smooth geometric transformation, simulating text bending and perspective changes, obtaining the deformed image; Step S24: multi-scale image generation, for adapting the model to different sizes of text, specifically, by generating multiple scale image versions, ensuring that the model can process text regions of various sizes, obtaining multi-scale images; Step S25: dataset segmentation, for obtaining training data and test data, specifically, performing dataset segmentation on the recognition historical original image dataset; Through the color normalization and the multi-scale image generation, the recognition current original image dataset is preliminarily processed to obtain a to-be-recognized image dataset, and through the color normalization, the random visual confusion, the geometric space deformation, the multi-scale image generation and the dataset segmentation, the recognition historical original image dataset is preliminarily processed to obtain a detection training set and a detection test set.

6. A deep learning-based multilingual mixed script recognition system for implementing the deep learning-based multilingual mixed script recognition method according to any one of claims 1-5, characterized in that: It comprises a text image acquisition module, a data preliminary processing module, a text detection model construction module, a text recognition model construction module and a multi-script mixed character recognition module. 7.The deep learning based multi-script mixed script recognition system of claim 6, wherein: The text image acquisition module is used for acquiring original data, obtaining a character recognition original image dataset through the acquisition of the original data, and sending the character recognition original image dataset to the data preliminary processing module; The data preliminary processing module is used for data preliminary processing, obtaining a to-be-recognized image dataset, a detection training set and a detection test set through the data preliminary processing, and sending the to-be-recognized image dataset to the multi-script mixed character recognition module and the detection training set and the detection test set to the text detection model construction module; The text detection model construction module is used for text detection model construction, processing the detection training set and the detection test set through the construction of a multi-granularity deep learning model as a text detection model, obtaining a recognition training set and a recognition test set, and sending the text detection model to the multi-script mixed character recognition module and the recognition training set and the recognition test set to the text recognition model construction module; The text recognition model construction module is used for text recognition model construction, processing current data through the construction of a bidirectional context network model as a text recognition model, and sending the text recognition model to the multi-script mixed character recognition module; The multi-script mixed character recognition module is used for multi-script mixed character recognition, obtaining a final text recognition result through the processing of current data by the text detection model combined with the text recognition model.

Images