Dynamic data processing and recognition method and device for mixed font text

Abstract

The present invention relates to a method and device for dynamic data processing and recognition of mixed-font text, belonging to the field of optical character recognition technology. The method comprises the following steps: collecting a mixed-font text image to be processed and inputting it into an enhanced DBNet detection network to locate the text area; integrating a lightweight MobileNetV3 font classification subnetwork, sharing underlying features with the detection backbone, training a multi-task network using a joint loss function, and outputting text areas with font type labels; then selecting a recognition branch based on the label, using a traditional CRNN for printed text recognition; replacing traditional convolution with linear variable convolution for handwritten text, using a heat map to guide kernel dynamic scaling, and embedding a multi-head attention mechanism in the BiLSTM layer for recognition; and finally fusing the results to output labeled detection and recognition results. The present invention improves the accuracy of mixed-font text data processing and recognition, enhances the adaptability and robustness of handwritten text recognition, and meets the needs of diverse text recognition.

Description

Technical Field

[0001] The present invention relates to a method and device for dynamic data processing and recognition of mixed-font text, and is particularly suitable for automatic recognition of mixed printed and handwritten text in industrial scenarios, belonging to the technical field of optical character recognition (OCR). Background Art

[0002] Writing is a tool used by early humans to record and express information through symbols. Chinese characters are one of the oldest written languages ​​in the world. As a crucial vehicle for Chinese culture, their existence holds profound significance. With the advancement of technology, digital images are now present in nearly every aspect of life. For example, in manufacturing plants, parts production orders, equipment inspection records, and workpiece quality inspection reports are often stored in paper form, often containing large amounts of printed and handwritten information. Currently, converting this paper data into electronic documents relies primarily on manual data entry and management, resulting in inefficient data processing, prone to errors, and difficult to trace. This is why optical character recognition (OCR) technology has emerged.

[0003] Optical character recognition technology has broad application prospects in the manufacturing industry. For example, in parts quality inspection, it can automatically parse paper work orders, such as first-article double-inspection records, to improve inspection efficiency and reduce human error. In equipment maintenance, it can quickly identify and convert paper equipment inspection records to optimize operation and maintenance processes. In production management, it can digitize the contents of paper records such as workpieces and bills of materials, improving information management. Furthermore, combining OCR with industrial automation systems can implement intelligent auditing and anomaly detection, improving data processing efficiency and intelligence in the manufacturing industry. Therefore, research on work order recognition technology for parts in the manufacturing industry will greatly enhance the digitization and automation of production data.

[0004] In recent years, with the rapid development of deep learning technology, the field of text detection and recognition has achieved significant breakthroughs and has become a mainstream research direction. Before the popularization of deep learning, traditional handwriting recognition methods mainly relied on steps such as image preprocessing, feature extraction, and classification and recognition. They usually required locating the text area, denoising, and tilt correction, and then using manual features such as oriented gradient histograms combined with classifiers for recognition. However, traditional methods suffer from low detection accuracy and poor recognition robustness when faced with complex scenarios such as mixed text in production work orders and handwritten text with different styles. There is also no end-to-end process designed for the mixed data characteristics of industrial scenarios. Summary of the Invention

[0005] In order to solve the above problems, the present invention proposes a dynamic data processing and recognition method and device for mixed-font text, which improves data segmentation accuracy through multi-scale feature fusion and dynamic convolution technology, and combines multi-task collaborative training to achieve end-to-end optimization of detection, classification and recognition.

[0006] The technical solution adopted by the present invention to solve its technical problems is:

[0007] In a first aspect, an embodiment of the present invention provides a method for dynamic data processing and recognition of mixed-font text, comprising the following steps:

[0008] S1: Collect the mixed font text image to be processed and input it into the enhanced DBNet text detection network to locate and detect the text area in the image. The enhanced DBNet text detection network embeds the coordinate attention (CA) mechanism in the FPN feature pyramid, establishes a dynamic association between spatial position and channel features through multi-scale feature fusion, and outputs a probability map and a threshold map;

[0009] S2 integrates a lightweight MobileNetV3 font classification subnetwork, shares underlying features with the detection backbone, and designs a joint loss function to train the network in a multi-task joint learning framework, outputting text regions with font type labels. The joint loss function includes detection loss and classification loss.

[0010] S3 selects the corresponding recognition branch according to the font type label. For the printed text area, it recognizes the printed text through the convolutional layer, recurrent layer, and transcription layer based on the traditional CRNN network;

[0011] S4, for handwritten text areas, uses linear variable convolution instead of traditional convolution, and adapts to the deformation of connected strokes through dynamic kernel scaling guided by heat maps. A multi-head attention mechanism is embedded in the BiLSTM layer of the CRNN network to establish a turning point spatiotemporal correlation model for handwritten text recognition;

[0012] S5, fusing the handwritten text recognition results with the printed text recognition results, and outputting the labeled text detection area results and recognition results.

[0013] As a possible implementation of this embodiment, the enhanced DBNet text detection network further includes:

[0014] The DBNet main network for text detection first performs deep feature extraction on the input image to generate a multi-scale feature map; then it fuses it with the FPN through upsampling operations to enhance the multi-scale semantic information; finally, it further predicts a probability map and a threshold map for the training phase, and the two are combined to obtain an approximate binary image result.

[0015] As a possible implementation of this embodiment, the binary image result is achieved by an approximate step function, which combines the predicted probability map and the predicted threshold map to obtain an approximate binary image, making the binarization process differentiable and facilitating back-propagation training;

[0016] The expression of the approximate step function is:

[0017] ,

[0018] in, represents an approximate binary image, is a factor, represents the predicted probability plot, Threshold map representing the predictions.

[0019] As a possible implementation of this embodiment, the coordinate attention (CA) mechanism includes:

[0020] Global information encoding performs global average pooling on the input feature map in the horizontal and vertical directions to generate two separate feature vectors:

[0021] ,

[0022] in, represents the input feature map, Indicates the number of channels, Indicates height, Indicates width; represents the horizontal attention feature, represents the vertical attention feature, is the channel index, is the position index in the width direction, is the position index in the height direction;

[0023] Interactive transformation, using shared convolutional layers for channel interaction:

[0024] ,

[0025] in, Represents horizontal attention features Perform 1D convolution (weights are ) and nonlinear activation ( ) and the features obtained after Represents nonlinear activation functions (such as ReLU, Sigmoid, etc.), introduces nonlinear transformations, and enhances the model's expressiveness; Indicates the effect on The 1D convolution weight matrix is ​​responsible for information interaction between channels and adjusting horizontal features; Represents the horizontal attention feature calculated by the formula, that is, the result of global pooling of the input feature map X along the height direction; Represents vertical attention features Perform 1D convolution (weights are ) and nonlinear activation ( ) after the features are obtained; Indicates the effect on The 1D convolution weight matrix is ​​responsible for information interaction between channels and adjusting vertical features; Represents the vertical attention feature calculated by the formula, that is, the result of global pooling of the input feature map X along the width direction;

[0026] Attention weight calculation, fusing the attention information in the horizontal and vertical directions, calculating the final attention weight, which is used to re-adjust the input feature map:

[0027] ,

[0028] in, represents the final attention weight, is the fusion weight, Represents the splicing operation of features, and Splicing in a certain dimension to fuse the horizontal and vertical attention information.

[0029] As a possible implementation of this embodiment, the coordinate attention (CA) mechanism is embedded in the fused multi-scale feature map, covering three scales of 1 / 16, 1 / 8, and 1 / 4, to fully explore the spatial information and channel dependency in features at different levels.

[0030] As a possible implementation of this embodiment, the lightweight MobileNetV3 font classification subnet includes five stages: MobileNetV3 Bottleneck feature compression, nonlinear activation, Dropout anti-overfitting mechanism, global average pooling and fully connected classification layer, and Softmax probability output. In the MobileNetV3 Bottleneck feature compression stage, depthwise separable convolution and Bottleneck structure are used to extract features from the input feature map, and the channels are weighted through the SE attention mechanism to finally output the feature map. In the nonlinear activation stage, a ReLU activation function is applied after the output of the Bottleneck feature compression stage to enhance the nonlinear fitting capability of the model. In the Dropout anti-overfitting mechanism stage, a Dropout operation is introduced after the output of the nonlinear activation stage to randomly mask some neurons to prevent model overfitting. In the global average pooling and fully connected classification layer stage, the feature map after Dropout is input into the global average pooling module to compress the spatial dimension to obtain a feature vector, which is then input into the fully connected layer for classification processing. In the Softmax probability output stage, the output of the fully connected layer is processed to generate a predicted probability for each font category.

[0031] As a possible implementation of this embodiment, the MobileNetV3 Bottleneck feature compression stage specifically includes the following steps:

[0032] The input feature map is channel-expanded through the first convolutional layer, and then each channel is independently convolved using depth-separable convolution. The channels are then weighted through the SE attention mechanism, and finally the channels are compressed through convolution to obtain the output feature map.

[0033] As a possible implementation method of this embodiment, the SE attention mechanism includes a squeeze stage and an excitation stage. The squeeze stage extracts channel descriptors, the excitation stage processes the channel descriptors to generate channel weights, and then multiplies the channel weights with the original feature map to weight the channels.

[0034] As a possible implementation method of this embodiment, in the global average pooling and fully connected classification layer stage, the global average pooling module compresses the spatial information of the feature map after Dropout into a value, obtains a feature vector with a length of 128, and then inputs the feature vector into a two-layer fully connected network for classification processing. The first layer of the fully connected network reduces the 128-dimensional feature vector to 64 dimensions, and the second layer of the fully connected network maps the 64-dimensional feature vector to the final category space; the category space includes two categories: handwriting and print; in the Softmax probability output stage, the Softmax function converts the output of the fully connected layer into a probability distribution, and the category with the largest probability in the output is the final prediction result of the model.

[0035] As a possible implementation method of this embodiment, the lightweight MobileNetV3 font classification subnet receives the low-dimensional feature map from the backbone network and the original text image as input, and relies on the 1 / 4 scale feature map extracted by the backbone network, and at the same time fuses an independent path feature from the original image input. The two are feature spliced ​​and convolutionally fused in the CONNECT module to construct a composite feature representation. The fused features are efficiently compressed through the MobileNetV3 module, and processed through global average pooling and Softmax activation function to output the probability distribution of the target font category.

[0036] As a possible implementation method of this embodiment, the joint loss function is defined as a weighted combination, including text detection loss and font classification loss. The weighting method selects uncertainty weighting. The model will automatically adjust the weights of the two during the training process to improve the stability and robustness of multi-task learning.

[0037] As a possible implementation method of this embodiment, the text detection loss consists of a binary image segmentation loss and a bounding box geometric regression loss. The binary image segmentation loss is a cross entropy loss based on pixel-level binary classification, and the bounding box geometric regression loss is used to constrain the difference in geometric parameters between the predicted bounding box and the true box.

[0038] As a possible implementation of this embodiment, the font classification loss uses a standard multi-class cross entropy loss as an optimization function of the classification target.

[0039] As a possible implementation of this embodiment, S3 includes the following steps:

[0040] Extracting features of the input sequence through a convolutional layer that uses a local connection and weight sharing mechanism to perform convolution operations and generate output features;

[0041] The features extracted by the convolutional layer are input into the recurrent layer for time series feature modeling. The recurrent layer uses an LSTM network, which includes an input gate, a forget gate, and an output gate. The gating mechanism retains important information and gradually forgets irrelevant data. The bidirectional LSTM structure is used to enhance the context modeling capability through the collaborative processing of forward LSTM units and backward LSTM units, and finally outputs the concatenation of the forward and backward hidden states.

[0042] The output of the recurrent layer is sequence-decoded through the Connectionist Temporal Classification (CTC) algorithm, and the output of the RNN is converted into the final text sequence. The CTC algorithm trains the model by mapping the label sequence to all alignments of the model output sequence and calculating the probabilities of these alignments to achieve sequence-to-sequence learning tasks.

[0043] As a possible implementation of this embodiment, the convolution operation of the convolution layer is expressed as:

[0044] ,

[0045] in, is the output feature at the time point, is the input sequence, is the convolution kernel, is the element index within the convolution kernel.

[0046] As a possible implementation of this embodiment, S4 includes the following steps:

[0047] The linear variable convolution (LD-Conv) is used to replace the traditional convolution in the convolutional neural network. The LD-Conv uses different numbers of convolution kernel parameters to extract features from handwritten text images.

[0048] Inserting a lightweight branch after the Stage-2 output of the convolutional neural network to further enhance feature representation through lightweight convolution;

[0049] Calculate the mean of the heat map and dynamically adjust the kernel weights of subsequent convolutional layers based on the mean of the heat map;

[0050] Introducing the Linear Deformation Module (LDM) to learn the affine transformation matrix and perform rigid transformation on characters;

[0051] Performing singular value decomposition on the affine transformation matrix and constraining it to be a diagonal matrix to avoid excessive deformation;

[0052] Furthermore, multi-head attention (MHA) is embedded in the bidirectional long short-term memory (BiLSTM) layer of the convolutional recurrent neural network (CRNN) to establish a turning point spatiotemporal correlation model to capture the contextual dependencies between different positions in the input sequence.

[0053] As a possible implementation of this embodiment, the linear variable convolution (LD-Conv) automatically adapts to changes in target shape through dynamic offsets, enhancing the model's flexibility and adaptability. The dynamic weight scaling mechanism adjusts the weight distribution of subsequent convolutional layers by calculating the mean of heatmaps. When the degree of connected strokes in an image is high, the weight of local dynamic convolution is increased to adapt to the deformed area; when the degree of connected strokes is low, the basic convolutional feature extraction capability is maintained. The affine transformation matrix learned by the linear deformation module (LDM) includes a scaling factor that controls the direction, a factor that controls the shear angle, and a factor responsible for translation to compensate for character scaling, shearing, and translation changes. The multi-head attention (MHA) maps input features into multiple different representation subspaces and independently performs a self-attention mechanism within each subspace to capture contextual dependencies between different positions in the input sequence. The self-attention mechanism maps input features into a query (Q), a key (K), and a value (V). The attention weight is calculated based on the dot product of Q and K, and the weighted sum is applied to V to output the contextual representation of the current time step. The multi-head attention (MHA) structure passes input features into multiple attention heads in parallel, calculates attention representations separately, fuses the outputs of each head through a splicing operation, and further integrates them through a linear transformation to obtain a global representation with richer context dependencies.

[0054] As a possible implementation of this embodiment, S5 includes the following steps:

[0055] Input original image I∈ , represents the height of the input feature map, Represents the width of the input feature map, 3 means that the image has three RGB channels, and the small text features are enhanced by improving the DBNet detection module and outputting the text area location and probability map P∈ and threshold graph T∈ , the differentiable binarization formula is used to obtain the binarization result; at the same time, the lightweight MobileNetV3 font classification subnetwork is used to extract ROI features for each detected text area and output the font probability yi∈[0,1];

[0056] Assign the text area to the corresponding handwritten text recognition branch or printed text recognition branch according to the obtained font probability yi;

[0057] For text regions assigned to the handwritten text recognition branch, an enhanced CRNN model, including LD-Conv dynamic convolution and multi-head attention (MHA) temporal modeling, is used to output handwritten text recognition results. For text regions assigned to the printed text recognition branch, a standard CRNN architecture is used to output printed text recognition results.

[0058] The handwritten text recognition results and the printed text recognition results are combined to generate the final text content and position.

[0059] As a possible implementation of this embodiment, the improved DBNet detection module includes an FPN+CA module for enhancing small text features. The differentiable binarization formula is: B=sign(PT×k), where k=50 controls the binarization slope. The loss function of the lightweight MobileNetV3 font classification subnetwork is a joint optimization loss function, including classification loss and balance weight λ, with a value of λ being 0.6; the convolution kernel size of the LD-Conv dynamic convolution in the enhanced CRNN model is dynamically adjusted according to the thermal value Hlocal of the connected stroke area, and the adjustment coefficient is α; the enhanced CRNN model also includes multi-head attention MHA temporal modeling, and Q, K, and V are respectively obtained by linear transformation of the recurrent layer features; the recognition loss function is a combination of CTC loss and branch loss.

[0060] In a second aspect, an embodiment of the present invention provides a dynamic data processing and recognition device for mixed-font text, comprising:

[0061] The text region localization module is used to collect mixed-font text images to be processed and input them into the enhanced DBNet text detection network to locate and detect text regions in the image. The enhanced DBNet text detection network embeds a coordinate attention (CA) mechanism in the FPN feature pyramid, establishes a dynamic association between spatial position and channel features through multi-scale feature fusion, and outputs a probability map and a threshold map.

[0062] The text region output module integrates a lightweight MobileNetV3 font classification subnetwork, shares underlying features with the detection backbone, and designs a joint loss function to train the multi-task joint learning framework network and output text regions with font type labels. The joint loss function includes detection loss and classification loss.

[0063] The printed text recognition module is used to select the corresponding recognition branch based on the font type label. For printed text areas, it uses the traditional CRNN network to recognize printed text through convolutional layers, recurrent layers, and transcription layers.

[0064] The handwritten text recognition module uses linear variable convolution instead of traditional convolution for handwritten text areas. It adapts to deformation of connected strokes through dynamic kernel scaling guided by heat maps. It also embeds a multi-head attention mechanism in the BiLSTM layer of the CRNN network and establishes a turning point spatiotemporal correlation model to recognize handwritten text.

[0065] The merged result output module is used to fuse the handwritten text recognition results with the printed text recognition results, and output the text detection area results and recognition results with labels.

[0066] The beneficial effects of the technical solutions of the embodiments of the present invention are as follows:

[0067] This paper improves the accuracy of localization detection of text regions through an enhanced DBNet text detection network and a coordinate attention mechanism. It also employs a lightweight MobileNetV3 font classification subnetwork and a multi-task joint learning framework to reduce model complexity and improve recognition efficiency. Separate recognition branches are designed for printed and handwritten text, improving the accuracy of mixed-font text recognition. Furthermore, the introduction of linear variable convolution and a multi-head attention mechanism enhances the adaptability and robustness of handwritten text recognition. The paper also improves data segmentation accuracy through multi-scale feature fusion and dynamic convolution techniques, and combines multi-task collaborative training to achieve end-to-end optimization of detection, classification, and recognition.

[0068] The present invention improves the processing and recognition accuracy of mixed-font text data, enhances the adaptability and robustness of handwritten text recognition, meets the needs of diverse text recognition, and is particularly suitable for the automated recognition of mixed printed and handwritten text in industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a flow chart of a method for dynamic data processing and recognition of mixed-font text according to an exemplary embodiment;

[0070] Figure 2 The figure is a schematic structural diagram of a device for dynamic data processing and recognition of mixed-font text according to an exemplary embodiment. DETAILED DESCRIPTION

[0071] In order to more clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0072] like Figure 1 As shown, an embodiment of the present invention provides a method for dynamic data processing and recognition of mixed font text, comprising the following steps:

[0073] S1, collects the mixed font text image to be processed, and inputs the collected text image into the enhanced DBNet text detection network to locate and detect the text area in the image. The enhanced DBNet text detection network embeds the coordinate attention (CA) mechanism in the FPN feature pyramid, establishes a dynamic association between spatial position and channel features through multi-scale feature fusion, and outputs a probability map and a threshold map;

[0074] S2 integrates a lightweight MobileNetV3 font classification subnetwork, shares underlying features with the detection backbone, and designs a joint loss function to train the network in a multi-task joint learning framework, outputting text regions with font type labels. The joint loss function includes detection loss and classification loss.

[0075] S3 selects the corresponding recognition branch according to the font type label. For the printed text area, it recognizes the printed text through the convolutional layer, recurrent layer, and transcription layer based on the traditional CRNN network;

[0076] S4, for handwritten text areas, uses linear variable convolution instead of traditional convolution, and adapts to the deformation of connected strokes through dynamic kernel scaling guided by heat maps. A multi-head attention mechanism is embedded in the BiLSTM layer of the CRNN network to establish a turning point spatiotemporal correlation model for handwritten text recognition;

[0077] S5, fusing the handwritten text recognition results with the printed text recognition results, and outputting the labeled text detection area results and recognition results.

[0078] As a possible implementation of this embodiment, the enhanced DBNet text detection network further includes:

[0079] The DBNet main network for text detection first performs deep feature extraction on the input image to generate a multi-scale feature map; then it fuses it with the FPN through upsampling operations to enhance the multi-scale semantic information; finally, it further predicts a probability map and a threshold map for the training phase, and the two are combined to obtain an approximate binary image result.

[0080] As a possible implementation of this embodiment, the binary image result is achieved by an approximate step function, which combines the predicted probability map and the predicted threshold map to obtain an approximate binary image, making the binarization process differentiable and facilitating back-propagation training;

[0081] The expression of the approximate step function is:

[0082] ,

[0083] in, represents an approximate binary image, is a factor, represents the predicted probability plot, Threshold map representing the predictions.

[0084] As a possible implementation of this embodiment, the coordinate attention (CA) mechanism includes:

[0085] Global information encoding performs global average pooling on the input feature map in the horizontal and vertical directions to generate two separate feature vectors:

[0086] ,

[0087] in, represents the input feature map, Indicates the number of channels, Indicates height, Indicates width; represents the horizontal attention feature, represents the vertical attention feature, is the channel index, is the position index in the width direction, is the position index in the height direction;

[0088] Interactive transformation, using shared convolutional layers for channel interaction:

[0089] ,

[0090] in, Represents horizontal attention features Perform 1D convolution (weights are ) and nonlinear activation ( ) and the features obtained after Represents nonlinear activation functions (such as ReLU, Sigmoid, etc.), introduces nonlinear transformations, and enhances the model's expressiveness; Indicates the effect on The 1D convolution weight matrix is ​​responsible for information interaction between channels and adjusting horizontal features; Represents the horizontal attention feature calculated by the formula, that is, the result of global pooling of the input feature map X along the height direction; Represents vertical attention features Perform 1D convolution (weights are ) and nonlinear activation ( ) after the features are obtained; Indicates the effect on The 1D convolution weight matrix is ​​responsible for information interaction between channels and adjusting vertical features; Represents the vertical attention feature calculated by the formula, that is, the result of global pooling of the input feature map X along the width direction;

[0091] Attention weight calculation, fusing the attention information in the horizontal and vertical directions, calculating the final attention weight, which is used to re-adjust the input feature map:

[0092] ,

[0093] in, represents the final attention weight, is the fusion weight, Represents the splicing operation of features, and Splicing in a certain dimension to fuse the horizontal and vertical attention information.

[0094] As a possible implementation of this embodiment, the coordinate attention (CA) mechanism is embedded in the fused multi-scale feature map, covering three scales of 1 / 16, 1 / 8, and 1 / 4, to fully explore the spatial information and channel dependency in features at different levels.

[0095] As a possible implementation of this embodiment, the lightweight MobileNetV3 font classification subnet includes five stages: MobileNetV3 Bottleneck feature compression, nonlinear activation, Dropout anti-overfitting mechanism, global average pooling and fully connected classification layer, and Softmax probability output. In the MobileNetV3 Bottleneck feature compression stage, depthwise separable convolution and Bottleneck structure are used to extract features from the input feature map, and the channels are weighted through the SE attention mechanism to finally output the feature map. In the nonlinear activation stage, a ReLU activation function is applied after the output of the Bottleneck feature compression stage to enhance the nonlinear fitting capability of the model. In the Dropout anti-overfitting mechanism stage, a Dropout operation is introduced after the output of the nonlinear activation stage to randomly mask some neurons to prevent model overfitting. In the global average pooling and fully connected classification layer stage, the feature map after Dropout is input into the global average pooling module to compress the spatial dimension to obtain a feature vector, which is then input into the fully connected layer for classification processing. In the Softmax probability output stage, the output of the fully connected layer is processed to generate a predicted probability for each font category.

[0096] As a possible implementation of this embodiment, the MobileNetV3 Bottleneck feature compression stage specifically includes the following steps:

[0097] The input feature map is channel-expanded through the first convolutional layer, and then each channel is independently convolved using depth-separable convolution. The channels are then weighted through the SE attention mechanism, and finally the channels are compressed through convolution to obtain the output feature map.

[0098] As a possible implementation method of this embodiment, the SE attention mechanism includes a squeeze stage and an excitation stage. The squeeze stage extracts channel descriptors, the excitation stage processes the channel descriptors to generate channel weights, and then multiplies the channel weights with the original feature map to weight the channels.

[0099] As a possible implementation method of this embodiment, in the global average pooling and fully connected classification layer stage, the global average pooling module compresses the spatial information of the feature map after Dropout into a value, obtains a feature vector with a length of 128, and then inputs the feature vector into a two-layer fully connected network for classification processing. The first layer of the fully connected network reduces the 128-dimensional feature vector to 64 dimensions, and the second layer of the fully connected network maps the 64-dimensional feature vector to the final category space; the category space includes two categories: handwriting and print; in the Softmax probability output stage, the Softmax function converts the output of the fully connected layer into a probability distribution, and the category with the largest probability in the output is the final prediction result of the model.

[0100] As a possible implementation method of this embodiment, the lightweight MobileNetV3 font classification subnet receives the low-dimensional feature map from the backbone network and the original text image as input, and relies on the 1 / 4 scale feature map extracted by the backbone network, and at the same time fuses an independent path feature from the original image input. The two are feature spliced ​​and convolutionally fused in the CONNECT module to construct a composite feature representation. The fused features are efficiently compressed through the MobileNetV3 module, and processed through global average pooling and Softmax activation function to output the probability distribution of the target font category.

[0101] As a possible implementation method of this embodiment, the joint loss function is defined as a weighted combination, including text detection loss and font classification loss. The weighting method selects uncertainty weighting. The model will automatically adjust the weights of the two during the training process to improve the stability and robustness of multi-task learning.

[0102] As a possible implementation method of this embodiment, the text detection loss consists of a binary image segmentation loss and a bounding box geometric regression loss. The binary image segmentation loss is a cross entropy loss based on pixel-level binary classification, and the bounding box geometric regression loss is used to constrain the difference in geometric parameters between the predicted bounding box and the true box.

[0103] As a possible implementation of this embodiment, the font classification loss uses a standard multi-class cross entropy loss as an optimization function of the classification target.

[0104] As a possible implementation of this embodiment, S3 includes the following steps:

[0105] Extracting features of the input sequence through a convolutional layer that uses a local connection and weight sharing mechanism to perform convolution operations and generate output features;

[0106] The features extracted by the convolutional layer are input into the recurrent layer for time series feature modeling. The recurrent layer uses an LSTM network, which includes an input gate, a forget gate, and an output gate. The gating mechanism retains important information and gradually forgets irrelevant data. The bidirectional LSTM structure is used to enhance the context modeling capability through the collaborative processing of forward LSTM units and backward LSTM units, and finally outputs the concatenation of the forward and backward hidden states.

[0107] The output of the recurrent layer is sequence-decoded through the Connectionist Temporal Classification (CTC) algorithm, and the output of the RNN is converted into the final text sequence. The CTC algorithm trains the model by mapping the label sequence to all alignments of the model output sequence and calculating the probabilities of these alignments to achieve sequence-to-sequence learning tasks.

[0108] As a possible implementation of this embodiment, the convolution operation of the convolution layer is expressed as:

[0109] ,

[0110] in, is the output feature at the time point, is the input sequence, is the convolution kernel, is the element index within the convolution kernel.

[0111] As a possible implementation of this embodiment, S4 includes the following steps:

[0112] The linear variable convolution (LD-Conv) is used to replace the traditional convolution in the convolutional neural network. The LD-Conv uses different numbers of convolution kernel parameters to extract features from handwritten text images.

[0113] Inserting a lightweight branch after the Stage-2 output of the convolutional neural network to further enhance feature representation through lightweight convolution;

[0114] Calculate the mean of the heat map and dynamically adjust the kernel weights of subsequent convolutional layers based on the mean of the heat map;

[0115] Introducing the Linear Deformation Module (LDM) to learn the affine transformation matrix and perform rigid transformation on characters;

[0116] Performing singular value decomposition on the affine transformation matrix and constraining it to be a diagonal matrix to avoid excessive deformation;

[0117] Furthermore, multi-head attention (MHA) is embedded in the bidirectional long short-term memory (BiLSTM) layer of the convolutional recurrent neural network (CRNN) to establish a turning point spatiotemporal correlation model to capture the contextual dependencies between different positions in the input sequence.

[0118] As a possible implementation of this embodiment, the linear variable convolution (LD-Conv) automatically adapts to changes in target shape through dynamic offsets, enhancing the model's flexibility and adaptability. The dynamic weight scaling mechanism adjusts the weight distribution of subsequent convolutional layers by calculating the mean of heatmaps. When the degree of connected strokes in an image is high, the weight of local dynamic convolution is increased to adapt to the deformed area; when the degree of connected strokes is low, the basic convolutional feature extraction capability is maintained. The affine transformation matrix learned by the linear deformation module (LDM) includes a scaling factor that controls the direction, a factor that controls the shear angle, and a factor responsible for translation to compensate for character scaling, shearing, and translation changes. The multi-head attention (MHA) maps input features into multiple different representation subspaces and independently performs a self-attention mechanism within each subspace to capture contextual dependencies between different positions in the input sequence. The self-attention mechanism maps input features into a query (Q), a key (K), and a value (V). The attention weight is calculated based on the dot product of Q and K, and the weighted sum is applied to V to output the contextual representation of the current time step. The multi-head attention (MHA) structure passes input features into multiple attention heads in parallel, calculates attention representations separately, fuses the outputs of each head through a splicing operation, and further integrates them through a linear transformation to obtain a global representation with richer context dependencies.

[0119] As a possible implementation of this embodiment, S5 includes the following steps:

[0120] Input original image I∈ , represents the height of the input feature map, Represents the width of the input feature map, 3 means that the image has three RGB channels, and the small text features are enhanced by improving the DBNet detection module and outputting the text area location and probability map P∈ and threshold graph T∈ , the differentiable binarization formula is used to obtain the binarization result; at the same time, the lightweight MobileNetV3 font classification subnetwork is used to extract ROI features for each detected text area and output the font probability yi∈[0,1];

[0121] Assign the text area to the corresponding handwritten text recognition branch or printed text recognition branch according to the obtained font probability yi;

[0122] For text regions assigned to the handwritten text recognition branch, an enhanced CRNN model, including LD-Conv dynamic convolution and multi-head attention (MHA) temporal modeling, is used to output handwritten text recognition results. For text regions assigned to the printed text recognition branch, a standard CRNN architecture is used to output printed text recognition results.

[0123] The handwritten text recognition results and the printed text recognition results are combined to generate the final text content and position.

[0124] As a possible implementation of this embodiment, the improved DBNet detection module includes an FPN+CA module for enhancing small text features. The differentiable binarization formula is: B=sign(PT×k), where k=50 controls the binarization slope. The loss function of the lightweight MobileNetV3 font classification subnetwork is a joint optimization loss function, including classification loss and balance weight λ, with a value of λ being 0.6; the convolution kernel size of the LD-Conv dynamic convolution in the enhanced CRNN model is dynamically adjusted according to the thermal value Hlocal of the connected stroke area, and the adjustment coefficient is α; the enhanced CRNN model also includes multi-head attention MHA temporal modeling, and Q, K, and V are respectively obtained by linear transformation of the recurrent layer features; the recognition loss function is a combination of CTC loss and branch loss.

[0125] The present invention proposes a dynamic two-stage text data processing and recognition method for mixed fonts, which mainly consists of two stages:

[0126] 1) Detection Phase: Text images are fed into an improved Differentiable Binarization (DBNet)-based algorithm framework. A semantic information enhancement module and a coordinate attention mechanism are embedded in the feature pyramid FPN, significantly improving the accuracy of small text localization. Simultaneously, a lightweight MobileNetV3 font classification subnetwork is designed. By jointly optimizing the detection and classification loss functions, this network achieves the coordinated output of text region localization and font type discrimination (printed / handwritten classification), providing critical prior knowledge for subsequent recognition.

[0127] 2) Recognition Phase: An innovative dual-branch processing framework is constructed. For printed text, a traditional CRNN architecture is used for character recognition and output. For handwritten text, a CRNN variant enhanced with linear variable convolution (LD-Conv) and multi-head attention (MHA) is employed. Through heatmap-guided dynamic kernel scaling and spatiotemporal context modeling, the accuracy of handwritten connected stroke recognition is improved and the recognition results are output. While maintaining a high standard for character recognition, feature sharing technology enables knowledge sharing within the dual-branch recognition model, improving the overall recognition accuracy of mixed text.

[0128] like Figure 2 As shown, an embodiment of the present invention provides a dynamic data processing and recognition device for mixed font text, comprising:

[0129] The text region location detection module is used to collect mixed font text images to be processed and input the collected text images into the enhanced DBNet text detection network to locate and detect the text regions in the images. The enhanced DBNet text detection network embeds a coordinate attention mechanism in the FPN feature pyramid, establishes a dynamic association between spatial position and channel features through multi-scale feature fusion, and outputs a probability map and a threshold map;

[0130] The text region output module integrates a lightweight MobileNetV3 font classification subnetwork, shares underlying features with the detection backbone, and designs a joint loss function to train the multi-task joint learning framework network and output text regions with font type labels. The joint loss function includes detection loss and classification loss.

[0131] The printed text recognition module is used to select the corresponding recognition branch based on the font type label. For printed text areas, it uses the traditional CRNN network to recognize printed text through convolutional layers, recurrent layers, and transcription layers.

[0132] The handwritten text recognition module uses linear variable convolution instead of traditional convolution for handwritten text areas. It adapts to deformation of connected strokes through dynamic kernel scaling guided by heat maps. It also embeds a multi-head attention mechanism in the BiLSTM layer of the CRNN network and establishes a turning point spatiotemporal correlation model to recognize handwritten text.

[0133] The merged result output module is used to fuse the handwritten text recognition results with the printed text recognition results, and output the text detection area results and recognition results with labels.

[0134] The specific process of the present invention for dynamic data processing and recognition of mixed font text is as follows.

[0135] 1. Collect mixed-font text images to be processed and input them into the enhanced DBNet text detection network to locate and detect text areas in the images: A coordinate attention (CA) mechanism is embedded in the FPN feature pyramid. Through multi-scale feature fusion, a dynamic association between spatial position and channel features is established to improve sensitivity to small text.

[0136] (1) DBNet text detection main network:

[0137] First, deep feature extraction is performed on the input image to generate a multi-scale feature map. This is then fused with the FPN through upsampling to enhance the multi-scale semantic information. On this basis, a probability map and a threshold map for the training phase are further predicted, and the two are combined to produce an approximate binary image result. To make the binarization process differentiable and facilitate backpropagation training, DBNet introduces an approximate step function, which effectively achieves high-precision image segmentation. The specific form of this approximate function is:

[0138] (1),

[0139] in represents an approximate binary image, is a factor with a value of 50. represents the predicted probability plot, Represents the predicted threshold map. The differentiable binarization formula has similar properties to the sigmoid function. After binarization, its value range is limited to (0, 1). DBNet adaptively binarizes each pixel, obtains a threshold through learning, and incorporates the binarization operation into the network for training. This ensures that each pixel in the image has an adaptive threshold, better distinguishing foreground and background during object detection, thereby improving the performance of the DBNet algorithm.

[0140] (2) Main network embedding coordinate attention mechanism (CA):

[0141] CA is an efficient and lightweight attention mechanism that aims to improve the model's sensitivity to the distribution of targets in space. It is particularly suitable for detection tasks of dense, small-sized targets while maintaining low computational overhead. Specifically, CA first performs global average pooling on the input feature map in the horizontal and vertical directions to generate two separate feature vectors. These two vectors can capture the global information of the features in different directions, allowing the model to perceive the importance of the target in different spatial dimensions. This separate spatial encoding method is different from traditional global pooling. It retains more position information, allowing the model to not only focus on salient target areas, but also perceive the position distribution of targets in different directions, improving adaptability to complex scenes.

[0142] CA mainly consists of three parts: global information encoding, interaction transformation, and attention weight calculation. First, given the input feature map , CA adopts along the height ( ) and width ( ) direction, and calculate the horizontal attention features and vertical attention features respectively. The formula is as follows:

[0143] (2),

[0144] in, and The CA mechanism obtains two independent direction-aware features on each channel.

[0145] In order to reduce the amount of calculation and improve the expressiveness of features, CA uses shared Convolutional layer Inter-channel interaction:

[0146] (3),

[0147] in, and yes The weight matrix of the convolution, Represents a nonlinear activation function. This step is used to enhance information in different directions, so that the model can learn the distribution pattern of the target in space, thereby improving the accuracy of attention. CA calculates the final attention weight by fusing the attention information in the horizontal and vertical directions. :

[0148] (4),

[0149] in, Represents the splicing operation of features, is the fusion weight, and the final attention weight Will be used to rescale the input feature map , which enables the network to focus more on important target areas and suppress irrelevant information.

[0150] (3) The CA attention mechanism module is embedded in the feature pyramid FPN of the DBNet network to fuse the spatial position information and generate attention weights to adjust the feature map.

[0151] In terms of structural design, CA is cleverly embedded into the fused multi-scale feature maps, covering scales of 1 / 16, 1 / 8, and 1 / 4, to fully exploit the spatial information and channel dependencies within features at different levels. Shallower scales (such as 1 / 4) preserve rich spatial detail, making them crucial for processing small text and objects with blurred edges. This detailed information is crucial for accurately localizing text regions and their boundaries. By introducing a coordinate attention mechanism at multiple scales, the model can more accurately perceive the distribution, structure, and underlying geometric relationships of text regions at different semantic levels. This multi-scale attention mechanism not only enhances the detection branch's ability to focus on local regions but also strengthens the model's responsiveness to subtle changes, thereby ensuring higher detection accuracy. Furthermore, CA provides more discriminative input for subsequent feature fusion in the CONNECT module, ensuring efficient integration and precise transfer of features at different scales. Through structured feature fusion, the overall robustness of the model is significantly improved, enabling more accurate and reliable detection results in complex scenarios.

[0152] Second, multi-task collaborative optimization implementation: Integrate a lightweight MobileNetV3 font classification subnetwork, share underlying features with the detection backbone, and design a joint loss function: including detection loss and classification loss. Output text regions with font type labels.

[0153] (1) Lightweight MobileNetV3 font classification subnetwork:

[0154] The lightweight MobileNetV3 font classification subnetwork receives low-dimensional feature maps from the backbone network and the original text image as input. Based on the MobileNetV3 network and through rational network design, this module effectively extracts font information from text regions and accurately classifies them. The lightweight MobileNetV3 font classification subnetwork consists of five main stages: MobileNetV3 Bottleneck feature compression, nonlinear activation, Dropout overfitting prevention, global average pooling and a fully connected classification layer, and Softmax probability output.

[0155] (1.1) MobileNetV3 Bottleneck feature compression:

[0156] When building the font classification model, we first used the MobileNetV3 architecture to extract image features. This architecture, by employing depthwise separable convolutions and a Bottleneck structure, significantly reduced computational overhead while maintaining model performance.

[0157] The Bottleneck layer is a core component of MobileNetV3. It compresses the input feature dimensions into a smaller number of channels before performing expansion and convolution calculations. This approach significantly reduces the amount of computation and improves network efficiency. In the font classification task, the Bottleneck layer can effectively extract important features from the font image while avoiding the interference of redundant information, providing a concise and efficient feature representation for subsequent classification tasks. The specific steps are as follows:

[0158] The input feature map is set to , Indicates the number of channels, and are height and width respectively. Convolution performs channel expansion:

[0159] (5),

[0160] in, is the feature map expanded by the first convolutional layer, , It is a commonly used activation function in MobileNetV3. Batch Normalization is used to normalize the input of the activation function to ensure the stability of the training process. is the number of intermediate channels after expansion, usually greater than .

[0161] Next is the depth-wise separable convolution, which performs independent convolution operations on each channel:

[0162] (6),

[0163] in, is the output feature map, represents the depthwise convolution operation, is the depthwise convolution kernel.

[0164] The channels are then weighted using the SE (Squeeze-and-Excitation) attention mechanism:

[0165] (7),

[0166] in, It is the channel descriptor extracted by the squeeze stage in the SE module.

[0167] After the Excitation stage and scaling, Convolution performs channel compression:

[0168] (8),

[0169] in, Represents the output feature map after processing by the SE module. is the weight matrix of the convolution kernel, which is used to process the feature map after squeeze and excitation. Perform convolution to adjust the number of channels. It is the bias term of the convolution operation. When the number of input and output channels is the same, a residual connection is added, and the final output feature map size is .

[0170] Through this stage, the model is able to capture the basic structural information in the image, such as the outline of the text, the thickness of the strokes, and the overall shape of the font, thus laying a solid foundation for the font recognition task.

[0171] (1.2) Nonlinear activation:

[0172] Activation functions play a crucial role in feature extraction. Typically, each layer of a convolutional neural network requires the introduction of nonlinear characteristics through activation functions, allowing the model to learn complex mapping relationships. For font classification tasks, nonlinear activation functions can help the model capture more complex differences in font styles.

[0173] This paper uses the ReLU activation function. The ReLU function effectively avoids the vanishing gradient problem and makes model training more efficient. Through nonlinear activation, the model not only learns simple linear features but also captures more complex and abstract font details, improving the accuracy of font classification. The use of the activation function enables the output of each layer to have richer expressive power, which is crucial for distinguishing subtle differences between different fonts.

[0174] In order to introduce nonlinear characteristics, the ReLU activation function is applied after the Bottleneck output:

[0175] (9),

[0176] Enhance the nonlinear fitting ability of the model, alleviate the gradient vanishing problem, retain important positive information and filter out invalid features.

[0177] (1.3) Dropout anti-overfitting mechanism:

[0178] Overfitting is a common problem in deep neural network training, especially when training data is limited or the model is overly complex. Overfitting can cause the model to perform well on the training set, but exhibit large errors on the test set or new samples. Therefore, it is crucial to employ appropriate regularization methods to prevent overfitting.

[0179] Dropout is an effective anti-overfitting mechanism. In the font classification model, it randomly "drops" some neurons in the neural network during the training process, thereby preventing the model from relying on certain specific features and forcing the model to learn more generalized feature representations. The Dropout operation is introduced to the ReLU output during the training phase. Dropout is performed according to a certain probability. To perform random masking:

[0180] (10),

[0181] in, Represents the probability of Dropout, which specifies the probability of a neuron being "discarded" during each training. Represents the feature map or neuron output after being processed by the ReLU activation function. This operation effectively prevents strong dependencies between neurons and enhances the model's adaptability to new samples.

[0182] By introducing the Dropout layer, the model can better learn the core features of font images during training, while avoiding learning features that are overly related to specific training samples, thereby improving the recognition stability under different font samples.

[0183] (1.4) Global average pooling and fully connected classification layer:

[0184] After feature extraction and nonlinear activation, the model needs to convert image features into classification labels. In this stage, the Global Average Pooling (GAP) layer is first used. Global average pooling compresses the spatial information of each feature map into a single value by calculating the average value of each feature map to obtain global information. This method has fewer parameters than traditional fully connected layers and can effectively reduce the risk of overfitting while preserving global information.

[0185] The output of global average pooling passes through a fully connected layer, which maps the extracted features to specific font categories. The fully connected layer weightedly combines the feature representations from different neurons and generates a corresponding score for each category. The core task of this stage is to help the model make accurate font predictions by learning the classification boundaries between samples.

[0186] The feature map after Dropout is input into the global average pooling module to compress the spatial dimension and obtain a vector of length 128:

[0187] (11),

[0188] Among them, the output feature vector It is the input of the subsequent classifier. Its advantages include reducing the number of parameters, providing global context information, and being robust under different input scales.

[0189] (1.5) Softmax probability output:

[0190] The model uses the Softmax activation function to process the output of the fully connected layer to generate the predicted probability of each font category. The Softmax function converts the output of the model into a probability distribution so that the output value of each category is between 0 and 1, and the sum of the probabilities of all categories is 1.

[0191] first, Connect to a two-layer fully connected network for classification processing. The first layer of full connection reduces the 128-dimensionality to 64-dimensionality:

[0192] (12),

[0193] in, is the weight matrix, size is 64×128, It is a bias term, which increases the nonlinear expression ability of the network through the ReLU activation function, and the second layer is mapped to the output category (Handwritten and printed):

[0194] (13),

[0195] This is the second fully connected layer, which converts the 64-dimensional vector output from the previous step Mapped to the final category space, the output dimension here is 2, that is, there are two categories (handwritten and printed), is the weight matrix, size 2×64, is the bias term.

[0196] Finally, it is converted into a probability distribution through the Softmax layer:

[0197] (14),

[0198] The two original scores (logits) in the previous step are converted into probability distributions through the Softmax activation function. Indicates the prediction for The probability of the class, this distribution satisfies the sum of the probabilities of all classes is 1, and the class with the largest probability in the output is the final prediction result of the model.

[0199] (2) The lightweight font classification subnetwork shares underlying features with the detection backbone:

[0200] The lightweight MobileNetV3 font classification subnetwork relies on the 1 / 4-scale feature maps extracted by the backbone network and simultaneously integrates features from an independent path of the original image input. These two subnetworks perform feature concatenation and convolutional fusion in the CONNECT module to construct a composite feature representation with richer semantic and stylistic information. The fused features are efficiently compressed by the MobileNetV3 module and processed through global average pooling and Softmax activation functions to output a probability distribution for the target font category.

[0201] (3) Multi-task joint learning framework network training:

[0202] In order to achieve the coordinated optimization of the two tasks of text detection and font classification, an end-to-end multi-task training mechanism was constructed to improve the performance of the overall model. The key to multi-task learning is to construct a reasonable loss function so that each subtask can be coordinated and optimized based on shared features. In this invention, the text detection task and the font classification task correspond to two sub-loss functions respectively, and their total loss is defined as a weighted combination, and the weighting method selects uncertainty weighting:

[0203] (15),

[0204] in, represents the text detection loss, represents the font classification loss, and are learnable parameters used to model the uncertainty between tasks. The model automatically adjusts the weights of these two parameters during training to improve the stability and robustness of multi-task learning. is a regular term, avoiding Infinitely small, keep optimization stable.

[0205] In the text detection part, the DBNet segmentation framework is used, and the loss function consists of binary image segmentation loss and bounding box geometry regression loss:

[0206] (16),

[0207] in, Is a regulating factor used to control the balance between geometric regression loss and binary image segmentation loss, geometric regression loss, Regress the bounding box geometry parameters of the text region, is the cross entropy loss based on pixel-level binary classification:

[0208] (17),

[0209] in, is the total number of pixels in the image, For the The true label of each pixel (0 or 1, indicating whether the pixel belongs to the text area). It is The predicted probability that a pixel belongs to the text area. It is used to constrain the geometric parameter difference between the predicted bounding box and the real box, often using Loss means:

[0210] (18),

[0211] in, is the number of text boxes, and Respectively represent The actual and predicted geometric parameters of a text box. The geometric parameters include the center coordinates, width, height, etc. of the box.

[0212] In the font classification task, the standard multi-class cross entropy loss is used as the optimization function of the classification objective:

[0213] (19),

[0214] in is the total number of font categories, is the true label of one-hot encoding, For prediction of category probability.

[0215] The combined application of the above strategies enables the model to achieve collaborative optimization of the two tasks, achieving a good balance between overall training efficiency and task performance.

[0216] 3. Printed text branch performance preservation: Based on the convolutional recurrent network of the traditional CRNN, printed text recognition is completed through convolutional layers, recurrent layers, and transcription layers.

[0217] (1) Convolutional layer extracts features:

[0218] The convolutional layer is the core component of CNN. It uses local connections and weight sharing to effectively reduce model complexity and computational complexity. Local connections reduce parameters, and weight sharing reduces input complexity, making CNN suitable for large-scale data processing. The convolution operation can be expressed as:

[0219] (20),

[0220] in, is the output feature at the time point, is the input sequence, is the convolution kernel, is the element index within the convolution kernel.

[0221] (2) Circular layer:

[0222] For the recognition of long text sequences, in addition to the spatial features extracted by the CNN, the LSTM is also required to build temporal features and enhance its contextual modeling capabilities. Its core consists of an input gate, a forget gate, and an output gate. The input gate controls the admission of new information, the forget gate determines which historical information should be discarded, and the output gate regulates the output of information in the current state. Through this gating mechanism, the LSTM effectively retains important information and gradually forgets irrelevant data, thereby alleviating the vanishing gradient problem and enabling the network to capture dependencies over longer time intervals.

[0223] Bidirectional LSTM enhances context modeling capabilities through bidirectional timeline processing. Its computation process can be decomposed into the collaboration of two independent LSTMs: forward and backward.

[0224] (2.1) Forward LSTM unit calculation (time step t From 1 to T ):

[0225] Input Gate:

[0226] (twenty one),

[0227] in, For input xt The weight matrix, is the previous hidden state The weight matrix is, Bias term. The function of the input gate is to control the current input information xt The Sigmoid activation function outputs a value of 0 to 1 to determine which new information needs to be stored in the memory unit.

[0228] Forget Gate:

[0229] (twenty two),

[0230] Its decision is based on historical memory Which information is discarded in the process? The closer the value is to 0, the more historical information is forgotten.

[0231] Candidate memory:

[0232] (twenty three),

[0233] It is used to generate a temporary memory state for the current time step, storing potentially useful information. The Tanh activation function is used to compress the value to [-1, 1] to avoid gradient explosion.

[0234] Memory Update:

[0235] (twenty four),

[0236] The function is to combine the results of the forget gate and the input gate to update the long-term memory state .

[0237] Output gate:

[0238] (25),

[0239] in, Adjust the output ratio of the current memory state, is the hidden state of the final output, which contains the semantic information of the current time step.

[0240] (2.1) Bidirectional state fusion:

[0241] The final output is the concatenation of the forward and backward hidden states:

[0242] (26),

[0243] in express Vector Sum Vector concatenation, the output dimension is 2×hidden layer size.

[0244] (3) Connection time classification sequence decoding:

[0245] After sequence modeling, the next step is sequence decoding, which converts the RNN output into a final text sequence. The Connectionist Temporal Classification (CTC) algorithm is used to convert the RNN output into a final label sequence. In CTC, the output of each time step does not need to be precisely aligned, but the model is allowed to learn alignment patterns in the sequence. It trains the model by mapping the label sequence to all alignments of the model output sequence and calculating the probabilities of these alignments. Compared to the traditional Softmax function, CTC is more suitable for handling situations where the output sequence does not completely match the target sequence. Therefore, when processing time series classification tasks such as license plate numbers, CTC can often better adapt to slight differences in fonts and sizes, while reducing the difficulty of labeling aligned samples.

[0246] definition πThe sequence path composed of the output sequence, given the input x In the case of , the output probability calculation formula is as follows:

[0247] (28),

[0248] in, It is a many-to-one mapping function, mainly used to remove duplicate and blank labels. Indicates all passing After the change is l Path π , the calculation formula for any path is as follows:

[0249] (29),

[0250] in, T represents the length of the input sequence, L is the output label.

[0251] The CTC loss function is a metric used to measure the difference between the model's predicted sequence and the true sequence in sequence-to-sequence learning tasks, especially when the input and output sequences are of different lengths. It automatically learns the mapping between the input sequence and the target sequence by considering all possible alignments without requiring explicit sequence alignment information. The calculation of CTC loss covers all possible paths, including repeated characters and blank labels, providing a flexible learning mechanism for the model. The CTC loss calculation formula is as follows:

[0252] (30),

[0253] in, D represents the set of all samples in the dataset, x represents the input feature sequence, y represents the corresponding true label sequence, p ( y|x ) represents a given input sequence x When , the output sequence is y probability.

[0254] 4. Special optimization of the handwriting branch: Linear variable convolution (LD-Conv) is used to replace traditional convolution, and dynamic kernel scaling guided by heat maps is used to adapt to the deformation of connected strokes. Multi-head attention (MHA) is embedded in the BiLSTM layer of the CRNN to establish a turning point spatiotemporal correlation model to complete the recognition of handwritten text.

[0255] (1) Linear variable convolution replaces CRNN traditional convolution:

[0256] LD-Conv is a new type of convolutional network specifically suited for handwritten text recognition. It can extract features from handwritten text images using a variety of convolution kernel parameters (e.g., 1, 2, 3, 4, 5, 6, 7, etc.). This flexible parameter selection enables LD-Conv to better adapt to the diversity of handwritten text than standard or deformable convolutions, particularly when processing complex handwritten text images, such as those with tilted characters and connected strokes. Furthermore, LD-Conv's design concept is extensible, allowing for customization of sampling shapes based on prior knowledge and automatic adaptation to changes in target shape through dynamic offsets, thereby enhancing the model's flexibility and adaptability.

[0257] To identify connected strokes in handwritten characters, a lightweight branch is inserted after the output of Stage 2 of the CNN. The goal is to further enhance feature representation through lightweight convolution, particularly improving the recognition of connected strokes, while reducing computational complexity and improving model efficiency and robustness.

[0258] Since the characteristics of connected strokes in different handwriting styles are different, a dynamic weight scaling mechanism is introduced to adjust the weight distribution of subsequent convolutional layers based on the mean value of the heat map. Specifically, the mean value of the heat map is first calculated:

[0259] (31),

[0260] Dynamically adjust the kernel weights of subsequent convolutional layers:

[0261] (32),

[0262] in, α is a learnable parameter with an initial value of 0.5 and is optimized through back propagation. The physical meaning of this mechanism is that when the degree of connected strokes in the image is high (i.e. When the stroke length is large, the network will enhance the weight of local dynamic convolution to adapt to the deformation area; when the stroke length is small, the basic convolution feature extraction capability is maintained.

[0263] In the handwriting recognition task, characters may be deformed due to the connection of strokes, including scaling, shearing and translation. To compensate for these changes, the linear deformation module (LDM) is introduced, which learns an affine transformation matrix :

[0264] (33),

[0265] in, , control Directional scaling to adapt to changes in character width; , Control the shear angle to simulate the tilt deformation caused by connected strokes; , Responsible for translation and compensating for local displacement errors caused by writing offset.

[0266] In order to avoid excessive deformation and destruction of the character topology, the affine matrix is ​​subjected to singular value decomposition (SVD) and its transformation matrix is ​​constrained. is a diagonal matrix, that is:

[0267] (34),

[0268] This constraint ensures that LDM mainly performs rigid transformations (rotation, translation, scaling), preventing the model from learning unreasonable distortions.

[0269] In order to efficiently generate affine matrix parameters, global average pooling (GAP) is used to extract the global information of the feature map to obtain the feature vector, and then the six-dimensional parameter vector is output through the fully connected layer. At the same time, in order to ensure the stability of training, the scaling factor is constrained during initialization , to avoid gradient instability caused by excessive deformation in the initial stage. To further enhance the network's attention to local deformation areas, a dynamic weight adjustment (DWA) mechanism is designed. The core idea is to use a dynamic mask To adjust the weight of the convolution kernel:

[0270] (35),

[0271] in, is the basic convolution kernel weight, learning common features (such as edges and corners); is a dynamic mask, Convolution and Sigmoid generation, the value of each position Indicates the degree of response enhancement of the convolution kernel at that location. : Enhance the convolution kernel response at this position, focus on the area of ​​continuous deformation, and improve the recognition accuracy. : Suppress background noise, such as paper wrinkles and ink contamination, to improve model robustness.

[0272] (2) Embedding Multi-Head Attention (MHA) in the BiLSTM layer of CRNN:

[0273] Based on the traditional BiLSTM network, a sequence modeling structure that integrates MHA is designed. This structure aims to enhance the model's ability to model complex relationships between characters and improve recognition robustness in handwritten text scenarios such as irregular character arrangement, deformation, and connected strokes.

[0274] The core idea of ​​MHA is to map the input features into multiple different representation subspaces and independently perform the self-attention mechanism in each subspace to capture the contextual dependencies between different positions in the input sequence. Each attention head can focus on learning the relationship between characters in different subspaces, so it has significant advantages in modeling complex character structures (such as nonlinear stroke connections in handwriting, deformation of upper and lower characters, etc.). The self-attention mechanism maps the input features into query ( Q ),key( K ) and value ( V ), and based on Q and K The dot product of Calculate the attention weight, V A weighted sum is performed to output the context representation of the current time step. The calculation process is as follows:

[0275] (36),

[0276] in, W Q 、 W K and W V is a learnable weight matrix, X are the input features. The attention score is then calculated by taking the dot product of the query matrix and the transpose of the key matrix and scaling it by the square root of the key vector dimension. In each head, its calculation formula is as follows:

[0277] (37),

[0278] in, is the dimension of the key vector.

[0279] The multi-head attention architecture passes input features to multiple attention heads in parallel, computes attention representations for each head, fuses the outputs of each head through a concatenation operation, and further integrates them through a linear transformation to produce a global representation that includes richer contextual dependencies. This mechanism significantly improves the model's ability to model interactions between characters, making it particularly suitable for handwritten text recognition tasks involving uneven character spacing or structural deformations.

[0280] During the model's sequence modeling phase, the initial feature sequence is first processed by a BiLSTM layer, which leverages its inherent forward and backward information flows to capture both forward and backward sequence information. The BiLSTM effectively learns the dependencies between time steps in the sequence through its update and reset gates. The output of the BiLSTM layer is then passed to the MHA. In this phase, each head reweights different parts of the sequence to capture dependencies across different dimensions. The model weights and fuses features across different dimensions, and the weighted features output by each head are aggregated to construct a comprehensive sequence representation. This comprehensive representation not only incorporates time series information but also complex interactions between characters within the sequence, such as uneven character spacing and alignment issues. This process combines the outputs of multiple heads through a concatenation operation, and further integrates this information through another linear layer to produce the final sequence feature representation.

[0281] 5. Based on the output of the labeled text detection area results by the improved text detection model, the handwritten text recognition results and printed text recognition results output by the handwritten text recognition model and the printed text recognition model are integrated to complete the detection and recognition of mixed font text.

[0282] (1) Text detection and font classification (preliminary stage):

[0283] Input original image I ∈R H×W×3 , output text area location (bounding box) and font category (handwritten / printed). Detection module (improved DBNet): enhance small text features through FPN+CA module and output probability map P ∈R H′×W′ and threshold map T ∈R H′×W′ Differentiable binarization (DB) formula:

[0284] ,

[0285] in k =50 controls the binarization slope, B is the binarization result.

[0286] Lightweight MobileNetV3 font classification subnetwork (MobileNetV3): For each detected text area Ri , extract ROI features and output font probability yi ∈[0, 1] (0: printed text, 1: handwritten text). Classification loss function (joint optimization):

[0287] ,

[0288] in, λ For the balance weight, take 0.6.

[0289] (2) Text region splitting (key bridging step):

[0290] Results sorted by font yi Assign text regions to corresponding recognition branches:

[0291] Diversion conditions:

[0292] .

[0293] (3) Dual-branch text recognition (post-stage):

[0294] Input cropped text area image R i , output recognition results S i .

[0295] Handwriting branch (enhanced CRNN): LD-Conv dynamic convolution: convolution kernel size according to heat map H Dynamic Adjustment: ,

[0296] in, α is the scaling factor, H local It is the thermal value of the connected area.

[0297] Multi-head Attention (MHA) Temporal Modeling:

[0298] ,

[0299] in Q , K , V They are obtained by linear transformation of the recurrent layer features.

[0300] Print branch: Use the standard CRNN architecture without dynamic convolution.

[0301] Recognition loss function (CTC+branch loss): ,

[0302] CTC loss calculates the probability of a character sequence:

[0303] ,

[0304] in π is the path, and B is the mapping from path to label.

[0305] (4) End-to-end process summary:

[0306] 1. Detection: DBNet locates text areas + MobileNetV3 classifies fonts;

[0307] 2. Diversion: Press yi Threshold assignment identification branch;

[0308] 3. Recognition: Handwritten text: LD-Conv + enhanced CRNN, printed text: standard recognition model;

[0309] 4. Output: Merge the results of the two branches to generate the final text content and position.

[0310] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for dynamic data processing and recognition of mixed font text, characterized in that: The steps include: S1: Collect the mixed font text image to be processed and input it into the enhanced DBNet text detection network to locate and detect the text area in the image. The enhanced DBNet text detection network embeds the coordinate attention mechanism in the FPN feature pyramid, establishes a dynamic association between spatial position and channel features through multi-scale feature fusion, and outputs a probability map and a threshold map; S2 integrates a lightweight MobileNetV3 font classification subnetwork, shares underlying features with the detection backbone, and designs a joint loss function to train the network in a multi-task joint learning framework, outputting text regions with font type labels. The joint loss function includes detection loss and classification loss. S3 selects the corresponding recognition branch according to the font type label. For the printed text area, it recognizes the printed text through the convolutional layer, recurrent layer, and transcription layer based on the traditional CRNN network; S4, for handwritten text areas, uses linear variable convolution instead of traditional convolution, and adapts to the deformation of connected strokes through dynamic kernel scaling guided by heat maps. A multi-head attention mechanism is embedded in the BiLSTM layer of the CRNN network to establish a turning point spatiotemporal correlation model for handwritten text recognition; S5, fusing the handwritten text recognition results with the printed text recognition results, and outputting the text detection area results and recognition results with labels; The process of adapting to joint deformation through dynamic kernel scaling guided by heat map includes: A lightweight branch is inserted after the Stage-2 output of the convolutional neural network to enhance the feature representation of the connected stroke area through lightweight convolution while reducing the computational complexity; By formula Calculate the mean of the heat map; Dynamically adjust the kernel weights of subsequent convolutional layers based on the mean value of the heat map: , in, It is a learnable parameter. When the degree of connected strokes is high, the weight of local dynamic convolution is enhanced to adapt to the deformation area. When the degree of connected strokes is low, the feature extraction ability of basic convolution is maintained. Learn an affine transformation matrix , compensating for scaling, shearing, and translation caused by connected strokes: (33)， in, , control Directional scaling to adapt to changes in character width; , Control the shear angle to simulate the tilt deformation caused by connected strokes; , Responsible for translation and compensating for local displacement errors caused by writing offset; For affine matrices Perform singular value decomposition and constrain its transformation matrix is a diagonal matrix: (34)； Using a dynamic mask To adjust the weight of the convolution kernel: (35)， in, is the basic convolution kernel weight, is a dynamic mask, Convolution and Sigmoid generation, the value of each position Indicates the degree of response enhancement of the convolution kernel at that location; when When , the convolution kernel response of the continuous deformation area is enhanced. Suppress background noise; The process of establishing a turning point spatiotemporal correlation model includes: A multi-head attention mechanism is embedded in the BiLSTM layer of the CRNN network to map the input features into multiple different representation subspaces. The self-attention mechanism is independently executed in each subspace to capture the contextual dependencies between different positions in the input sequence. The input features are mapped to query Q, key K and value V respectively through the formula: (36)， in, W Q 、 W K and W V is a learnable weight matrix, X is the input feature; Calculate attention weights: (37)， in, is the dimension of the key vector; Perform weighted summation on V to obtain the context representation of the current time step; The input features are passed into multiple attention heads in parallel, and the attention representations are calculated separately. The outputs of each head are fused through splicing operations and further integrated through linear transformation to obtain a global representation containing rich context dependencies.

2. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The enhanced DBNet text detection network also includes: The DBNet main network for text detection first performs deep feature extraction on the input image to generate a multi-scale feature map; then it fuses it with the FPN through upsampling operations to enhance the multi-scale semantic information; finally, it further predicts a probability map and a threshold map for the training phase, and the two are combined to obtain an approximate binary image result.

3. The method for dynamic data processing and recognition of mixed font text according to claim 2, characterized in that: The binary image result is achieved by an approximate step function, which combines the predicted probability map and the predicted threshold map to obtain an approximate binary image, making the binarization process differentiable and facilitating back-propagation training; The expression of the approximate step function is: ， in, represents an approximate binary image, is a factor, represents the predicted probability plot, Threshold map representing the predictions.

4. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The coordinate attention mechanism includes: Global information encoding performs global average pooling on the input feature map in the horizontal and vertical directions to generate two separate feature vectors: ， in, represents the input feature map, Indicates the number of channels, Indicates height, Indicates width; represents the horizontal attention feature, represents the vertical attention feature, is the channel index, is the position index in the width direction, is the position index in the height direction; Interactive transformation, using shared convolutional layers for channel interaction: ， in, Represents horizontal attention features The weight is The features obtained after 1D convolution and nonlinear activation, represents a nonlinear activation function, Indicates the effect on 1D convolution weight matrix; represents the horizontal attention feature calculated by the formula; Represents vertical attention features The weight is Features obtained after 1D convolution and nonlinear activation; Indicates the effect on 1D convolution weight matrix; represents the vertical attention feature calculated by the formula; Attention weight calculation, fusing the attention information in the horizontal and vertical directions, calculating the final attention weight, which is used to re-adjust the input feature map: ， in, represents the final attention weight, is the fusion weight, Represents the splicing operation of features, and Splicing in a certain dimension to fuse the horizontal and vertical attention information.

5. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The lightweight MobileNetV3 font classification subnetwork includes five stages: MobileNetV3 Bottleneck feature compression, nonlinear activation, Dropout anti-overfitting mechanism, global average pooling and fully connected classification layer, and Softmax probability output. In the MobileNetV3 Bottleneck feature compression stage, depthwise separable convolution and Bottleneck structure are used to extract features from the input feature map, and the SE attention mechanism is used to weight the channels to finally output the feature map. In the nonlinear activation stage, the ReLU activation function is applied after the output of the Bottleneck feature compression stage to enhance the nonlinear fitting ability of the model; in the Dropout anti-overfitting mechanism stage, the Dropout operation is introduced after the output of the nonlinear activation stage to randomly block some neurons to prevent the model from overfitting; in the global average pooling and fully connected classification layer stage, the feature map after Dropout is input into the global average pooling module to compress the spatial dimension and obtain the feature vector, which is then input into the fully connected layer for classification processing; in the Softmax probability output stage, the output of the fully connected layer is processed to generate the predicted probability of each font category.

6. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The lightweight MobileNetV3 font classification subnetwork receives the low-dimensional feature map from the backbone network and the original text image as input, and relies on the 1 / 4 scale feature map extracted by the backbone network. At the same time, it fuses an independent path feature from the original image input. The two are feature spliced ​​and convolutionally fused in the CONNECT module to construct a composite feature representation. The fused features are efficiently compressed by the MobileNetV3 module and processed through global average pooling and Softmax activation function to output the probability distribution of the target font category.

7. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The S3 includes the following steps: Extracting features of the input sequence through a convolutional layer that uses a local connection and weight sharing mechanism to perform convolution operations and generate output features; The features extracted by the convolutional layer are input into the recurrent layer for time series feature modeling. The recurrent layer uses an LSTM network, which includes an input gate, a forget gate, and an output gate. The gating mechanism retains important information and gradually forgets irrelevant data. The bidirectional LSTM structure is used to enhance the context modeling capability through the collaborative processing of forward LSTM units and backward LSTM units, and finally outputs the concatenation of the forward and backward hidden states. The output of the recurrent layer is sequence-decoded by a connection-based temporal classification algorithm, and the output of the RNN is converted into a final text sequence. The connection-based temporal classification algorithm trains the model by mapping the label sequence to all alignments of the model output sequence and calculating the probabilities of these alignments to achieve sequence-to-sequence learning tasks.

8. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The S4 comprises the following steps: Linear variable convolution is used to replace traditional convolution in convolutional neural networks. The linear variable convolution uses different numbers of convolution kernel parameters to extract features from handwritten text images. Inserting a lightweight branch after the Stage-2 output of the convolutional neural network to further enhance feature representation through lightweight convolution; Calculate the mean of the heat map and dynamically adjust the kernel weights of subsequent convolutional layers based on the mean of the heat map; Introducing the linear deformation module, learning the affine transformation matrix, and performing rigid transformation on characters; Performing singular value decomposition on the affine transformation matrix and constraining it to be a diagonal matrix to avoid excessive deformation; Furthermore, multi-head attention is embedded in the bidirectional long short-term memory layer of the convolutional recurrent neural network to establish a turning point spatiotemporal correlation model to capture the contextual dependencies between different positions in the input sequence.

9. A dynamic data processing and recognition device for mixed font text, characterized in that: include: The text region location detection module is used to collect mixed-font text images to be processed and input them into the enhanced DBNet text detection network to locate and detect text regions in the image. The enhanced DBNet text detection network embeds a coordinate attention mechanism in the FPN feature pyramid, establishes a dynamic association between spatial position and channel features through multi-scale feature fusion, and outputs a probability map and a threshold map. The text region output module integrates a lightweight MobileNetV3 font classification subnetwork, shares underlying features with the detection backbone, and designs a joint loss function to train the multi-task joint learning framework network and output text regions with font type labels. The joint loss function includes detection loss and classification loss. The printed text recognition module is used to select the corresponding recognition branch based on the font type label. For printed text areas, it uses the traditional CRNN network to recognize printed text through convolutional layers, recurrent layers, and transcription layers. The handwritten text recognition module uses linear variable convolution instead of traditional convolution for handwritten text areas. It adapts to deformation of connected strokes through dynamic kernel scaling guided by heat maps. It also embeds a multi-head attention mechanism in the BiLSTM layer of the CRNN network and establishes a turning point spatiotemporal correlation model to recognize handwritten text. The merging result output module is used to fuse the handwritten text recognition results with the printed text recognition results, and output the text detection area results and recognition results with labels; The process of adapting to joint deformation through dynamic kernel scaling guided by heat map includes: A lightweight branch is inserted after the Stage-2 output of the convolutional neural network to enhance the feature representation of the connected stroke area through lightweight convolution while reducing the computational complexity; By formula Calculate the mean of the heat map; Dynamically adjust the kernel weights of subsequent convolutional layers based on the mean value of the heat map: , in, It is a learnable parameter. When the degree of connected strokes is high, the weight of local dynamic convolution is enhanced to adapt to the deformation area. When the degree of connected strokes is low, the feature extraction ability of basic convolution is maintained. Learn an affine transformation matrix , compensating for scaling, shearing, and translation caused by connected strokes: (33)， in, , control Directional scaling to adapt to changes in character width; , Control the shear angle to simulate the tilt deformation caused by connected strokes; , Responsible for translation and compensating for local displacement errors caused by writing offset; For affine matrices Perform singular value decomposition and constrain its transformation matrix is a diagonal matrix: (34)； Using a dynamic mask To adjust the weight of the convolution kernel: (35)， in, is the basic convolution kernel weight, is a dynamic mask, Convolution and Sigmoid generation, the value of each position Indicates the degree of response enhancement of the convolution kernel at that location; when When , the convolution kernel response of the continuous deformation area is enhanced. Suppress background noise; The process of establishing a turning point spatiotemporal correlation model includes: A multi-head attention mechanism is embedded in the BiLSTM layer of the CRNN network to map the input features into multiple different representation subspaces. The self-attention mechanism is independently executed in each subspace to capture the contextual dependencies between different positions in the input sequence. The input features are mapped to query Q, key K and value V respectively through the formula: (36)， in, W Q 、 W K and W V is a learnable weight matrix, X is the input feature; Calculate attention weights: (37)， in, is the dimension of the key vector; Perform weighted summation on V to obtain the context representation of the current time step; The input features are passed into multiple attention heads in parallel, and the attention representations are calculated separately. The outputs of each head are fused through splicing operations and further integrated through linear transformation to obtain a global representation containing rich context dependencies.

Images