Dynamic data processing and identification method and device for mixed font text

Abstract

The invention discloses a dynamic data processing and recognition method and device for a mixed font text, and belongs to the technical field of optical character recognition, and the method comprises the following steps: collecting a mixed font text image to be processed, and inputting the mixed font text image into an enhanced DBNet detection network to position a text region; integrating a lightweight MobileNetV3 font classification sub-network, sharing bottom layer features with a detection trunk, training a multi-task network through a joint loss function, and outputting a text region with a font type label; then selecting identification branches according to the labels, and identifying the printed form by using the traditional CRNN; according to the handwritten form, traditional convolution is replaced by linear variable convolution, kernel dynamic scaling is guided by a thermodynamic diagram, and multi-head attention mechanism recognition is embedded in a BiLSTM layer; and finally, results are fused, and detection and identification results with labels are output. According to the method, the mixed font text data processing and recognition precision is improved, the adaptability and robustness of handwritten text recognition are enhanced, and various text recognition requirements are met.

Description

Technical Field

[0001] The present invention relates to a method and device for dynamic data processing and recognition of mixed-font texts, and is particularly applicable to the automatic recognition of printed and handwritten mixed texts in industrial scenarios, belonging to the technical field of optical character recognition (OCR). Background Art

[0002] Characters are a way and tool for humans to record and express information through symbols in the early days. Chinese characters are one of the oldest characters in the world and have profound significance as an important carrier of Chinese culture. With the development of technology, digital images exist in almost all aspects of life. For example, in the manufacturing process of manufacturing factories, most of the part production work orders, equipment inspection records, workpiece quality inspection reports, etc. are still stored in paper form, and a lot of printed and handwritten information is included. At present, to convert such paper data into electronic document form mainly relies on manual entry and management, resulting in low data processing efficiency, easy errors, and difficult information traceability. Therefore, the optical character recognition (OCR) technology came into being accordingly.

[0003] In the manufacturing scenario, the application prospect of optical character recognition technology is extensive. For example, in the part quality inspection link, it can automatically analyze paper work orders such as first-piece double-inspection records, improve inspection efficiency, and reduce human errors; in terms of equipment maintenance, it can quickly identify and convert paper equipment inspection records to optimize the operation and maintenance process; in production management, it can digitally store the content of paper records such as workpieces and bill of materials to improve the informatization management level. In addition, by combining OCR with industrial automation systems, functions such as intelligent auditing and anomaly detection can also be realized to improve the data processing efficiency and intelligent level of manufacturing. Therefore, research on the part work order recognition technology in manufacturing will greatly improve the digitization and automation level of production data.

[0004] In recent years, with the rapid development of deep learning technology, significant breakthroughs have been made in the field of text detection and recognition, becoming the mainstream research direction. Before the popularization of deep learning, traditional handwritten recognition methods mainly relied on steps such as image preprocessing, feature extraction, and classification recognition. Usually, it was necessary to locate the text area, denoise, correct the inclination, and then use handcrafted features such as histogram of oriented gradients combined with a classifier for recognition. However, traditional methods have problems of low detection accuracy and poor recognition robustness when facing complex scenarios such as the existence of mixed texts and inconsistent handwritten text styles in production work orders, and no end-to-end process is designed for the mixed data characteristics of industrial scenarios. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a method and device for dynamic data processing and recognition of mixed-font texts, which improve the data segmentation accuracy through multi-scale feature fusion and dynamic convolution technology, and realize end-to-end optimization of detection, classification, and recognition by combining multi-task collaborative training.

[0006] The technical solution adopted by the present invention to solve its technical problems is as follows: In a first aspect, a method for dynamic data processing and recognition of mixed-font texts provided by an embodiment of the present invention includes the following steps: S1, Collect a mixed-font text image to be processed and input it into an enhanced DBNet text detection network to locate and detect the text area in the image. The enhanced DBNet text detection network embeds a coordinate attention (CA) mechanism in the FPN feature pyramid, establishes a dynamic association between spatial position and channel features through multi-scale feature fusion, and outputs a probability map and a threshold map; S2, Integrate a lightweight MobileNetV3 font classification sub-network, share the underlying features with the detection backbone, and design a joint loss function to train a multi-task joint learning framework network, and output a text area with a font type label. The joint loss function includes a detection loss and a classification loss; S3, Select the corresponding recognition branch according to the font type label. For the printed text area, based on the traditional CRNN network, the printed text is recognized through a convolutional layer, a recurrent layer, and a transcription layer; S4, For the handwritten text area, use linear variable convolution to replace traditional convolution, adapt to the connected pen deformation through heatmap-guided kernel dynamic scaling, and embed a multi-head attention mechanism in the BiLSTM layer of the CRNN network to establish a turning point spatio-temporal association model to recognize the handwritten text; S5, Fuse the handwritten text recognition result and the printed text recognition result, and output the text detection area result and the recognition result with labels.

[0007] As a possible implementation manner of this embodiment, the enhanced DBNet text detection network further includes: The DBNet text detection main network. The DBNet text detection main network first extracts deep features from the input image to generate multi-scale feature maps; then fuses them with the FPN through an upsampling operation to enhance multi-scale semantic information; finally, further predicts a probability map and a threshold map for the training stage, and combines the two to obtain an approximate binary image result.

[0008] As a possible implementation manner of this embodiment, the binary image result is realized through an approximate step function. The approximate step function combines the predicted probability map and the predicted threshold map to obtain an approximate binary image, making the binarization process differentiable and facilitating backpropagation training; The expression of the approximate step function is as follows: , where, represents the approximate binary image, is a factor, represents the predicted probability map, represents the predicted threshold map.

[0009] As a possible implementation of this embodiment, the coordinate attention (CA) mechanism includes: Global information encoding, performing global average pooling on the input feature map in the horizontal and vertical directions respectively to generate two separate feature vectors: , where, represents the input feature map, represents the number of channels, represents the height, represents the width; represents the horizontal attention feature, represents the vertical attention feature, is the channel index, is the position index in the width direction, is the position index in the height direction; Interaction transformation, using a shared convolutional layer for inter-channel interaction: , where, represents the feature obtained after performing 1D convolution (with weight ) and non-linear activation ( ) on the horizontal attention feature , introducing non-linear transformation to enhance the model's expression ability; represents the non-linear activation function (such as ReLU, Sigmoid, etc.); represents the 1D convolutional weight matrix acting on , responsible for inter-channel information interaction and adjusting the horizontal direction features; represents the horizontal attention feature calculated by the formula, that is, the result of global pooling of the input feature map X along the height direction; represents the feature obtained after performing 1D convolution (with weight ) and non-linear activation ( ) on the vertical attention feature ; represents the 1D convolutional weight matrix acting on , responsible for inter-channel information interaction and adjusting the vertical direction features; denotes the vertical attention feature obtained through formula calculation, which is the result of global pooling of the input feature map X along the width direction; Attention weight calculation, which fuses the attention information in the horizontal and vertical directions to calculate the final attention weight for readjusting the input feature map: , where, denotes the finally obtained attention weight, is the fusion weight, denotes the concatenation operation of features, which concatenates and on a certain dimension to fuse the attention information in the horizontal and vertical directions.

[0010] As a possible implementation manner of this embodiment, the coordinate attention (CA) mechanism is embedded into the fused multi-scale feature map, covering three scales of 1 / 16, 1 / 8, and 1 / 4, so as to fully exploit the spatial information and channel dependence in different hierarchical features.

[0011] As a possible implementation manner of this embodiment, the lightweight MobileNetV3 font classification subnet includes five stages: MobileNetV3 Bottleneck feature compression, non-linear activation, Dropout overfitting prevention mechanism, global average pooling and fully connected classification layer, and Softmax probability output; in the MobileNetV3 Bottleneck feature compression stage, depthwise separable convolution and Bottleneck structure are used to extract features from the input feature map, and channel weighting is performed through the SE attention mechanism, and finally the feature map is output; in the non-linear activation stage, the ReLU activation function is applied after the output of the Bottleneck feature compression stage to enhance the non-linear fitting ability of the model; in the Dropout overfitting prevention mechanism stage, the Dropout operation is introduced after the output of the non-linear activation stage to randomly mask some neurons to prevent the model from overfitting; in the global average pooling and fully connected classification layer stage, the feature map after Dropout is input into the global average pooling module to compress the spatial dimension to obtain the feature vector, and then the feature vector is input into the fully connected layer for classification processing; in the Softmax probability output stage, the output of the fully connected layer is processed to generate the prediction probability of each font category.

[0012] As a possible implementation manner of this embodiment, the MobileNetV3 Bottleneck feature compression stage specifically includes the following steps: The input feature map is subjected to channel expansion through the first convolutional layer, then depthwise separable convolution is used to perform independent convolution operations on each channel. Next, channel weighting is performed through the SE attention mechanism, and finally channel compression is carried out through convolution to obtain the output feature map.

[0013] As a possible implementation of this embodiment, the SE attention mechanism includes a squeeze stage and an excitation stage. The squeeze stage extracts channel descriptors, and the excitation stage processes the channel descriptors to generate channel weights, and then multiplies the channel weights by the original feature map to weight the channels.

[0014] As a possible implementation of this embodiment, in the global average pooling and fully connected classification layer stage, the global average pooling module compresses the spatial information of the feature map after Dropout into a value to obtain a feature vector of length 128. Then, the feature vector is input into a two-layer fully connected network for classification processing. The first fully connected network reduces the 128-dimensional feature vector to 64 dimensions, and the second fully connected network maps the 64-dimensional feature vector to the final class space; the class space includes two classes: handwritten and printed; in the Softmax probability output stage, the Softmax function converts the output of the fully connected layer into a probability distribution, and the class with the highest probability in the output is the final prediction result of the model.

[0015] As a possible implementation of this embodiment, the lightweight MobileNetV3 font classification subnet receives the low-dimensional feature map from the backbone network and the original text image as inputs, and relies on the 1 / 4-scale feature map extracted by the backbone network. At the same time, it fuses an independent path feature from the original image input. The two are subjected to feature splicing and convolution fusion in the CONNECT module to construct a composite feature representation. The fused feature is efficiently compressed through the MobileNetV3 module and processed by the global average pooling and Softmax activation function to output the probability distribution of the target font class.

[0016] As a possible implementation of this embodiment, the joint loss function is defined in a weighted combination form, including text detection loss and font classification loss. The weighting method selects uncertainty weighting, and the model will automatically adjust the weights of the two during the training process to improve the stability and robustness of multi-task learning.

[0017] As a possible implementation of this embodiment, the text detection loss consists of a binary map segmentation loss and a bounding box geometric regression loss. The binary map segmentation loss is a cross-entropy loss based on pixel-level binary classification, and the bounding box geometric regression loss is used to constrain the geometric parameter differences between the predicted bounding box and the ground truth box.

[0018] As a possible implementation of this embodiment, the font classification loss uses the standard multi-class cross-entropy loss as the optimization function for the classification target.

[0019] As a possible implementation of this embodiment, the S3 includes the following steps: Extract the features of the input sequence through a convolutional layer. The convolutional layer performs convolutional operations using the local connection and weight sharing mechanism to generate output features. Input the features extracted by the convolutional layer into a recurrent layer for modeling time series features. The recurrent layer uses an LSTM network, which includes an input gate, a forget gate, and an output gate. It retains important information and gradually forgets irrelevant data through the gating mechanism, and adopts a bidirectional LSTM structure. The collaborative processing of the forward LSTM unit and the backward LSTM unit enhances the context modeling ability, and finally outputs the concatenated result of the forward and backward hidden states. Decode the output of the recurrent layer through the Connectionist Temporal Classification (CTC) algorithm to convert the output of the RNN into the final text sequence. The CTC algorithm trains the model by mapping the label sequence to all alignments of the model output sequence and calculating the probabilities of these alignments, thus achieving the sequence-to-sequence learning task.

[0020] As a possible implementation of this embodiment, the convolutional operation of the convolutional layer is expressed as: , where, is the output feature at time point, is the input sequence, is the convolutional kernel, is the element index within the convolutional kernel.

[0021] As a possible implementation of this embodiment, the S4 includes the following steps: Use Linear Deformable Convolution (LD-Conv) to replace the traditional convolution in the convolutional neural network. The LD-Conv uses different numbers of convolutional kernel parameters to extract features from the handwritten text image. Insert a lightweight branch after the output of Stage-2 of the convolutional neural network to further enhance the feature representation through lightweight convolution. Calculate the mean of the heatmap and dynamically adjust the kernel weights of the subsequent convolutional layers according to the mean of the heatmap. Introduce a Linear Deformation Module (LDM) to learn the affine transformation matrix and perform rigid transformation on the characters. Perform singular value decomposition on the affine transformation matrix and constrain it to a diagonal matrix to avoid excessive deformation. Moreover, multi-head attention (MHA) is embedded in the bidirectional long short-term memory (BiLSTM) layer of the convolutional recurrent neural network (CRNN) to establish a spatio-temporal correlation model for turning points, so as to capture the context dependencies between different positions in the input sequence.

[0022] As a possible implementation of this embodiment, the linear deformable convolution (LD-Conv) automatically adapts to the changes in the target shape through dynamic offsets, enhancing the flexibility and adaptability of the model. The dynamic weight scaling mechanism adjusts the weight distribution of subsequent convolutional layers by calculating the mean of the heatmap. When the degree of connected strokes in the image is high, it enhances the weights of local dynamic convolutions to adapt to the deformed regions; when the connected strokes are few, it maintains the basic convolutional feature extraction ability. The affine transformation matrix learned by the linear deformation module (LDM) includes a scaling factor for controlling the direction, a factor for controlling the shear angle, and a factor for responsible for translation, to compensate for the scaling, shearing, and translation changes of the characters. The multi-head attention (MHA) maps the input features into multiple different representation subspaces and independently executes the self-attention mechanism within each subspace to capture the context dependencies between different positions in the input sequence. The self-attention mechanism maps the input features into query (Q), key (K), and value (V) respectively, and calculates the attention weights based on the dot product of Q and K, and performs weighted summation on V to output the context representation at the current time step. The multi-head attention (MHA) structure parallelly inputs the input features into multiple attention heads, calculates the attention representations respectively, and fuses the outputs of each head through a concatenation operation and further integrates them through a linear transformation to obtain a global representation containing richer context dependencies.

[0023] As a possible implementation of this embodiment, the S5 includes the following steps: Input the original image I ∈ , denotes the height of the input feature map, denotes the width of the input feature map, 3 indicates that the image has three RGB channels. Enhance the small text features through an improved DBNet detection module and output the text region position and probability map P ∈ and the threshold map T ∈ , and obtain the binarization result using the differentiable binarization formula; meanwhile, use the lightweight MobileNetV3 font classification sub-network to extract ROI features for each detected text region and output the font probability yi ∈ [0, 1]; Allocate the text regions to the corresponding handwritten text recognition branch or printed text recognition branch according to the obtained font probability yi; For the text regions assigned to the handwritten text recognition branch, an enhanced CRNN model is used, which includes LD-Conv dynamic convolution and multi-head attention MHA temporal modeling, and outputs the handwritten text recognition results; for the text regions assigned to the printed text recognition branch, a standard CRNN architecture is used to output the printed text recognition results; The combined handwritten text recognition results and printed text recognition results are merged to generate the final text content and location.

[0024] As a possible implementation of this embodiment, the improved DBNet detection module includes an FPN+CA module for enhancing small text features. The differentiable binarization formula is: B = sign(P - T×k), where k = 50 controls the binarization slope. The loss function of the lightweight MobileNetV3 font classification sub-network is a joint optimization loss function, including classification loss and balance weight λ, and the value of λ is 0.6; the convolution kernel size of the LD-Conv dynamic convolution in the enhanced CRNN model is dynamically adjusted according to the connected stroke region heat value Hlocal, and the adjustment coefficient is α; the enhanced CRNN model also includes multi-head attention MHA temporal modeling, and Q, K, and V are linearly transformed from the recurrent layer features respectively; the recognition loss function is a combination of CTC loss and branch loss.

[0025] In a second aspect, a dynamic data processing and recognition device for mixed font text provided by an embodiment of the present invention includes: A text region positioning and detection module, configured to collect a mixed font text image to be processed and input it into an enhanced DBNet text detection network to perform positioning detection on the text regions in the image. The enhanced DBNet text detection network embeds a coordinate attention (CA) mechanism in the FPN feature pyramid, establishes a dynamic association between spatial positions and channel features through multi-scale feature fusion, and outputs a probability map and a threshold map; A text region output module, configured to integrate a lightweight MobileNetV3 font classification sub-network, share underlying features with the detection backbone, and design a joint loss function to perform multi-task joint learning framework network training, and output text regions with font type labels. The joint loss function includes detection loss and classification loss; A printed text recognition module, configured to select a corresponding recognition branch according to the font type label. For printed text regions, based on a traditional CRNN network, the printed text is recognized through a convolutional layer, a recurrent layer, and a transcription layer; A handwritten text recognition module is used for the handwritten text area. It replaces the traditional convolution with linear variable convolution, adapts to the connected stroke deformation through heatmap-guided kernel dynamic scaling, and embeds a multi-head attention mechanism in the BiLSTM layer of the CRNN network to establish a spatio-temporal correlation model at turning points for recognizing handwritten text; A merged result output module is used to fuse the handwritten text recognition result and the printed text recognition result, and output the result of the text detection area with labels and the recognition result.

[0026] The beneficial effects of the technical solution of the embodiment of the present invention are as follows: Through the enhanced DBNet text detection network and the coordinate attention mechanism, the present invention improves the positioning detection accuracy of the text area; adopts the lightweight MobileNetV3 font classification sub-network and the multi-task joint learning framework to reduce the model complexity and improve the recognition efficiency; designs recognition branches for printed text and handwritten text respectively to improve the recognition accuracy of mixed-font text; introduces linear variable convolution and multi-head attention mechanism to enhance the adaptability and robustness of handwritten text recognition. The present invention improves the data segmentation accuracy through multi-scale feature fusion and dynamic convolution technology, and realizes the end-to-end optimization of detection, classification and recognition through multi-task collaborative training.

[0027] The present invention improves the processing and recognition accuracy of mixed-font text data, enhances the adaptability and robustness of handwritten text recognition, meets the diverse text recognition requirements, and is especially suitable for the automatic recognition of printed and handwritten mixed text in industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flowchart of a dynamic data processing and recognition method for mixed-font text shown according to an exemplary embodiment; Figure 2 is a schematic structural diagram of a dynamic data processing and recognition device for mixed-font text shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] To more clearly illustrate the technical features of the solution of the present invention, the present invention will be elaborated in detail below through specific embodiments and in conjunction with its drawings.

[0030] As Figure 1 shown, a dynamic data processing and recognition method for mixed-font text provided by an embodiment of the present invention includes the following steps: S1. Collect the mixed-font text image to be processed, and input the collected text image into the enhanced DBNet text detection network to locate and detect the text area in the image. The enhanced DBNet text detection network embeds the Coordinate Attention (CA) mechanism in the FPN feature pyramid, establishes the dynamic association between spatial position and channel features through multi-scale feature fusion, and outputs the probability map and the threshold map. S2. Integrate the lightweight MobileNetV3 font classification sub-network, share the underlying features with the detection backbone, and design the joint loss function to train the multi-task joint learning framework network, and output the text area with the font type label. The joint loss function includes the detection loss and the classification loss. S3. Select the corresponding recognition branch according to the font type label. For the printed text area, based on the traditional CRNN network, the printed text is recognized through the convolutional layer, the recurrent layer, and the transcription layer. S4. For the handwritten text area, use the linear variable convolution to replace the traditional convolution, adapt to the cursive deformation through the heat map-guided kernel dynamic scaling, and embed the multi-head attention mechanism in the BiLSTM layer of the CRNN network to establish the turning point spatio-temporal association model to recognize the handwritten text. S5. Fuse the recognition results of the handwritten text and the printed text, and output the results of the text detection area with labels and the recognition results.

[0031] As a possible implementation of this embodiment, the enhanced DBNet text detection network further includes: The DBNet text detection main network. The DBNet text detection main network first extracts the deep features of the input image to generate the multi-scale feature map; then fuses with the FPN through the upsampling operation to enhance the multi-scale semantic information; finally, further predicts the probability map and the threshold map for the training stage, and combines the two to obtain an approximate binary image result.

[0032] As a possible implementation of this embodiment, the binary image result is realized by the approximate step function. The approximate step function combines the predicted probability map and the predicted threshold map to obtain an approximate binary image, making the binarization process differentiable and facilitating the backpropagation training. The expression of the approximate step function is: , where represents the approximate binary map, is a factor, represents the predicted probability map, represents the predicted threshold map.

[0033] As a possible implementation of this embodiment, the Coordinate Attention (CA) mechanism includes: Global information encoding, which performs global average pooling on the input feature map in the horizontal and vertical directions respectively to generate two separate feature vectors: , where, represents the input feature map, represents the number of channels, represents the height, represents the width; represents the horizontal attention feature, represents the vertical attention feature, is the channel index, is the position index in the width direction, is the position index in the height direction; Interactive transformation, using a shared convolutional layer for inter-channel interaction: , where, represents the feature obtained after performing 1D convolution (with weight ) and non-linear activation ( ) on the horizontal attention feature , introducing non-linear transformation to enhance the model's expressive ability; represents the non-linear activation function (such as ReLU, Sigmoid, etc.); represents the 1D convolution weight matrix acting on , responsible for inter-channel information interaction and adjusting the horizontal direction features; represents the horizontal attention feature obtained through formula calculation, that is, the result of global pooling of the input feature map X along the height direction; represents the feature obtained after performing 1D convolution (with weight ) and non-linear activation ( ) on the vertical attention feature ; represents the 1D convolution weight matrix acting on , responsible for inter-channel information interaction and adjusting the vertical direction features; represents the vertical attention feature obtained through formula calculation, that is, the result of global pooling of the input feature map X along the width direction; Attention weight calculation, fusing the attention information in the horizontal and vertical directions, calculating the final attention weight for re-adjusting the input feature map: , where, represents the finally obtained attention weight, is the fusion weight, represents the concatenation operation of features, which concatenates and along a certain dimension to fuse the attention information in the horizontal and vertical directions.

[0034] As a possible implementation of this embodiment, the Coordinate Attention (CA) mechanism is embedded in the fused multi-scale feature map, covering three scales of 1 / 16, 1 / 8, and 1 / 4, to fully exploit the spatial information and channel dependencies in different hierarchical features.

[0035] As a possible implementation of this embodiment, the lightweight MobileNetV3 font classification subnet includes five stages: MobileNetV3 Bottleneck feature compression, non-linear activation, Dropout overfitting prevention mechanism, global average pooling and fully connected classification layer, and Softmax probability output; in the MobileNetV3 Bottleneck feature compression stage, depthwise separable convolution and Bottleneck structure are used to extract features from the input feature map, and channel weighting is performed through the SE attention mechanism, and finally the output feature map is obtained; in the non-linear activation stage, the ReLU activation function is applied after the output of the Bottleneck feature compression stage to enhance the non-linear fitting ability of the model; in the Dropout overfitting prevention mechanism stage, the Dropout operation is introduced after the output of the non-linear activation stage to randomly mask some neurons to prevent the model from overfitting; in the global average pooling and fully connected classification layer stage, the feature map after Dropout is input into the global average pooling module to compress the spatial dimension to obtain the feature vector, and then the feature vector is input into the fully connected layer for classification processing; in the Softmax probability output stage, the output of the fully connected layer is processed to generate the prediction probability of each font category.

[0036] As a possible implementation of this embodiment, the MobileNetV3 Bottleneck feature compression stage specifically includes the following steps: The input feature map is channel-expanded through the first convolutional layer, then depthwise separable convolution is used to perform independent convolution operations on each channel, then channel weighting is performed through the SE attention mechanism, and finally channel compression is performed through convolution to obtain the output feature map.

[0037] As a possible implementation of this embodiment, the SE attention mechanism includes a squeeze stage and an excitation stage. The squeeze stage extracts the channel descriptor, the excitation stage processes the channel descriptor to generate the channel weight, and then the channel weight is multiplied by the original feature map to weight the channels.

[0038] As a possible implementation of this embodiment, in the global average pooling and fully connected classification layer stage, the global average pooling module compresses the spatial information of the feature map after Dropout into a value to obtain a feature vector with a length of 128, and then inputs the feature vector into a two-layer fully connected network for classification processing. The first fully connected network reduces the 128-dimensional feature vector to 64 dimensions, and the second fully connected network maps the 64-dimensional feature vector to the final class space; the class space includes two classes: handwritten and printed; in the Softmax probability output stage, the Softmax function converts the output of the fully connected layer into a probability distribution, and the class with the highest probability in the output is the final prediction result of the model.

[0039] As a possible implementation of this embodiment, the lightweight MobileNetV3 font classification subnet receives the low-dimensional feature map from the backbone network and the original text image as inputs, relies on the 1 / 4-scale feature map extracted by the backbone network, and at the same time fuses an independent path feature from the original image input. The two are subjected to feature concatenation and convolution fusion in the CONNECT module to construct a composite feature representation. The fused feature is efficiently compressed by the MobileNetV3 module and processed by the global average pooling and Softmax activation function to output the probability distribution of the target font class.

[0040] As a possible implementation of this embodiment, the joint loss function is defined in a weighted combination form, including text detection loss and font classification loss. The weighting method selects uncertainty weighting, and the model will automatically adjust the weights of the two during the training process to improve the stability and robustness of multi-task learning.

[0041] As a possible implementation of this embodiment, the text detection loss consists of a binary graph segmentation loss and a bounding box geometric regression loss. The binary graph segmentation loss is a cross-entropy loss based on pixel-level binary classification, and the bounding box geometric regression loss is used to constrain the geometric parameter differences between the predicted bounding box and the ground truth box.

[0042] As a possible implementation of this embodiment, the font classification loss uses the standard multi-class cross-entropy loss as the optimization function for the classification target.

[0043] As a possible implementation of this embodiment, the S3 includes the following steps: Extract the features of the input sequence through a convolutional layer. The convolutional layer performs convolutional operations using the local connection and weight sharing mechanisms to generate output features; The features extracted by the convolutional layer are input into the recurrent layer for modeling time series features. The recurrent layer uses an LSTM network, which includes an input gate, a forget gate, and an output gate. It retains important information through a gating mechanism and gradually forgets irrelevant data. Moreover, a bidirectional LSTM structure is adopted, and the context modeling ability is enhanced through the collaborative processing of the forward LSTM unit and the backward LSTM unit. Finally, the concatenation result of the forward and backward hidden states is output. The output of the recurrent layer is decoded through the Connectionist Temporal Classification (CTC) algorithm to convert the output of the RNN into the final text sequence. The CTC algorithm trains the model by mapping the label sequence to all alignments of the model output sequence and calculating the probabilities of these alignments, thus realizing the sequence-to-sequence learning task.

[0044] As a possible implementation manner of this embodiment, the convolution operation of the convolutional layer is expressed as: , where, is the output feature at time point, is the input sequence, is the convolution kernel, is the element index within the convolution kernel.

[0045] As a possible implementation manner of this embodiment, the S4 includes the following steps: Replace the traditional convolution in the convolutional neural network with Linear Dilated Convolution (LD-Conv), and the LD-Conv uses different numbers of convolution kernel parameters to extract features in the handwritten text image; Insert a lightweight branch after the output of Stage-2 of the convolutional neural network to further enhance the feature representation through lightweight convolution; Calculate the mean of the heatmap, and dynamically adjust the kernel weights of the subsequent convolutional layers according to the mean of the heatmap; Introduce a Linear Deformation Module (LDM) to learn the affine transformation matrix and perform a rigid transformation on the characters; Perform singular value decomposition on the affine transformation matrix and constrain it to a diagonal matrix to avoid excessive deformation; Moreover, embed the Multi-Head Attention (MHA) in the Bidirectional Long Short-Term Memory (BiLSTM) layer of the Convolutional Recurrent Neural Network (CRNN) to establish a turning point spatio-temporal correlation model to capture the context dependencies between different positions in the input sequence.

[0046] As a possible implementation of this embodiment, the linear variable convolution (LD-Conv) automatically adapts to the changes in the target shape through dynamic offsets, enhancing the flexibility and adaptability of the model. The dynamic weight scaling mechanism adjusts the weight distribution of subsequent convolutional layers by calculating the mean of the heatmap. When the degree of connected strokes in the image is high, it enhances the weights of local dynamic convolutions to adapt to the deformed regions; when the connected strokes are few, it maintains the basic convolutional feature extraction ability. The affine transformation matrix learned by the linear deformation module (LDM) includes a scaling factor for controlling the direction, a factor for controlling the shear angle, and a factor for responsible for translation to compensate for the scaling, shearing, and translation changes of the characters. The multi-head attention (MHA) maps the input features into multiple different representation subspaces and independently executes the self-attention mechanism within each subspace to capture the context dependencies between different positions in the input sequence. The self-attention mechanism maps the input features into queries (Q), keys (K), and values (V) respectively, calculates the attention weights based on the dot product of Q and K, and performs weighted summation on V to output the context representation at the current time step. The multi-head attention (MHA) structure parallelly inputs the input features into multiple attention heads, calculates the attention representations respectively, fuses the outputs of each head through a concatenation operation, and further integrates them through a linear transformation to obtain a global representation containing richer context dependencies.

[0047] As a possible implementation of this embodiment, the S5 includes the following steps: Input the original image I ∈ , represents the height of the input feature map, represents the width of the input feature map, 3 indicates that the image has three RGB channels. Enhance the small text features through an improved DBNet detection module and output the text region position and probability map P ∈ and the threshold map T ∈ , and obtain the binarization result using the differentiable binarization formula; meanwhile, use a lightweight MobileNetV3 font classification sub-network to extract ROI features for each detected text region and output the font probability yi ∈ [0, 1]; Allocate the text regions to the corresponding handwritten text recognition branch or printed text recognition branch according to the obtained font probability yi; For the text regions allocated to the handwritten text recognition branch, use an enhanced CRNN model, including LD-Conv dynamic convolution and multi-head attention MHA temporal modeling, to output the handwritten text recognition results; for the text regions allocated to the printed text recognition branch, use a standard CRNN architecture to output the printed text recognition results; Merge the obtained handwritten text recognition results and printed text recognition results to generate the final text content and position.

[0048] As a possible implementation of this embodiment, the improved DBNet detection module includes an FPN+CA module for enhancing small text features. The differentiable binarization formula is: B = sign(P - T×k), where k = 50 controls the binarization slope. The loss function of the lightweight MobileNetV3 font classification sub-network is a joint optimization loss function, including classification loss and balance weight λ, and the value of λ is 0.6; the convolution kernel size of the LD-Conv dynamic convolution in the enhanced CRNN model is dynamically adjusted according to the heat value Hlocal of the cursive region, and the adjustment coefficient is α; the enhanced CRNN model also includes multi-head attention MHA time series modeling, and Q, K, and V are linearly transformed from the recurrent layer features respectively; the recognition loss function is a combination of CTC loss and branch loss.

[0049] The present invention proposes a dynamic two-stage text data processing and recognition method for mixed fonts, mainly divided into two stages: 1) Detection stage: Input the text image into an improved algorithm framework based on differentiable binarization (DBNet), embed a semantic information enhancement module and a coordinate attention mechanism in the feature pyramid FPN, and significantly improve the small text localization accuracy. At the same time, design a lightweight MobileNetV3 font classification sub-network, and through joint optimization of the detection and classification loss functions, realize the collaborative output of text area localization and font type discrimination (printing / handwriting classification), providing key prior knowledge for subsequent recognition.

[0050] 2) Recognition stage: Innovatively construct a double-branch processing framework: For the printing branch, perform text recognition based on the traditional CRNN structure and output the recognition result; for the handwriting, use a CRNN variant enhanced by linear variable convolution (LD-Conv) and multi-head attention (MHA), and through heatmap-guided dynamic scaling of the convolution kernel and spatio-temporal context modeling, improve the recognition accuracy of the cursive region of the handwriting and output the recognition result. While maintaining a high standard character recognition rate, realize knowledge sharing of the double-branch recognition model through feature sharing technology, and improve the comprehensive recognition accuracy of mixed text.

[0051] As Figure 2 shown, a dynamic data processing and recognition device for mixed font text provided by an embodiment of the present invention includes: A text area positioning and measuring module for collecting a mixed font text image to be processed, and inputting the collected text image into an enhanced DBNet text detection network to perform positioning detection on the text area in the image. The enhanced DBNet text detection network embeds a coordinate attention mechanism in the FPN feature pyramid, establishes a dynamic association between spatial position and channel features through multi-scale feature fusion, and outputs a probability map and a threshold map; The text area output module is used to integrate a lightweight MobileNetV3 font classification sub-network, share underlying features with the detection backbone, design a joint loss function, train a multi-task joint learning framework network, and output text areas with font type labels. The joint loss function includes detection loss and classification loss; The printed text recognition module is used to select the corresponding recognition branch according to the font type label. For printed text areas, based on the traditional CRNN network, the printed text is recognized through convolutional layers, recurrent layers, and transcription layers; The handwritten text recognition module is used for handwritten text areas. It uses linear variable convolutions to replace traditional convolutions, adapts to cursive deformations through heatmap-guided kernel dynamic scaling, and embeds a multi-head attention mechanism in the BiLSTM layer of the CRNN network to establish a turning point spatio-temporal correlation model for recognizing handwritten text; The merged result output module is used to fuse the handwritten text recognition result and the printed text recognition result, and output the text detection area result and recognition result with labels.

[0052] The specific process of the present invention for dynamically processing and recognizing mixed font texts is as follows.

[0053] 1. Collect mixed font text images to be processed, and input the collected text images into an enhanced DBNet text detection network to locate and detect text areas in the images: Embed a coordinate attention (CA) mechanism in the FPN feature pyramid, establish dynamic associations between spatial positions and channel features through multi-scale feature fusion, and improve the sensitivity to small texts.

[0054] (1) The DBNet text detection main network: First, deep feature extraction is performed on the input image to generate multi-scale feature maps; then, upsampling operations are used to fuse with the FPN to enhance multi-scale semantic information. On this basis, a probability map and a threshold map for the training stage are further predicted, and the two are combined to obtain an approximate binary image result. To make the binarization process differentiable and facilitate backpropagation training, DBNet introduces an approximate step function, effectively achieving high-precision image segmentation. The specific form of this approximate function is: (1), where represents the approximate binary map, is a factor with a value of 50, represents the predicted probability map, Represents the predicted threshold map. The differentiable binarization formula has similar properties to the sigmoid function. After binarization, its value range is restricted to (0, 1). DBNet performs adaptive binarization on each pixel point, learns a threshold through training, and incorporates the binarization operation into the network. This enables each pixel in the image to have an adaptive threshold, better distinguishing the foreground and background in the object detection process, and thereby improving the efficiency of the DBNet algorithm.

[0055] (2) Embedding the Coordinate Attention (CA) mechanism in the main network: CA is an efficient and lightweight attention mechanism designed to enhance the model's sensitivity to the spatial distribution of objects, especially suitable for detecting dense and small-sized objects while maintaining a low computational overhead. Specifically, CA first performs global average pooling on the input feature map in the horizontal and vertical directions respectively, generating two separate feature vectors. These two vectors can capture the global information of the features in different directions, enabling the model to perceive the importance of objects in different spatial dimensions. This separate spatial encoding method is different from traditional global pooling as it retains more location information, allowing the model to not only focus on significant object regions but also perceive the position distribution of objects in different directions, improving its adaptability to complex scenes.

[0056] CA mainly consists of three parts: global information encoding, interactive transformation, and attention weight calculation. First, given the input feature map , CA adopts global pooling operations along the height ( ) and width ( ) directions respectively to calculate the horizontal attention feature and the vertical attention feature. The formulas are as follows: (2), where and respectively represent the global feature aggregation in the height direction and the width direction. Through this operation, the CA mechanism obtains two independent direction-aware features on each channel.

[0057] To reduce the computational load and simultaneously enhance the feature representation ability, CA uses a shared convolutional layer for inter-channel interaction: (3), where and are the weight matrices of the convolution, Represents a non - linear activation function. The role of this step is to enhance information in different directions, enabling the model to learn the distribution pattern of the target in space, thereby improving the accuracy of attention. CA calculates the final attention weights by fusing horizontal and vertical attention information. : (4), where, represents the concatenation operation of features, is the fusion weight, and the finally obtained attention weight will be used to readjust the input feature map , enabling the network to pay more attention to important target regions while suppressing irrelevant information.

[0058] (3) Embed the CA attention mechanism module into the feature pyramid FPN of the DBNet network, fuse spatial location information, and generate attention weights to adjust the feature map.

[0059] In terms of structural design, CA is cleverly embedded in the fused multi - scale feature maps, covering three scales of 1 / 16, 1 / 8, and 1 / 4, to fully exploit the spatial information and channel dependencies in features at different levels. Especially at the shallower scale (such as 1 / 4), this level retains rich spatial detail information, which is the key area for processing small texts and targets with blurred edges. These detail information is crucial for accurately locating text regions and their boundaries. By introducing the coordinate attention mechanism at multiple scales, the model can more precisely perceive the distribution, structure, and potential geometric relationships of text regions at different semantic levels. This multi - scale attention mechanism not only improves the attention ability of the detection branch to local regions but also enhances the model's response to subtle changes, thus ensuring higher detection accuracy. At the same time, CA provides a more discriminative input for the subsequent feature fusion of the CONNECT module, ensuring the efficient integration and accurate transmission of features at different scales. Through structured feature fusion, the overall robustness of the model is significantly improved, enabling more accurate and reliable detection results in complex scenarios.

[0060] II. Multi - task collaborative optimization implementation: Integrate a lightweight MobileNetV3 font classification sub - network, share the underlying features with the detection backbone, and design a joint loss function: including detection loss and classification loss. Output text regions with font type labels.

[0061] (1) Lightweight MobileNetV3 font classification sub - network: The lightweight MobileNetV3 font classification sub-network receives the low-dimensional feature map from the backbone network and the original text image as inputs. Based on the MobileNetV3 network and through reasonable network design, this module can effectively extract the font information in the text area and perform accurate classification. The lightweight MobileNetV3 font classification sub-network mainly includes the following five stages: MobileNetV3 Bottleneck feature compression, non-linear activation, Dropout overfitting prevention mechanism, global average pooling and fully-connected classification layer, and Softmax probability output.

[0062] (1.1) MobileNetV3 Bottleneck feature compression: In the construction of the font classification model, the MobileNetV3 architecture is first used to extract image features. This architecture significantly reduces the computational overhead while ensuring the model performance by adopting depthwise separable convolutions and Bottleneck structures.

[0063] The Bottleneck layer is the core component of MobileNetV3. It compresses the dimension of the input features into a smaller number of channels, then performs expansion and convolution calculations. This way significantly reduces the computational amount and improves the efficiency of the network. In the font classification task, the Bottleneck layer can effectively extract the important features in the font image, while avoiding the interference of redundant information, providing a concise and efficient feature representation for the subsequent classification task. The specific steps are as follows: The input feature map is set as , representing the number of channels, and are the height and width respectively. Channel expansion is performed through convolution: (5), where, is the feature map expanded by the first convolutional layer, , is a commonly used activation function in MobileNetV3, and BN batch normalization (Batch Normalization) is used to normalize the input of the activation function to ensure the stability of the training process, is the number of intermediate channels after expansion, usually greater than .

[0064] Next is the depthwise separable convolution, which performs independent convolution operations on each channel: (6), where, is the output feature map, represents the depth convolution operation, is the depth convolution kernel.

[0065] Subsequently, channel weighting is performed through the SE (Squeeze-and-Excitation) attention mechanism: (7), where, is the channel descriptor extracted in the squeeze stage of the SE module.

[0066] After passing through the Excitation stage and scaling, through convolution for channel compression: (8), where, represents the output feature map after being processed by the SE module. is the weight matrix of the convolution kernel, used to convolve the feature map after being processed by squeeze and excitation for adjusting the number of channels. is the bias term of the convolution operation. When the number of input and output channels is the same, a residual connection is added, and the final output feature map size is .

[0067] Through this stage, the model can capture the basic structural information in the image, such as the outline of the text, the thickness of the strokes, and the overall shape of the font, thus laying a solid foundation for the font recognition task.

[0068] (1.2)Nonlinear activation: In the feature extraction process, the activation function plays a crucial role. Usually, each layer of the convolutional neural network needs to introduce nonlinear characteristics through the activation function so that the model can learn complex mapping relationships. For the font classification task, the nonlinear activation function can help the model capture more complex font style differences.

[0069] The present invention uses the ReLU activation function. The ReLU function can effectively avoid the gradient vanishing problem and make the model training more efficient. Through nonlinear activation, the model can not only learn simple linear features, but also capture more complex and abstract font details, improving the accuracy of font classification. The use of the activation function enables the output of each layer to have richer expressive power, which is crucial for distinguishing the subtle differences between different fonts.

[0070] To introduce nonlinear characteristics, the ReLU activation function is applied after the Bottleneck output: (9), Enhance the non - linear fitting ability of the model, alleviate the vanishing gradient problem, retain important positive information and filter out invalid features.

[0071] (1.3)Dropout overfitting prevention mechanism: During the training process of deep neural networks, overfitting is a common problem, especially when the training data is scarce or the model is too complex. Overfitting will cause the model to perform well on the training set, but have large errors on the test set or new samples. Therefore, it is particularly important to adopt appropriate regularization methods to prevent overfitting.

[0072] Dropout is an effective overfitting prevention mechanism. In the font classification model, it randomly "drops" some neurons in the neural network during the training process, thus avoiding the model relying on certain specific features and forcing the model to learn more general feature representations. Introduce the Dropout operation on the ReLU output during the training phase. Dropout randomly masks according to a certain probability as follows: (10), where represents the probability of Dropout, specifying the probability that neurons are "dropped" during each training. represents the output of the feature map or neuron after being processed by the ReLU activation function. This operation effectively prevents strong dependencies between neurons and enhances the adaptability of the model to new samples.

[0073] By introducing the Dropout layer, the model can better learn the core features of font images during the training process, while avoiding learning features that are too related to specific training samples, improving the recognition stability under different font samples.

[0074] (1.4)Global average pooling and fully - connected classification layer: After feature extraction and non - linear activation, the model needs to convert the image features into classification labels. At this stage, first use the Global Average Pooling (GAP) layer. Global average pooling is an operation that compresses the spatial information of each feature map into a single value. It obtains global information by calculating the average value of each feature map. This method has fewer parameters compared to traditional fully - connected layers, can effectively reduce the risk of overfitting, and retain global information.

[0075] The output of the global average pooling passes through a fully connected layer, which is used to map the extracted feature maps to specific font categories. The fully connected layer combines the feature representations from different neurons through weighted summation and generates a corresponding score for each category. The core task at this stage is to help the model finally make accurate font predictions by learning the classification boundaries between samples.

[0076] The feature maps after Dropout are input into the global average pooling module to compress the spatial dimension and obtain a vector of length 128: (11), where the output feature vector is the input to the subsequent classifier. Its advantages include reducing the number of parameters, providing global context information, and being robust at different input scales.

[0077] (1.5)Softmax probability output: The model uses the Softmax activation function to process the output of the fully connected layer and generate the prediction probabilities for each font category. The Softmax function converts the output of the model into a probability distribution, such that the output value for each category is between 0 and 1, and the sum of the probabilities for all categories is 1.

[0078] First, Two fully connected networks are connected for classification processing. The first fully connected layer reduces the dimension from 128 to 64: (12), where is the weight matrix with a size of 64×128, is the bias term. After passing through the ReLU activation function, the non-linear expression ability of the network is increased. The second layer maps to the output categories (handwritten and printed): (13), This is the second fully connected layer, which maps the 64-dimensional vector output from the previous step to the final category space. Here, the output dimension is 2, indicating two categories (handwritten and printed), is the weight matrix with a size of 2×64, is the bias term.

[0079] Finally, it is converted into a probability distribution through the Softmax layer: (14), After passing through the Softmax activation function, the two original scores (logits) from the previous step are converted into a probability distribution, representing the prediction for the The probability of a class, where this distribution satisfies that the sum of probabilities for all classes is 1, and the class with the highest probability in the output is the final prediction result of the model.

[0080] (2) The lightweight font classification sub-network shares underlying features with the detection backbone: The lightweight MobileNetV3 font classification sub-network relies on the feature map at 1 / 4 scale extracted by the backbone network and simultaneously fuses an independent path feature from the original image input. The two are feature concatenated and convolutionally fused in the CONNECT module to construct a composite feature representation with more semantic and style information. The fused features are efficiently compressed via the MobileNetV3 module and processed through global average pooling and the Softmax activation function, etc., to output the probability distribution of the target font class.

[0081] (3) Training of the multi-task joint learning framework network: To achieve the collaborative optimization of the two tasks of text detection and font classification, an end-to-end multi-task training mechanism is constructed to improve the performance of the overall model. The key to multi-task learning lies in constructing a reasonable loss function so that each sub-task can be collaboratively optimized on the basis of sharing features. In the present invention, the text detection task and the font classification task respectively correspond to two sub-loss functions, and the total loss is defined in a weighted combination form, and the weighting method selects uncertainty weighting: (15), where, represents the text detection loss, represents the font classification loss, and are learnable parameters used to model the uncertainty between tasks. The model will automatically adjust the weights of the two during the training process to improve the stability and robustness of multi-task learning. is a regularization term to avoid being infinitesimal and keep the optimization stable.

[0082] In the text detection part, the segmentation framework of DBNet is adopted, and the loss function consists of the binary map segmentation loss and the bounding box geometric regression loss: (16), where, is a regulation factor used to control the balance between the geometric regression loss and the binary map segmentation loss, the geometric regression loss, regresses the geometric parameters of the bounding box of the text region, is the cross-entropy loss based on pixel-level binary classification: (17), where, is the total number of pixels in the image, is the true label of the th pixel (0 or 1, indicating whether the pixel belongs to the text region). is the predicted probability that the th pixel belongs to the text region. And is used to constrain the geometric parameter differences between the predicted bounding box and the true box, and often adopts loss representation: (18), where is the number of text boxes, and respectively represent the true geometric parameters and predicted geometric parameters of the th text box. Geometric parameters include the center coordinates, width, height, etc. of the box.

[0083] In the font classification task, the standard multi-class cross-entropy loss is adopted as the optimization function for the classification objective: (19), where is the total number of font categories, is the true label in one-hot encoding, is the probability of being predicted as class .

[0084] The combined application of the above strategies enables the model to achieve collaborative optimization of the two tasks and achieve a good balance between the overall training efficiency and task performance.

[0085] III. Printed text branch performance maintenance: Based on the traditional CRNN convolutional recurrent network, the recognition of printed text is completed through the convolutional layer, recurrent layer, and transcription layer.

[0086] (1) Convolutional layer feature extraction: The convolutional layer is the core component of the CNN. It adopts the local connection and weight sharing mechanism, effectively reducing the model complexity and computational amount. Local connection reduces parameters, and weight sharing reduces the input complexity, making the CNN suitable for large-scale data processing. The convolution operation can be expressed as: (20), where is the output feature at time point, is the input sequence, is the convolutional kernel, is the element index within the convolutional kernel.

[0087] (2) Recurrent layer: For the recognition task of long text sequences, in addition to the CNN responsible for extracting spatial features, an LSTM is also required to establish time series features and improve the context modeling ability. Its core consists of an input gate, a forget gate, and an output gate. The input gate controls the storage of new information, the forget gate determines which historical information needs to be discarded, and the output gate adjusts the information output of the current state. Through the gating mechanism, the LSTM can effectively retain important information and gradually forget irrelevant data, thus alleviating the vanishing gradient problem and enabling the network to capture dependencies over longer time intervals.

[0088] The bidirectional LSTM enhances the context modeling ability through bidirectional processing along the time axis. Its computational process can be decomposed into the collaboration of two independent LSTMs: forward and backward: (2.1)Forward LSTM cell calculation (time step t from 1 to T ): Input gate: (21), where, is the weight matrix of the input xt , is the weight matrix of the previous hidden state , and is the bias term. The role of the input gate is to control the storage weight of the current input information xt . By outputting a value between 0 and 1 through the Sigmoid activation function, it determines which new information needs to be stored in the memory cell.

[0089] Forget gate: (22), which determines which information to discard from the historical memory . The closer the value is to 0, the more historical information is forgotten.

[0090] Candidate memory: (23), whose role is to generate the temporary memory state at the current time step and store potentially useful information. The Tanh activation function is used to compress the value to [-1, 1] to avoid gradient explosion.

[0091] Memory update: (24), whose role is to combine the results of the forget gate and the input gate to update the long-term memory state .

[0092] Output gate: (25), Among them, Adjust the output ratio of the current memory state, which is the hidden state of the final output and contains the semantic information of the current time step.

[0093] (2.1) Bidirectional state fusion: The final output is the concatenation of the forward and backward hidden states: (26), where denotes vector sum vector concatenation, and the output dimension is 2 × hidden layer size.

[0094] (3) Connectionist temporal classification sequence decoding: After sequence modeling, the next step is sequence decoding, that is, converting the output of the RNN into the final text sequence. The connectionist temporal classification (CTC) algorithm is used to convert the output of the RNN into the final label sequence. In CTC, it is not necessary to perform precise alignment for the output of each time step, but the model is allowed to learn the alignment patterns in the sequence. It trains the model by mapping the label sequence to all alignment ways of the model output sequence and calculating the probabilities of these alignment ways. Compared with the traditional Softmax function, CTC is more suitable for dealing with the situation where the output sequence does not exactly match the target sequence. Therefore, when dealing with temporal classification tasks such as license plate numbers, CTC can often better adapt to the slight differences in font and size, while reducing the annotation difficulty of the alignment samples.

[0095] Define π as the sequence path composed of the output sequence. Given the input x , the probability calculation formula of the output is as follows: (28), where is a many-to-one mapping function, mainly to remove duplicate and blank labels. denotes all paths that are after passing through l π , and the calculation formula for any path is as follows: (29), where T denotes the length of the input sequence, L is the output label.

[0096] The CTC loss function is a metric used to measure the difference between the predicted sequence and the true sequence in sequence-to-sequence learning tasks, especially when the lengths of the input sequence and the output sequence are inconsistent. By considering all possible alignment ways, it automatically learns the mapping relationship between the input sequence and the target sequence without explicit sequence alignment information. The calculation of the CTC loss covers all possible paths including repeated characters and blank labels, thus providing a flexible learning mechanism for the model. The formula for calculating the CTC loss is as follows: (30), where, D represents the set of all samples in the dataset, x represents the input feature sequence, y represents the corresponding true label sequence, p ( y|x ) indicates that when the input sequence is x , the probability that the output sequence is y .

[0097] IV. Special Optimization for the Handwritten Branch: Replace the traditional convolution with linear deformable convolution (LD-Conv), and adapt to the connected stroke deformation through heatmap-guided kernel dynamic scaling; Embed multi-head attention (MHA) in the BiLSTM layer of CRNN to establish a spatio-temporal correlation model for turning points and complete the recognition of handwritten text.

[0098] (1) Replace the traditional convolution of CRNN with linear deformable convolution: LD-Conv is a new type of convolution operation, especially suitable for handwritten text recognition tasks. It can use different numbers of convolution kernel parameters (such as 1, 2, 3, 4, 5, 6, 7, etc.) to extract features from handwritten text images. This flexible parameter selection enables LD-Conv to better adapt to the diversity of handwritten styles than standard convolution or deformable convolution, especially when dealing with complex handwritten text images, such as character skew, connected strokes and other deformations. In addition, the design concept of LD-Conv is extensible, and specific sampling shapes can be customized according to prior knowledge and automatically adapt to the changes of the target shape through dynamic offsets, thus enhancing the flexibility and adaptability of the model.

[0099] To identify the connected stroke areas in handwritten characters, a lightweight branch is inserted after the output of Stage-2 of the CNN. The purpose is to further enhance the feature representation through lightweight convolution, especially to improve the recognition of connected stroke areas, while reducing the computational complexity, enhancing the model efficiency and robustness.

[0100] Since the connected stroke features of different handwritten styles are different, a dynamic weight scaling mechanism is introduced to adjust the weight distribution of the subsequent convolution layers with the heatmap mean. Specifically, first calculate the mean of the heatmap: (31), Dynamically adjust the kernel weights of subsequent convolutional layers: (32), Among them, α is a learnable parameter, with an initial value set to 0.5 and optimized through backpropagation. The physical meaning of this mechanism is that when the degree of connected strokes in the image is relatively high (i.e., is large), the network will enhance the weights of local dynamic convolutions to adapt to the deformed area; when there are fewer connected strokes, it will maintain the basic convolutional feature extraction ability.

[0101] In the handwritten character recognition task, characters may be deformed due to connected strokes, including scaling, shearing, and translation. To compensate for these changes, a linear deformation module (LDM) is introduced, which learns an affine transformation matrix : (33), Among them, , controls the scaling in the direction to adapt to the width change of the character; , controls the shear angle to simulate the inclined deformation caused by connected strokes; , is responsible for translation to compensate for the local displacement error caused by writing offset.

[0102] To avoid excessive deformation from destroying the character topology, the affine matrix is subjected to singular value decomposition (SVD) and its transformation matrix is constrained to be a diagonal matrix, that is: (34), This constraint ensures that the LDM mainly performs rigid transformations (rotation, translation, scaling) and avoids the model learning unreasonable distortion deformations.

[0103] To efficiently generate the affine matrix parameters, global average pooling (GAP) is used to extract the global information of the feature map to obtain a feature vector, and then a six-dimensional parameter vector is output through a fully connected layer. At the same time, to ensure training stability, the scaling factor is constrained during initialization to avoid excessive deformation in the initial stage leading to unstable gradients. To further enhance the network's attention to the local deformation area, a dynamic weight adjustment (DWA) mechanism is designed. Its core idea is to use a dynamic mask to adjust the weights of the convolutional kernel: (35), Among them, is the weight of the basic convolutional kernel, learning general features (such as edges and corners); is the dynamic mask, generated by convolution and Sigmoid. The value at each position represents the degree of response enhancement of the convolutional kernel at that location. When : enhance the response of the convolutional kernel at this position, focus on the connected-stroke deformation area, and improve the recognition accuracy. When : suppress background noise, such as paper wrinkles and ink stains, and improve the robustness of the model.

[0104] (2) Embed multi-head attention (MHA) in the BiLSTM layer of CRNN: Based on the traditional BiLSTM network, a sequence modeling structure integrating MHA is designed. This structure aims to enhance the model's ability to model complex relationships between characters and improve the recognition robustness in handwritten text scenarios with irregular character arrangements, deformations, and connected strokes.

[0105] The core idea of MHA is to map the input features into multiple different representation subspaces and independently execute the self-attention mechanism within each subspace, thereby capturing the context dependencies between different positions in the input sequence. Each attention head can focus on learning the relationships between characters in different subspaces, so it has significant advantages in modeling complex character structures (such as non-linear stroke connections and up-and-down character deformations in handwritten scripts). The self-attention mechanism maps the input features into queries ( Q ), keys ( K ), and values ( V ) respectively, and calculates the attention weights based on the dot product of Q and K , and performs weighted summation on V to output the context representation at the current time step. Its calculation process is as follows: (36), where, W Q , W K and W V are learnable weight matrices, X is the input feature. Then, calculate the attention scores by taking the dot product of the query matrix and the transpose of the key matrix and scaling by the square root of the key vector dimension. In each head, its calculation formula is as follows: (37), where, is the dimension of the key vector.

[0106] The multi-head attention structure parallelly inputs the input features into multiple attention heads, calculates the attention representations respectively, fuses the outputs of each head through a concatenation operation, and further integrates them through a linear transformation to obtain a global representation containing richer context dependencies. This mechanism significantly improves the model's ability to model the interactions between characters, especially suitable for handwritten text recognition tasks with uneven character spacing or structural deformations.

[0107] In the sequence modeling stage of the model, the initial feature sequence is first processed by the BiLSTM layer, which uses its inherent forward and backward information flows to process the sequence in order to capture the forward and backward sequence information. BiLSTM effectively learns the dependencies at each time step in the sequence through its update gate and reset gate. The output of the BiLSTM layer is then passed to the MHA. At this stage, each head re-weights different parts of the sequence to capture dependencies in different dimensions. The model weights and fuses the features in different dimensions, and the weighted features output by each head are aggregated to construct a comprehensive sequence representation. This comprehensive representation contains not only the time series information but also the complex interactions between the characters within the sequence, such as uneven character gaps and alignment issues. This process merges the multi-head outputs through a concatenation operation and further integrates the information through another linear layer to produce the final sequence feature representation.

[0108] V. Based on the improved text detection model, the text detection region results with labels are output, and the handwritten text recognition results and printed text recognition results output by the handwritten text recognition model and the printed text recognition model are fused to complete the detection and recognition of mixed-font text.

[0109] (1) Text Detection and Font Classification (Pre-stage): Input the original image I ∈R H×W×3 , and output the text region position (bounding box) and font category (handwritten / printed). Detection module (improved DBNet): Enhance the small text features through the FPN+CA module and output the probability map P ∈R H′×W′ and the threshold map T ∈R H′×W′ . Differentiable Binarization (DB) formula: , where k =50 controls the binarization slope, B is the binarization result.

[0110] Lightweight MobileNetV3 font classification sub-network (MobileNetV3): For each detected text region Ri , extract the ROI features and output the font probabilityyi ∈ [0, 1] (0: printed, 1: handwritten). Classification loss function (joint optimization): , where λ is the balancing weight, taking 0.6.

[0111] (2) Text area shunting (key bridging step): According to the font classification result yi allocate the text area to the corresponding recognition branch: Shunting condition: .

[0112] (3) Dual-branch text recognition (posterior stage): Input the cropped text area image R i , and output the recognition result S i .

[0113] Handwritten branch (enhanced CRNN): LD-Conv dynamic convolution: The convolution kernel size is dynamically adjusted according to the heat map H : , where α is the scaling factor, H local is the heat value of the cursive area.

[0114] Multi-head attention (MHA) temporal modeling: , where Q , K , V are respectively obtained by linear transformation of the recurrent layer features.

[0115] Printed branch: Use the standard CRNN architecture without dynamic convolution.

[0116] Recognition loss function (CTC + branch loss): , The CTC loss calculates the character sequence probability: , where π is the path, and B is the mapping from the path to the label.

[0117] (4) End-to-end process summary: 1. Detection: DBNet locates the text area + MobileNetV3 classifies the font; 2. Shunting: According to yiThreshold assignment recognition branch; 3. Recognition: Handwritten: LD-Conv + Enhanced CRNN, Printed: Standard recognition model; 4. Output: Combine the results of the two branches to generate the final text content and location.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that the specific implementation manners of the present invention can still be modified or equivalently replaced. Any modification or equivalent replacement without departing from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A method for dynamic data processing and recognition of mixed font text, characterized in that: The steps include: S1, collect the mixed font text image to be processed, and input the enhanced DBNet text detection network to locate and detect the text area in the image. The enhanced DBNet text detection network embeds the coordinate attention mechanism in the FPN feature pyramid, establishes the dynamic association between the spatial position and the channel feature through multi-scale feature fusion, and outputs the probability map and the threshold map; S2, integrates a lightweight MobileNetV3 font classification subnetwork, shares underlying features with the detection backbone, and designs a joint loss function to perform multi-task joint learning framework network training and output text areas with font type labels. The joint loss function includes detection loss and classification loss. S3, selects the corresponding recognition branch according to the font type label. For the printed text area, based on the traditional CRNN network, the printed text is recognized through the convolution layer, the recurrent layer, and the transcription layer; S4, for the handwritten text area, linear variable convolution is used to replace the traditional convolution, and the dynamic scaling of the kernel guided by the heat map is used to adapt to the deformation of the connected strokes. A multi-head attention mechanism is embedded in the BiLSTM layer of the CRNN network to establish a turning point spatiotemporal correlation model to recognize handwritten text; S5, integrating the handwritten text recognition result and the printed text recognition result, and outputting the text detection area result and the recognition result with labels.

2. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The enhanced DBNet text detection network also includes: DBNet text detection main network, the DBNet text detection main network The DBNet text detection main network first performs deep feature extraction on the input image to generate a multi-scale feature map; then fuses it with FPN through upsampling operation to enhance multi-scale semantic information; finally, further predicts the probability map and the threshold map for the training stage, and the two are combined to obtain an approximate binary image result.

3. The method for dynamic data processing and recognition of mixed font text according to claim 2, characterized in that: The binary image result is realized by an approximate step function, which combines the predicted probability map and the predicted threshold map to obtain an approximate binary image, making the binarization process differentiable and convenient for back-propagation training; The expression of the approximate step function is: ， in, represents an approximate binary image, is a factor, represents the predicted probability plot, Threshold map representing the predictions.

4. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The coordinate attention mechanism includes: Global information encoding, global average pooling is performed on the input feature map in the horizontal and vertical directions to generate two separate feature vectors: ， in, represents the input feature map, Indicates the number of channels, Indicates height, Indicates width; represents the horizontal attention feature, represents the vertical attention feature, is the channel index, is the position index in the width direction, is the position index in the height direction; Interactive transformation, using shared convolutional layers for channel interaction: ， in, Represents the horizontal attention feature The weight is The features obtained after 1D convolution and nonlinear activation, represents a nonlinear activation function, Indicates the effect on 1D convolution weight matrix; represents the horizontal attention feature calculated by the formula; Represents the vertical attention feature The weight is Features obtained after 1D convolution and nonlinear activation; Indicates the effect on 1D convolution weight matrix; represents the vertical attention feature calculated by the formula; Attention weight calculation, fusing the horizontal and vertical attention information, calculating the final attention weight, which is used to re-adjust the input feature map: ， in, represents the final attention weight, is the fusion weight, Represents the concatenation operation of features. and Splice in a certain dimension to fuse the attention information in the horizontal and vertical directions.

5. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The lightweight MobileNetV3 font classification subnet includes five stages: MobileNetV3 Bottleneck feature compression, nonlinear activation, Dropout anti-overfitting mechanism, global average pooling and fully connected classification layer, and Softmax probability output. In the MobileNetV3 Bottleneck feature compression stage, the input feature map is extracted using deep separable convolution and Bottleneck structure, and the channels are weighted through the SE attention mechanism to finally output the feature map. In the nonlinear activation stage, the ReLU activation function is applied after the output of the Bottleneck feature compression stage to enhance the nonlinear fitting ability of the model; in the Dropout anti-overfitting mechanism stage, the Dropout operation is introduced after the output of the nonlinear activation stage to randomly block some neurons to prevent the model from overfitting; in the global average pooling and fully connected classification layer stage, the feature map after Dropout is input into the global average pooling module to compress the spatial dimension and obtain the feature vector, which is then input into the fully connected layer for classification processing; in the Softmax probability output stage, the output of the fully connected layer is processed to generate the predicted probability of each font category.

6. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The lightweight MobileNetV3 font classification subnet receives the low-dimensional feature map from the backbone network and the original text image as input, and relies on the 1 / 4 scale feature map extracted by the backbone network, while fusing an independent path feature from the original image input. The two are feature spliced ​​and convolutionally fused in the CONNECT module to construct a composite feature representation. The fused features are efficiently compressed through the MobileNetV3 module, and processed through global average pooling and Softmax activation function to output the probability distribution of the target font category.

7. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The S3 comprises the following steps: Extracting features of the input sequence through a convolutional layer, wherein the convolutional layer performs convolution operations using a local connection and weight sharing mechanism to generate output features; The features extracted by the convolution layer are input into the circulation layer to model the time series features. The circulation layer adopts an LSTM network, which includes an input gate, a forget gate, and an output gate. The gating mechanism is used to retain important information and gradually forget irrelevant data. The bidirectional LSTM structure is used to enhance the context modeling capability through the collaborative processing of the forward LSTM unit and the reverse LSTM unit, and finally the splicing result of the forward and backward hidden states is output; The output of the recurrent layer is sequence-decoded by a connection temporal classification algorithm, and the output of the RNN is converted into a final text sequence. The connection temporal classification algorithm trains the model by mapping the label sequence to all alignments of the model output sequence and calculating the probabilities of these alignments to achieve sequence-to-sequence learning tasks.

8. The method for dynamic data processing and recognition of mixed font text according to claim 1, characterized in that: The S4 comprises the following steps: A linear variable convolution is used to replace the traditional convolution in the convolutional neural network, and the linear variable convolution uses different numbers of convolution kernel parameters to extract features in the handwritten text image; Inserting a lightweight branch after the Stage-2 output of the convolutional neural network to further enhance feature representation through lightweight convolution; Calculate the mean of the heat map, and dynamically adjust the kernel weights of subsequent convolutional layers according to the mean of the heat map; Introduce the linear deformation module, learn the affine transformation matrix, and perform rigid transformation on characters; Performing singular value decomposition on the affine transformation matrix and constraining it to be a diagonal matrix to avoid excessive deformation; Furthermore, multi-head attention is embedded in the bidirectional long short-term memory layer of the convolutional recurrent neural network to establish a turning point spatiotemporal correlation model to capture the contextual dependencies between different positions in the input sequence.

9. The method for dynamic data processing and recognition of mixed font text according to any one of claims 1 to 8, characterized in that: The S5 comprises the following steps: Input original image I∈ , represents the height of the input feature map, Represents the width of the input feature map, 3 means that the image has three channels, RGB, and the small text features are enhanced by improving the DBNet detection module and outputting the text area location and probability map P∈ and threshold graph T∈ , the differentiable binarization formula is used to obtain the binarization result; at the same time, the lightweight MobileNetV3 font classification subnetwork is used to extract ROI features for each detected text area and output the font probability yi∈[0,1]; According to the obtained font probability yi, the text area is assigned to the corresponding handwritten text recognition branch or printed text recognition branch; For the text area assigned to the handwritten text recognition branch, the enhanced CRNN model, including LD-Conv dynamic convolution and multi-head attention MHA temporal modeling, is used to output the handwritten text recognition results; for the text area assigned to the printed text recognition branch, the standard CRNN architecture is used to output the printed text recognition results; The handwritten text recognition results and the printed text recognition results are combined to generate the final text content and position.

10. A dynamic data processing and recognition device for mixed font text, characterized in that: include: The text area positioning module is used to collect the mixed font text image to be processed and input the enhanced DBNet text detection network to locate the text area in the image. The enhanced DBNet text detection network embeds the coordinate attention mechanism in the FPN feature pyramid, establishes the dynamic association between the spatial position and the channel feature through multi-scale feature fusion, and outputs the probability map and the threshold map; The text region output module is used to integrate the lightweight MobileNetV3 font classification subnetwork, share the underlying features with the detection backbone, and design a joint loss function to perform multi-task joint learning framework network training and output text regions with font type labels. The joint loss function includes detection loss and classification loss. The printed text recognition module is used to select the corresponding recognition branch according to the font type label. For the printed text area, the printed text is recognized through the convolution layer, the recurrent layer, and the transcription layer based on the traditional CRNN network; The handwritten text recognition module uses linear variable convolution to replace traditional convolution in the handwritten text area, adapts to the deformation of connected strokes through dynamic kernel scaling guided by heat map, embeds a multi-head attention mechanism in the BiLSTM layer of the CRNN network, and establishes a turning point spatiotemporal correlation model to recognize handwritten text; The merged result output module is used to fuse the handwritten text recognition results with the printed text recognition results, and output the text detection area results and recognition results with labels.

Images