Character recognition method for adapting handwritten form to printed form
By adding the LoRA module to the printed text model and fine-tuning the handwritten text data, the problem of poor recognition results for both printed and handwritten text was solved, achieving flexible adaptation and resource-saving text recognition results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2026-03-17
AI Technical Summary
In existing technologies, the difference in data distribution between printed and handwritten characters leads to a decrease in printed character recognition performance when training with handwritten character data, and existing methods cannot flexibly adapt to handwritten character recognition functions.
The LoRA method is used to fine-tune the printed model. By using low-rank decomposition and parameter freezing, the baseline model is trained using printed data. After adding the LoRA module, it is fine-tuned using handwritten data. Only the parameters of the LoRA module are updated, saving computational resources and time.
It achieves flexible adaptation in both printed and handwritten scenarios, saving computing resources and time while improving recognition performance.
Smart Images
Figure CN121686486A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of chip processing technology, and specifically relates to a method for character recognition that adapts to handwritten characters on printed text. Background Technology
[0002] In existing technologies, Optical Character Recognition (OCR) technology is widely used in various scenarios, such as street view text recognition, document text recognition, and a range of other applications. Currently, the common approach is still based on deep learning, using a large amount of training data to achieve a fit to the training data.
[0003] Existing training methods for handwritten scenarios typically involve mixing handwritten and printed data together for model training.
[0004] However, the existing technology has the following problems:
[0005] Because printed text data and handwritten text data have different distributions, and handwritten text is more complex and cannot be as neat as printed text, adding handwritten text data to printed text data for training will reduce the recognition effect of printed text. Furthermore, this training method cannot flexibly adapt to whether or not to use the handwritten text recognition function.
[0006] In addition, the technical terms in this field also include:
[0007] OCR: Optical Character Recognition is a technology designed to convert text content in an image into machine-encoded text data.
[0008] Low-rank decomposition: Low-rank decomposition is a mathematical method that aims to decompose a large, complex matrix into the product of two or more smaller, simpler, and lower-rank matrices.
[0009] LoRA (Low-Rank Adaptation of Large Language Models) is a technique used in the training of large language models to reduce training costs by decomposing a large matrix into the product of two smaller matrices.
[0010] CNN (Convolutional Neural Network) is a special type of neural network that is particularly well-suited for processing data with a grid structure, such as images, videos, and audio signals. Summary of the Invention
[0011] To address the aforementioned issues, this application aims to: draw inspiration from the LoRA method used in the field of large language models for fine-tuning vertical domain-specific data; firstly, train the model using printed data and freeze these parameters; then, fine-tune the model using handwritten data based on the printed model by employing a low-rank decomposition method; thereby enabling the model to be adapted to both handwritten and printed recognition scenarios.
[0012] Specifically, the present invention provides a method for character recognition adapted to handwritten characters on printed text, the method comprising the following steps:
[0013] S1. First, train the baseline model:
[0014] The model employs the ResNet18 network with a CNN architecture. It is trained using printed data and optimized using the Adam algorithm with an adaptive learning rate. Training for 30 epochs means optimizing the ResNet18 model 30 times using the printed data. These 30 optimizations can be adjusted based on the specific task and dataset to obtain the baseline model's weight parameters. These baseline model weight parameters are obtained through parameter updates during training. At the end of each epoch, the current network parameters become the model weights for that epoch. The final baseline model weights are the parameter values after 30 epochs of training, where all weights belong to the printed data.
[0015] S2. Based on the baseline model obtained in step S1, add operators to store the handwritten weights, i.e., add the weights of the newly added LoRA module:
[0016] LoRA modules are added to certain layers of the ResNet18 network. The LoRA modules are described below:
[0017] 1) For the selected network layer, retain the original weight matrix W;
[0018] 2) Introduce two new low-rank matrices A and B, where A has the shape [d, r] and B has the shape [r, d], where d is the dimension of the original weights and r is the selected rank, r: A = N(0, σ 2 ), where N represents a normal distribution, 0 is the mean of the distribution, and σ² is the variance of the distribution. This means that the elements of matrix A are randomly initialized from a normal distribution with a mean of 0 and a variance of σ². This initialization method helps to provide a good starting point for the LoRA module at the beginning of training.
[0019] 3) The new weight calculation method is: W+AB;
[0020] S3. Using handwritten digit data, train a model with the LoRA module added:
[0021] After the operators are added in step S2, training is performed based on the baseline model. The Adam optimizer is still used to optimize the network after the addition. Unlike step S1, where all weights of the model are updated when training the baseline model, here only the weights of the added operators, namely the weights of operators A and B, are updated during training, while the other weights are frozen. This will save model training resources and training time. The process is as follows:
[0022] 1. First, set all parameters of the baseline model to not require gradient calculation: require_grad = False;
[0023] 2. Then, only set the newly added LoRA module, i.e., matrices A and B, to require gradient calculation: require_grad = True;
[0024] 3. During backpropagation, only the parameters of the LoRA module are updated, while the original model parameters remain unchanged; this reduces the number of parameters that need to be updated, thus saving computational resources and time. 4. LoRA module weights are saved: the weight values of matrices A and B are saved; for each layer with LoRA added, the following is saved:
[0025] Matrix A has a shape of [d, r], and matrix B has a shape of [r, d], where d is the input dimension of the layer and r is the selected rank.
[0026] S4. Save the weights of all LoRA modules obtained during training, further including:
[0027] S4.1 After training, save the weights of the A and B operators of the LoRA module. These two operators are only the weights of the handwritten characters.
[0028] S4.2, depending on the scenario, if it is only a printed text scenario, the baseline model can be loaded for inference. If there is a handwritten text scenario, then it is only necessary to add the LoRA module weights saved in step S4.1 to the weights of the corresponding layers. For each layer with LoRA added, the final weight calculation is: W_final=W_base+AB, where W_base is the weight of the baseline model, and A and B are the weights of the LoRA module. This formula is a variation of the formula W+AB in S2, demonstrating the flexibility of adapting to different scenarios, namely handwritten and printed text scenarios.
[0029] The ResNet18 is a deep residual network, and its specific structure is as follows:
[0030] Input layer: Accepts 224x224x3 image input, 7x7 convolutional layer, 64 filters, 23x3 stride max pooling layer, 24 stride residual blocks, each containing 2 3x3 convolutional layers;
[0031] First residual block: [64,64] x 2
[0032] The second residual block: [128, 128] x 2
[0033] The third residual block: [256, 256] x 2
[0034] The fourth residual block: [512, 512] x 2
[0035] Global average pooling layer
[0036] Fully connected layer;
[0037] The output is the number of classes. The structure of each residual block consists of two 3x3 convolutional layers and a short-circuit connection; the short-circuit connection allows gradients to flow directly through the network, which helps to solve the gradient vanishing problem in deep networks.
[0038] The printed data in step S1 is obtained through data collection.
[0039] In step S1, the formula for Adam is as follows:
[0040] m_t=β_1*m_{t-1}+(1-β_1)*g_tv_t
[0041] =β_2*v_{t-1}+(1-β_2)*g_t^2m_t_hat
[0042] =m_t / (1-β_1^t)v_t_hat=v_t / (1-β_2^t)θ_t
[0043] =θ_{t-1}-α*m_t_hat / (sqrt(v_t_hat)+ε)
[0044] Where: m_t and v_t are the first and second moment estimates, respectively; β_1 and β_2 are the decay rates; α is the learning rate; ε is a small constant to prevent division by zero errors; and g_t is the current gradient.
[0045] In step S1, the ResNet18 model was optimized 30 times using these printed data. The optimization process specifically includes the following steps:
[0046] Forward propagation: Input data is passed through the network to obtain prediction results and calculate the loss;
[0047] Backpropagation: Calculate the loss function value using the predicted results and the true labels: Calculate the gradient parameter update of the loss function with respect to each parameter;
[0048] This process of updating network parameters using the Adam optimizer based on the calculated gradients is repeated 30 times, or 30 epochs, each time using all the training data.
[0049] In step S2, it is assumed that LoRA modules were added to layers [64,64], [128,128], [256,256], and [512,512] respectively; d in the LoRA module represents the dimension of the input weights, so d is 64, 128, 256, and 512 respectively; r represents the rank of the matrix, r: A=N(0,σ 2 );
[0050] Assuming the rank is 2, in the layer [64,64], the weight of A is [64,2] and the weight of B is [2,64]. The final input weight [64,64] is added together with the original weight (i.e., the weight that has not gone through LoRA) after passing through modules A and B.
[0051] In step S4.2, the scene determination can be achieved in the following ways:
[0052] 1. Users manually select the scene: printed text or handwritten text;
[0053] 2. Use a simple classifier to predict the type of the input image;
[0054] 3. If the characteristics of the current task are known, the task type can be automatically selected.
[0055] In step S4, the process of saving the weights of the LoRA module to achieve text recognition further includes:
[0056] 1. For printed text recognition: the original baseline model is used directly for inference, without needing to load the weights of the LoRA module;
[0057] 2. For handwriting recognition, including:
[0058] a. Load the baseline model;
[0059] b. Load the saved LoRA module weights, i.e., matrices A and B;
[0060] c. During inference, for layers with added LoRA, W+AB is used as the new weights for calculation;
[0061] d. Model where other layers remain unchanged;
[0062] 3. Output the final recognition result.
[0063] In addition to using the ResNet18 network, this method is also applicable to other CNN architectures, including MobileNet V1 / V2 / V3. This is because the method uses the LoRA module for adaptation and does not depend on a specific CNN architecture; any CNN network with a similar hierarchical structure can apply this method.
[0064] Therefore, the advantage of this application is:
[0065] 1. It can save a significant amount of computing resources and time. During training, only the parameters of the LoRA module need to be updated.
[0066] 2. It fully demonstrates the flexibility of the application layer, and can be adapted according to whether the scenario requires handwriting recognition. Attached Figure Description
[0067] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.
[0068] Figure 1 This is a schematic diagram of the specific structure of the resnet18 network in this application.
[0069] Figure 2 This is a schematic diagram of adding LoRA modules to certain layers of the ResNet18 network in this application.
[0070] Figure 3 This is a flowchart illustrating the method described in this application.
[0071] Figure 4 This is a pseudocode diagram of step S2 in the method of this application.
[0072] Figure 5 This is a pseudocode diagram of step S4 in the method of this application. Detailed Implementation
[0073] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.
[0074] This invention relates to a character recognition method that adapts to handwritten characters on printed text, belonging to deep learning and OCR text recognition for model fine-tuning, and is applicable to character recognition projects.
[0075] like Figure 3 As shown, the main implementation steps of the method are as follows:
[0076] Step S1. First, train the baseline model.
[0077] The model uses the ResNet18 network with a CNN architecture (see the detailed structure diagram of the ResNet18 network for details). Figure 1 As shown on the left, ResNet18 is a deep residual network, and its specific structure is as follows:
[0078] Input layer: Accepts 224x224x3 image input, 7x7 convolutional layer, 64 filters, 23x3 stride max pooling layer, 24 stride residual blocks, each containing 2 3x3 convolutional layers;
[0079] First residual block: [64,64] x 2
[0080] The second residual block: [128, 128] x 2
[0081] The third residual block: [256, 256] x 2
[0082] The fourth residual block: [512, 512] x 2
[0083] Global average pooling layer
[0084] Fully connected layer;
[0085] The output is the number of categories;
[0086] Each residual block consists of two 3x3 convolutional layers and a shortcut connection; the shortcut connection allows gradients to flow directly through the network, which helps to solve the gradient vanishing problem in deep networks.
[0087] Besides the ResNet18 network, other CNN architectures are also applicable to this method, including MobileNet V1 / V2 / V3, because the method uses the LoRA module for adaptation, without relying on a specific CNN architecture; any CNN network with a similar hierarchical structure can apply this method; the ResNet18 network is trained using printed data (here, printed data is collected data), and the training optimization algorithm used is Adam. Adam (Adaptive Moment Estimation) is an adaptive learning rate optimization algorithm, and its main formula is as follows:
[0088] m_t=β_1*m_{t-1}+(1-β_1)*g_tv_t
[0089] =β_2*v_{t-1}+(1-β_2)*g_t^2m_t_hat
[0090] = m_t / (1-β_1^t)v_t_hat
[0091] = v_t / (1-β_2^t)θ_t
[0092] =θ_{t-1}-α*m_t_hat / (sqrt(v_t_hat)+ε)
[0093] Where: m_t and v_t are the first and second moment estimates respectively; β_1 and β_2 are the decay rates; α is the learning rate; ε is a small constant to prevent division by zero errors; and g_t is the current gradient.
[0094] The ResNet18 model was trained for 30 epochs (i.e., optimized 30 times using these printed data) to obtain the weight parameters of the baseline model. These baseline model weight parameters were obtained through parameter updates during training. After each epoch, the current network parameters became the model weights for that epoch. The final baseline model weights were the parameter values after 30 epochs of training, where all weights belong to the printed data. The optimization process for 30 epochs specifically included the following steps:
[0095] Forward propagation: Input data is passed through the network to obtain prediction results and calculate the loss;
[0096] Backpropagation: Calculate the loss function value using the predicted results and the true labels: Calculate the gradient parameter update of the loss function with respect to each parameter;
[0097] This process of updating network parameters using the Adam optimizer based on the calculated gradients is repeated 30 times, or 30 epochs, each time using all the training data.
[0098] Step S2. Based on the baseline model obtained in Step S1, add operators to store handwritten weights: add LoRA modules (such as...) to some layers of the ResNet18 network. Figure 2 As shown in the image, the LoRA module is described below:
[0099] 1) For the selected network layer, assuming it is [64,64], [128,128], retain the original weight matrix W;
[0100] 2) Introduce two new low-rank matrices A and B, where A has the shape [d, r] and B has the shape [r, d], where d is the dimension of the original weights and r is the selected rank, which is 2 here;
[0101] 3) The new weight calculation method is: W+AB, pseudocode as follows: Figure 4 As shown;
[0102] Suppose LoRA modules are added to layers [64,64], [128,128], [256,256], and [512,512] (you can also try adding them to other layers). Generally, intermediate layers are chosen because they typically capture more complex features, such as... Figure 2 As shown on the right, in the LoRA module, d represents the dimension of the input weights, which are 64, 128, 256, and 512 respectively; r represents the rank of the matrix, r: A = N(0, σ 2 );
[0103] Assuming the rank is 2, taking the [64,64] level as an example, Figure 2 The weight of A on the right is [64,2], and the weight of B is [2,64]. The final input weight [64,64] is added to the original weight (i.e., the weight before LoRA) after passing through modules A and B.
[0104] In short, such as Figure 1 As shown, this LoRA module was added to layers [64,64], [128,128], [256,256], and [512,512] respectively (you can also try adding it to other layers, prioritizing layers with a large number of parameters). The added operator is... Figure 2 The right-hand side of the diagram shows that in the LoRA module, 'd' represents the dimension of the input weights, which are 64, 128, 256, and 512 respectively. 'r' represents the rank of the matrix, here set to 2. Taking the [64, 64] layer as an example... Figure 2 The weight of A on the right is [64,2], and the weight of B is [2,64]. The final input weight [64,64] is added to itself after passing through modules A and B.
[0105] Step S3. Train the model with the LoRA module added using handwritten data:
[0106] After the operator is added in step S2, training is performed based on the baseline model. The Adam optimizer is still used to optimize the network after the addition. Unlike step S1, where all weights of the model are updated during baseline model training, here only the weights of the added operator are updated during training. Figure 2 The weights of operators A and B on the right side are frozen, while other weights are frozen, such as... Figure 1 The weights on the left side of the model will be frozen, which will greatly save model training resources and training time. The process is as follows: 1. First, set all parameters of the baseline model to not require gradient calculation (requires_grad=False).
[0107] 2. Then set only the newly added LoRA module (matrices A and B) to require gradient calculation (requires_grad=True);
[0108] 3. During backpropagation, only the parameters of the LoRA module are updated, while the original model parameters remain unchanged; this greatly reduces the number of parameters that need to be updated, thus saving computational resources and time; 4. Saving LoRA module weights: This saves the weight values of matrices A and B. This is done for each layer with LoRA added.
[0109] Matrix A has a shape of [d, r].
[0110] Matrix B has a shape of [r, d].
[0111] Where d is the input dimension of the layer, and r is the selected rank (2 here);
[0112] Step S4. Save the weights of all LoRA modules obtained during training. The pseudocode is as follows: Figure 5 As shown: The process of saving the weights of the LoRA module to realize text recognition is as follows: 1. For printed text recognition: the original baseline model is used directly for inference, and the weights of the LoRA module do not need to be loaded;
[0113] 2. For handwritten character recognition:
[0114] a. Load the baseline model
[0115] b. Load the saved LoRA module weights (matrices A and B)
[0116] c. During inference, for layers with added LoRA, W+AB is used as the new weights for calculation.
[0117] d. Model where other layers remain unchanged;
[0118] 3. Output the final recognition result;
[0119] Further includes:
[0120] S4.1 After training, save the weights of the A and B operators of the LoRA module. These two operators are only the weights of the handwritten characters.
[0121] S4.2, based on the scenario, where scenario determination can be achieved in the following ways:
[0122] 1. Users manually select the scene (printed text / handwritten text);
[0123] 2. Use a simple classifier to predict the type of the input image;
[0124] 3. Automatically select based on task type (if the characteristics of the current task are known). In this embodiment, the user manually selects the scenario.
[0125] If it is only a printed text scenario, the baseline model can be loaded for inference. If there is a handwritten text scenario, it is only necessary to add the LoRA module weights saved in step S4.1 to the weights of the corresponding layers. For each layer with LoRA added, the final weight calculation is: W_final = W_base + AB, where W_base is the weight of the baseline model, and A and B are the weights of the LoRA module, which fully demonstrates the flexibility of adapting to different scenarios, namely handwritten and printed text scenarios.
[0126] In summary, the special feature of this method is that:
[0127] 1. LoRA was applied to the recognition of printed and handwritten text;
[0128] 2. An adaptable handwritten and printed character recognition scheme is proposed.
[0129] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method of recognizing characters of a handwritten script adapted to a printed script, characterized by, The method comprises the following steps: S1. First, train a baseline model: The model adopts a resnet18 network of a CNN architecture, the resnet18 network is trained by using printed data, and an adaptive learning rate optimization algorithm Adam is used as the training optimization algorithm. The resnet18 model is optimized for 30 epochs by using the printed data, that is, the resnet18 model is optimized for 30 times according to the specific task and the data set, and the weight parameters of the baseline model are obtained. The weight parameters of the baseline model are obtained through parameter updating in the training process; at the end of each epoch, the current network parameters are the model weights of the epoch; and the final baseline model weight is the parameter value after 30 epochs of training. The weights of the baseline model belong to printed bodies; S2. Based on the baseline model obtained in step S1, an operator for saving the weight of the handwritten body is added, that is, the weight of the newly added LoRA module: The LoRA module is added to part of the layers of the resnet18 network, and the LoRA module is described as follows: 1) For the selected network layer, the original weight matrix W is reserved; 2), two new low-rank matrices A and B are introduced, where A has a shape of [d, r] and B has a shape of [r, d], d is the dimension of the original weight, and r is the selected rank, r: A = N(0, σ 2 ), where N represents a normal distribution, 0 is the mean of the distribution, and σ2 is the variance of the distribution. This means that the elements of matrix A are randomly initialized from a normal distribution with a mean of 0 and a variance of σ2. This initialization method helps to provide a good starting point for the LoRA module at the beginning of training. 3) The new weight calculation method is W+AB; S3. Train the model added with the LoRA module by using handwritten data: After the operator in step S2 is added, the baseline model is trained, and the Adam optimizer is still used to optimize the added network. Different from updating all the weights of the model when training the baseline model in step S1, only the weights of the added operators, that is, the weights of the A and B operators, are updated in the training process, and other weights are frozen. In this way, the model training resources and the training time can be saved, and the process is as follows:
1. First, set all the parameters of the baseline model to not require gradient calculation: requires_grad=False; 2. Then, only set the newly added LoRA module, that is, the A and B matrices, to require gradient calculation: requires_grad=True; 3. During back propagation, only the parameters of the LoRA module are updated, and the original model parameters remain unchanged; in this way, the number of parameters to be updated can be reduced, thereby saving computing resources and time; 4. Save the LoRA module weight: save the weight values of the A and B matrices; for each layer added with the LoRA, save: The A matrix has a shape of [d, r], and the B matrix has a shape of [r, d], where d is the input dimension of the layer, and r is the selected rank; S4. Save the weights of all the LoRA modules obtained by training, further comprising: S4.
1. Save the weights of the A and B operators of the LoRA module after training. The two operators are only the weights of the handwritten body; S4.2, according to the scene, if it is only a printed body scene, load the base model inference, if there is a handwritten scene, only need to add the LoRA module weight saved in step S4.1 to the corresponding layer weight, for each layer added with LoRA, the final weight calculation is: W_final = W_base + AB, wherein W_base is the weight of the base model, A and B are the weights of the LoRA module, this formula is a variant of the formula W+AB in S2, which shows the flexibility of adapting to different scenes, namely handwritten and printed body scenes.
2. The method of claim 1, wherein the method further comprises: determining a font of the printed text; and determining a font of the handwritten text. The ResNet18 is a deep residual network, and its specific structure is as follows: an input layer: accepting 224x224x3 image input, a 7x7 convolutional layer, 64 filters, a step of 23x3, a maximum pooling layer, a step of 24, a residual block containing 2 3x3 convolutional layers; The first residual block: [64, 64]x2 The second residual block: [128, 128]x2 The third residual block: [256, 256]x2 The fourth residual block: [512, 512]x2 Global average pooling layer Fully connected layer The output is the number of categories The structure of each residual block includes two 3x3 convolutional layers and a shortcut connection; the shortcut connection allows the gradient to flow directly through the network, which helps to solve the gradient vanishing problem in deep networks.
3. The method of claim 1, wherein the step of converting the printed text into the handwritten text comprises the steps of: converting the printed text into the handwritten text by using a handwriting font; and converting the converted handwritten text into the handwritten text by using a handwriting recognition engine. The printed body data in step S1 is obtained by collection.
4. The method of claim 1, wherein the method further comprises: determining a font of the printed text; and determining a font of the handwritten text. In step S1, the formula of Adam is as follows: m_t = beta_1 * m_{t-1} + (1-beta_1) * g_t v_t = beta_2 * v_{t-1} + (1-beta_2) * g_t^2 m_t_hat = m_t / (1-beta_1^t) v_t_hat = v_t / (1-beta_2^t) theta_t = theta_{t-1} - alpha * m_t_hat / (sqrt(v_t_hat) + epsilon) Where: m_t, v_t are first and second moment estimates respectively; beta_1, beta_2 are decay rates; alpha is the learning rate; epsilon is a small constant to prevent division by zero error; g_t is the current gradient.
5. The method of claim 1, wherein the step of converting the printed text into the handwritten text comprises the steps of: converting the printed text into the handwritten text by using a handwriting font; and converting the converted handwritten text into the handwritten text by using a handwriting recognition engine. In step S1, the resnet18 model is optimized 30 times using these printed body data, and the optimization process specifically includes the following steps: Forward propagation: input data passes through the network to get the predicted result and calculate the loss; Using the predicted result and the real label to calculate the loss function value, back propagation: calculating the gradient of the loss function for each parameter, parameter update; Using the Adam optimizer to update the network parameters according to the calculated gradient, this process is repeated 30 times, i.e. 30 epochs, and the entire training data is used each time.
6. The method of claim 1, wherein the method further comprises: determining a font of the printed text; and determining a font of the handwritten text. In the step S2, it is assumed that LoRA modules are added at [64, 64], [128, 128], [256, 256], and [512, 512] layers respectively; d in the LoRA module represents the dimension of the input weight, and d is 64, 128, 256, and 512 respectively; r represents the rank of the matrix, r: A = N(0, σ 2 ). Assuming that the rank is 2 here, in the [64, 64] layer, the weight of A is [64, 2], and the weight of B is [2, 64], and finally the input weight [64, 64] is added to the original weight, i.e. the weight without LoRA, after passing through the A and B modules.
7. The method of claim 1, wherein the method further comprises: determining a font of the printed text; and determining a font of the handwritten text. In step S4.2, the scene judgment can be realized by the following way: 1, User manually selects the scene: printed or handwritten; 2, Use a simple classifier to predict the type of input image; 3, If the characteristics of the current task are known, the task type can be automatically selected.
8. The method of claim 1, wherein the method further comprises: determining a font of the printed text; and determining a font of the handwritten text based on the determined font of the printed text. The process of saving the weight of the LoRA module to realize the character recognition in the step S4 further includes: 1, For printed body recognition: directly use the original baseline model for inference, without loading the weight of the LoRA module; 2, For handwritten body recognition, including: a. Load the baseline model; b. Load the saved LoRA module weight, that is, the A and B matrices; c. During inference, for the layer added with LoRA, use W+AB as the new weight for calculation; d. Other layers remain unchanged; 3, Output the final recognition result.
9. The method of claim 1, wherein the method further comprises: determining a font of the printed text; and determining a font of the handwritten text. In addition to using the resnet18 network, other CNN structure networks are also applicable to the method, including mobileNetV1 / V2 / V3, because the method uses the LoRA module for adaptation and does not depend on a specific CNN architecture; any CNN network with a similar hierarchical structure can apply the method.