Character recognition semantic model training method and system based on image information

Abstract

The invention relates to the technical field of character recognition semantic models, in particular to a character recognition semantic model training method and system based on image information. The method comprises the following steps: constructing a semantic score data set; training a semantic scoring model based on the semantic scoring data set; constructing a character recognition model, and extracting time sequence features of an input image through an image feature network; calculating a first loss function based on the difference between the time sequence characteristics and the target text, and performing micro decoding operation on the time sequence characteristics through the first loss function to determine a decoded text; inputting the decoded text into a semantic scoring model to obtain a semantic score, and calculating a second loss function based on the semantic score; and combining the first loss function and the second loss function to obtain a total loss function, and optimizing parameters of the character recognition model through back propagation. According to the method, model parameters are optimized through back propagation, and the problem that training is unstable due to the fact that a semantic loss function cannot be differentiated in a traditional method is solved.

Description

Technical Field

[0001] This application relates to the field of semantic model technology for character recognition, and in particular to a method and system for training a semantic model for character recognition based on image information. Background Technology

[0002] Handwritten character recognition plays a crucial role in invoice processing, form automation, and document digitization. With the widespread adoption of mobile terminals and embedded devices, higher demands are being placed on the real-time performance, robustness, and cross-language adaptability of recognition systems.

[0003] Current handwritten character recognition technologies primarily rely on image feature extraction and CTC decoding algorithms, but they suffer from the following drawbacks: low accuracy in recognizing similar-looking characters, and susceptibility to semantic breaks in entire sentences due to single-character errors; the training process relies solely on image feature loss, lacking semantic coherence constraints, resulting in model output text that does not conform to linguistic logic; and in some schemes that introduce semantic constraints, the semantic loss function is poorly designed (e.g., non-differentiable), making it impossible to optimize through backpropagation and affecting training stability. These issues lead to a high application cost for existing models in scenarios such as handwritten form entry and ancient text translation, requiring extensive manual proofreading. Summary of the Invention

[0004] This application provides a method and system for training a semantic model for text recognition based on image information to solve the above-mentioned problems.

[0005] Firstly, this application provides a method for training a semantic model for text recognition based on image information, the method comprising:

[0006] S1. Construct a semantic scoring dataset, which contains semantically coherent positive sample text and semantically incoherent negative sample text;

[0007] S2. Based on the semantic scoring dataset, train a semantic scoring model, which can output a semantic coherence score for the input text;

[0008] S3. Construct a text recognition model, which includes an image feature network. For an input image, extract temporal features through the image feature network.

[0009] S4. Based on the difference between the temporal features and the target text, calculate the first loss function, and perform differentiable decoding on the temporal features using the first loss function to determine the decoded text;

[0010] S5. Input the decoded text into the semantic scoring model to obtain a semantic score, and calculate a second loss function based on the semantic score;

[0011] S6. Combine the first loss function and the second loss function to obtain the total loss function, and optimize the parameters of the character recognition model through backpropagation.

[0012] By introducing a semantic scoring model as an external supervisor, the character recognition model learns not only the mapping relationship between image features and text during training but also forces the model to consider the semantic coherence of the output text. This helps the model avoid relying solely on visual similarity when recognizing similar-looking characters, instead making decisions based on contextual semantics, thereby reducing semantic breaks in the entire sentence caused by single-character errors. This method, through semantic loss constraints, makes the model more inclined to output characters that conform to the context, improving overall readability. Secondly, this method employs differentiable decoding operations, allowing the semantic loss to optimize model parameters through backpropagation, solving the training instability problem caused by the non-differentiability of the semantic loss function in traditional methods. This ensures the efficiency and convergence of the training process, avoiding gradient vanishing or exploding, and is particularly suitable for deep neural networks. Furthermore, the entire process is end-to-end, requiring no manual intervention, reducing application costs.

[0013] Optionally, in step S1, the positive sample text is derived from semantically coherent text; the negative sample text is generated by replacing characters at random positions in the positive sample text with similar-looking or homophonous characters.

[0014] For the negative sample text, its label is set based on the ratio of the number of replacement characters to the text length; the more replacement characters, the lower the label value.

[0015] By using the above technical solution, negative samples are generated by replacing similar-looking or homophonous characters, simulating common recognition errors in real-world scenarios. This enables the semantic scoring model to accurately distinguish between minor semantic deviations and severe semantic breaks, avoiding subjective bias. The labels are dynamically set based on the replacement ratio, rather than simple binary labels. This introduces a continuous semantic scoring mechanism, allowing the semantic scoring model to output more nuanced confidence values, guiding the model to gradually optimize its output, avoiding over-penalizing minor errors, and improving training stability.

[0016] Optionally, in step S2, the semantic scoring model is a Bert-type model, whose input is a text sequence and whose output is a semantic score normalized by a Sigmoid layer, with the score value between 0 and 1; the loss function used when training the semantic scoring model is the binary cross-entropy loss.

[0017] By employing the above technical solutions, the semantic scoring module is brought in line with the latest NLP technologies using BERT-like models. This facilitates the use of open-source pre-trained models, reduces training costs and time, and simplifies the training process by using binary cross-entropy loss. It avoids the complexity of multi-classification and enables the semantic scoring model to converge quickly. In text recognition training, this high-precision scoring provides a reliable foundation for the second loss function, making semantic constraints more effective.

[0018] Optionally, the image feature network includes a convolutional neural network and a bidirectional long short-term memory network, wherein the convolutional neural network is used to extract image features, and the bidirectional long short-term memory network is used to output temporal features.

[0019] Through the above technical solutions, the combination of convolution and bidirectional LSTM realizes end-to-end feature mapping from original pixels to character sequences, maintaining the dual expression of spatial local information and global sequence dependence. CNN can efficiently handle image deformation and noise, and is suitable for the diversity of handwritten characters. At the same time, temporal features lay the foundation for CTC loss calculation, enabling the model to handle indefinite-length sequences and adapt to more complex scenarios.

[0020] Optionally, in step S4, the first loss function is the CTC loss function, which is calculated based on the alignment path between the temporal features and the target text; the differentiable decoding operation is implemented using the Gumbel-Softmax decoding mechanism.

[0021] Through the above technical solutions, the sequence alignment capability of CTC loss and the gradient transitivity of differentiable decoding are realized, enabling the semantic scoring loss to influence the parameter update of the feature network in the same training iteration. This solves the problem of non-differentiability of semantic constraints in traditional methods. CTC loss does not require precise alignment, is suitable for handwritten characters with varying lengths, and reduces annotation costs. Gumbel-Softmax introduces noise and temperature parameters, approximating the argmax operation, making the gradient of the decoded text computable, thus seamlessly integrating semantic loss, avoiding training interruptions caused by non-differentiable functions, and improving stability and convergence speed.

[0022] Optionally, the Gumbel-Softmax decoding mechanism includes:

[0023] Add Gumbel noise to the time-series features;

[0024] The time-series features after adding noise are processed using the Softmax function and temperature parameters to obtain an approximate one-hot distribution;

[0025] The argmax operation is performed on the approximate one-hot distribution to obtain the decoding sequence, and the whitespace characters are removed from the decoding sequence to obtain the decoded text.

[0026] The above technical solution provides a complete gradient path while maintaining the near-discrete nature of the decoding process, enabling the subsequent semantic scoring loss to directly affect the learning of temporal features. It avoids local optima through randomness while maintaining differentiability, making the decoded text more diverse and stable. Gumbel noise simulates the sampling process and enhances the exploration capability, while the temperature parameter controls the degree of approximation, allowing fuzzy decision-making in the early stage of training and gradually hardening it in the later stage to improve accuracy.

[0027] Optionally, in step S5, the second loss function is calculated based on the semantic score, which is output by a pre-trained semantic scoring model on the decoded text.

[0028] By using the above technical solution, semantic scoring is transformed into a differentiable numerical loss, enabling semantic constraints to work together with CTC gradients in the same training step. This improves the model's learning of semantic integrity, allowing the character recognition model to directly benefit from semantic knowledge without the need for internal integration of complex NLP modules. Semantic scoring, as external supervision, provides an objective measure of semantic quality, making loss calculation more accurate.

[0029] Optionally, in step S6, the total loss function is a weighted sum of the first loss function and the second loss function, and the sum of the weight coefficients is 1;

[0030] The backpropagation optimization employs a phased training strategy:

[0031] In the first stage, the character recognition model is pre-trained using only the first loss function;

[0032] In the second stage, the character recognition model is jointly trained using the total loss function.

[0033] During the second phase of joint training, the parameters of the semantic scoring model are kept frozen, and only the parameters of the character recognition model are optimized.

[0034] Through the above technical solution, phased training achieves rapid convergence of visual features first, and then fine-grained correction through semantic constraints. This avoids gradient invariance problems caused by semantic scoring noise in the initial stage, ensuring gradual training and avoiding initial semantic interference. The phased strategy improves stability, and pre-training allows the model to master visual features first, and then introduces semantic constraints to prevent gradient conflicts.

[0035] Optionally, during the inference phase, for the input image, the text can be directly decoded and output using the character recognition model (CTC), without needing to call the semantic scoring model.

[0036] By omitting the forward computation of the semantic scoring model, the computational resource consumption and response time during inference are significantly reduced, meeting the requirements of real-time applications. In the inference stage, the character recognition model is directly used for CTC decoding without calling the semantic scoring model, which significantly reduces inference latency, meets real-time requirements, and simplifies the deployment process.

[0037] Secondly, this application provides a method and system for training a semantic model for text recognition based on image information, the system comprising:

[0038] The data construction module is used to construct a semantic scoring dataset, which contains semantically coherent positive sample text and semantically incoherent negative sample text.

[0039] The semantic training module is used to train a semantic scoring model based on the semantic scoring dataset. The semantic scoring model is able to output a semantic coherence score for the input text.

[0040] The feature analysis module is used to construct a text recognition model, which includes an image feature network. For an input image, temporal features are extracted through the image feature network.

[0041] The micro-decoding module is used to calculate a first loss function based on the difference between the temporal features and the target text, and to perform differentiable decoding on the temporal features through the first loss function to determine the decoded text;

[0042] The semantic scoring module is used to input the decoded text into the semantic scoring model to obtain a semantic score, and to calculate a second loss function based on the semantic score;

[0043] The backpropagation module is used to combine the first loss function and the second loss function to obtain the total loss function, and to optimize the parameters of the character recognition model through backpropagation.

[0044] Optionally, in the data construction module, the positive sample text is derived from semantically coherent text; the negative sample text is generated by replacing characters at random positions in the positive sample text with similar-looking or homophonous characters.

[0045] For the negative sample text, its label is set based on the ratio of the number of replacement characters to the text length; the more replacement characters, the lower the label value.

[0046] Optionally, in the semantic scoring module, the semantic scoring model is a Bert-type model, whose input is a text sequence and whose output is a semantic score normalized by a Sigmoid layer, with the score value between 0 and 1; the loss function used when training the semantic scoring model is the binary cross-entropy loss.

[0047] Optionally, in the feature analysis module, the image feature network includes a convolutional neural network and a bidirectional long short-term memory network. The convolutional neural network is used to extract image features, and the bidirectional long short-term memory network is used to output temporal features.

[0048] Optionally, in the micro-decoding module, the first loss function is the CTC loss function, which is calculated based on the alignment path between the temporal features and the target text; the differentiable decoding operation is implemented using the Gumbel-Softmax decoding mechanism.

[0049] Optionally, in the micro-decoding module, the Gumbel-Softmax decoding mechanism includes:

[0050] Add Gumbel noise to the time-series features;

[0051] The time-series features after adding noise are processed using the Softmax function and temperature parameters to obtain an approximate one-hot distribution;

[0052] The argmax operation is performed on the approximate one-hot distribution to obtain the decoding sequence, and the whitespace characters are removed from the decoding sequence to obtain the decoded text.

[0053] Optionally, in the semantic scoring module, the second loss function is calculated based on the semantic score, which is output by a pre-trained semantic scoring model on the decoded text.

[0054] Optionally, in the backpropagation module, the total loss function is a weighted sum of the first loss function and the second loss function, and the sum of the weight coefficients is 1;

[0055] The backpropagation optimization employs a phased training strategy:

[0056] In the first stage, the character recognition model is pre-trained using only the first loss function;

[0057] In the second stage, the character recognition model is jointly trained using the total loss function.

[0058] During the second phase of joint training, the parameters of the semantic scoring model are kept frozen, and only the parameters of the character recognition model are optimized.

[0059] Optionally, in the feature analysis module, for the input image, the text is directly decoded and output by the character recognition model (CTC), without calling the semantic scoring model. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a schematic diagram illustrating an application scenario provided in one embodiment of this application;

[0062] Figure 2 A flowchart illustrating a text recognition semantic model training method based on image information, provided in one embodiment of this application;

[0063] Figure 3 This is a schematic diagram of the structure of a text recognition semantic model training system based on image information, provided in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0065] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0066] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0067] Current handwritten character recognition technologies primarily rely on image feature extraction and CTC decoding algorithms, but they suffer from the following drawbacks: low accuracy in recognizing similar-looking characters, and susceptibility to semantic breaks in entire sentences due to single-character errors; the training process relies solely on image feature loss, lacking semantic coherence constraints, resulting in model output text that does not conform to linguistic logic; and in some schemes that introduce semantic constraints, the semantic loss function is poorly designed (e.g., non-differentiable), making it impossible to optimize through backpropagation and affecting training stability. These issues lead to a high application cost for existing models in scenarios such as handwritten form entry and ancient text translation, requiring extensive manual proofreading.

[0068] Based on this, this application provides a method and system for training a semantic model for character recognition based on image information. By introducing a semantic scoring model as an external supervisor, the character recognition model not only learns the mapping relationship between image features and text during training, but also forces the consideration of the semantic coherence of the output text. This helps the model avoid relying solely on visual similarity when recognizing similar-looking characters, but instead makes decisions based on contextual semantics, thereby reducing semantic breaks in the entire sentence caused by single-character errors. This method, through semantic loss constraints, makes the model more inclined to output characters that conform to the context, improving overall readability. Secondly, this method uses differentiable decoding operations (such as Gumbel-Softmax), allowing the semantic loss to optimize model parameters through backpropagation, solving the training instability problem caused by the non-differentiability of the semantic loss function in traditional methods. This ensures the efficiency and convergence of the training process, avoiding gradient vanishing or exploding, and is especially suitable for deep neural networks. In addition, the entire process is end-to-end, requiring no manual intervention, reducing application costs.

[0069] Figure 1 This application provides an illustration of an application scenario. When performing handwritten character recognition, the method provided in this application enables the semantic loss to optimize model parameters through backpropagation, thus solving the training instability problem caused by the non-differentiability of the semantic loss function in traditional methods.

[0070] Specifically, the method provided in this application can be applied to any server. The server interacts with a sample database to obtain sample texts provided by the database. By introducing a semantic scoring model as an external supervisor, the character recognition model not only learns the mapping relationship between image features and text during training, but also forces the semantic coherence of the output text to be considered. This helps the model avoid relying solely on visual similarity when recognizing similar-looking characters, but instead makes decisions based on contextual semantics, thereby reducing semantic breaks in the entire sentence caused by single-character errors.

[0071] For specific implementation details, please refer to the following examples.

[0072] Figure 2 This is a flowchart illustrating a method for training a semantic model for text recognition based on image information, provided as an embodiment of this application. The method of this embodiment can be applied to servers in the above-described scenarios. Figure 2 As shown, the method includes:

[0073] S1. Construct a semantic scoring dataset, which contains semantically coherent positive sample text and semantically incoherent negative sample text;

[0074] S2. Based on the semantic scoring dataset, train a semantic scoring model, which can output a semantic coherence score for the input text;

[0075] S3. Construct a text recognition model, which includes an image feature network. For an input image, extract temporal features through the image feature network.

[0076] S4. Based on the difference between the temporal features and the target text, calculate the first loss function, and perform differentiable decoding on the temporal features using the first loss function to determine the decoded text;

[0077] S5. Input the decoded text into the semantic scoring model to obtain a semantic score, and calculate a second loss function based on the semantic score;

[0078] S6. Combine the first loss function and the second loss function to obtain the total loss function, and optimize the parameters of the character recognition model through backpropagation.

[0079] Character recognition models are typically trained solely on image features and CTC loss, lacking constraints on the semantic coherence of the output text. This leads to semantic breaks when similar-looking characters are mistaken. By constructing a semantic scoring dataset containing positive and negative text, a model capable of providing a numerical score for text coherence can be trained. Introducing this score as an additional loss term into the training process adds semantic constraints to the gradient propagation path, achieving joint optimization of image features and semantic consistency.

[0080] The semantic scoring model collects semantically coherent positive sample text and generates negative sample text by replacing characters with similar shapes or homophones at random positions. The negative sample labels are set according to the proportion of replaced characters. The semantic scoring model adopts a BERT-like structure, taking a text sequence as input and outputting a coherence score normalized by a Sigmoid layer. The character recognition model extracts spatial features from the image using a convolutional neural network, then outputs temporal features via a bidirectional long short-term memory network. The difference between the temporal features and the target text is used to calculate the first loss function using CTC loss. Gumbel-Softmax differentiable decoding is applied to the temporal features to obtain the decoded text. The decoded text is then scored by the semantic scoring model, and a second loss function is calculated based on the score. The two loss functions are added together with preset weights to form the total loss function. During backpropagation, only the character recognition model parameters are updated.

[0081] By introducing a semantic scoring model as an external supervisor, the text recognition model not only learns the mapping relationship between image features and text during training, but also forces the consideration of the semantic coherence of the output text. This helps the model avoid relying solely on visual similarity when identifying characters with similar shapes, but instead makes decisions by combining context semantics, thus reducing the semantic breakage of the entire sentence caused by single-word errors. For example, in handwritten character recognition, the characters 'ji' and 'yi' are visually similar but have different semantics. Traditional models are prone to confusion, while this method, through semantic loss constraints, makes the model more inclined to output characters that conform to the context, improving the overall readability. Secondly, this method adopts differentiable decoding operations (such as Gumbel-Softmax), enabling the semantic loss to optimize model parameters through backpropagation, solving the problem of unstable training caused by the non-differentiability of the semantic loss function in traditional methods. This ensures the efficiency and convergence of the training process, avoiding gradient vanishing or explosion, especially suitable for deep neural networks. In addition, the entire process is end-to-end without manual intervention, reducing the application cost.

[0082] Through a multi-task learning framework, the combination of low-level image feature learning and high-level semantic understanding enhances the generalization ability of the model. Traditional text recognition models often overfit to the visual patterns in the training data, while this method, through semantic regularization, makes the model pay more attention to language logic, thus performing more robustly in cross-language or low-resource scenarios. For example, in multi-language handwritten character recognition, the model can adapt to the grammar structures of different languages and output coherent text. At the same time, this method is compatible with the existing CTC framework and is easy to integrate into industrial systems without introducing additional inference overhead because the semantic scoring model is only used in the training stage, and the optimized text recognition model is directly used in the inference stage, ensuring real-time performance. Through semantic integration and differentiable optimization, a balance between recognition accuracy and semantic quality is achieved, providing a more reliable solution for handwritten text recognition.

[0083] Replaceable or deformable implementation methods: The positive sample text can be sourced from corpora in different languages; character-level perturbations, synonym replacement, or random deletion can be used for negative sample generation; the semantic scoring model can be replaced with pre-trained language models such as RoBERTa and ALBERT; the convolutional layers of the image feature network can adopt structures such as ResNet and EfficientNet, and BiLSTM can be replaced with a Transformer encoder; different temperature schedules of Gumbel-Softmax or Straight-Through Gumbel sampling can be used for the differentiable decoding mechanism; the first loss function can be replaced with an improved version of CTC (such as Focal-CTC), and the weight ratio of the second loss function can be dynamically adjusted.

[0084] In some embodiments, in step S1, the positive sample text is derived from semantically coherent text; the negative sample text is generated by replacing characters at random positions in the positive sample text with similar-looking or homophonic characters; for the negative sample text, its label is set based on the ratio of the number of replaced characters to the text length, and the more the number of replaced characters, the lower the label value.

[0085] The quality of the semantic scoring dataset directly affects the discriminative ability of the semantic scoring model. If the negative samples are only random noises, it is difficult for the model to learn fine-grained semantic disruption features; by implanting similar-looking or homophonic errors in real semantically coherent text, common errors in actual handwriting or OCR scenarios can be simulated, enabling the scoring model to capture subtle semantic incoherence signals.

[0086] Technical solution and component functions: The positive sample text is collected from natural language sources such as news, forms or documents, and is uniformly labeled as label 1. The negative sample text generation module randomly selects several positions (k) in the character sequence of the positive sample text, and performs similar-looking or homophonic character replacement operations on each position, and the generated text is recorded as the negative sample; the label calculation formula is y = 1 - (k / L), where L is the text length. The negative samples and the positive samples together form a training batch for the semantic scoring model to learn or fine-tune.

[0087] Generating negative samples by replacing similar-looking or homophonic characters simulates common recognition errors in real scenarios (such as character confusion in handwritten scripts), enabling the semantic scoring model to accurately distinguish between minor semantic deviations and severe semantic breaks. For example, in Chinese character recognition, when "用户" is replaced with "用户" (similar-looking character) or "永户" (homophonic character), the model will evaluate the degree of semantic degradation according to the replacement ratio, thereby training a more sensitive scorer. This data construction method is low-cost and scalable, does not require manual annotation, and avoids subjective biases, and is particularly suitable for large-scale applications. Secondly, the label is dynamically set based on the replacement ratio, rather than a simple binary label, which introduces a continuous semantic scoring mechanism, enabling the semantic scoring model to output more delicate confidence values instead of making hard judgments. In character recognition training, this scoring can be used as a soft constraint to guide the model to gradually optimize the output, avoid over-penalizing minor errors, and improve training stability.

[0088] From a model generalization perspective, this data augmentation strategy exposes the semantic scoring model to diverse error patterns, making it more robust to unknown data. Traditional semantic models often rely on clean data, but this method injects noise through negative samples, improving the model's tolerance to anomalous inputs and thus maintaining reliability in complex handwritten scenarios (such as illegible handwriting). Furthermore, the label ratio setting allows for dynamic adjustment of the semantic loss weights, achieving adaptive optimization during joint training. For example, when the character recognition model outputs text with fewer replacement characters, the semantic loss is smaller, encouraging the model to prioritize correcting serious errors; conversely, a larger penalty is applied, pushing the model to focus on semantic consistency. This lays a solid foundation for the entire training process, significantly improving the accuracy and practicality of semantic scoring, and indirectly enhancing the performance of the character recognition model.

[0089] Replaceable or modified implementation methods: The strategy for selecting replacement characters can be extended to automatic matching based on glyph similarity threshold, homophone database based on phonetic similarity, or high-probability errors predicted by language models; the replacement ratio k / L can adopt a piecewise linear or exponential decay method; the negative sample labels can adopt a smooth probability distribution rather than a linear mapping to adapt to the needs of different loss functions.

[0090] In some embodiments, in step S2, the semantic scoring model is a Bert-type model, whose input is a text sequence and whose output is a semantic score normalized by a Sigmoid layer, with the score value between 0 and 1; the loss function used when training the semantic scoring model is the binary cross-entropy loss.

[0091] The BERT model possesses powerful contextual representation capabilities, enabling it to capture long-distance dependencies. Mapping its output to a Sigmoid layer in the [0,1] interval yields a probability estimate of text coherence. Binary cross-entropy loss effectively performs probabilistic regression on the labels of positive and negative samples, minimizing the KL divergence between predicted scores and labels during model training.

[0092] The semantic scoring model's embedding layer uses a pre-trained WordPiece vocabulary, which is then encoded through a multi-layer self-attention Transformer. The final pooling vector is mapped to a scalar z via a fully connected layer, and the sigmoid function σ(z) generates the score s∈[0,1]. During training, the cross-entropy loss is calculated for each sample. ,in, For cross-entropy loss, This represents the total number of samples. For the first Sample labels for each sample. For the first Semantic scores for each sample.

[0093] Using a BERT-like model aligns the semantic scoring module with state-of-the-art NLP techniques, facilitating the use of open-source pre-trained models (such as Chinese BERT) and reducing training costs and time. Furthermore, the binary cross-entropy loss simplifies the training process, avoiding the complexity of multi-class classification and enabling rapid convergence of the semantic scoring model. In text recognition training, this high-precision scoring provides a reliable foundation for the second loss function, making semantic constraints more effective. For example, when semantic contradictions occur in the decoded text, the BERT model can keenly detect and output low scores, thereby correcting the text recognition model through backpropagation. This ensures the authority and practicality of the semantic scoring module, providing core support for the entire method.

[0094] Replaceable or modified implementations: Bert can be replaced by similar pre-trained models such as RoBERTa, DistilBERT, and ERNIE; the Sigmoid normalization layer can be replaced with a softmax binary output; the loss function can be replaced with FocalLoss or weighted cross-entropy to adapt to the imbalanced ratio of positive and negative samples.

[0095] In some embodiments, the image feature network includes a convolutional neural network and a bidirectional long short-term memory network, wherein the convolutional neural network is used to extract image features and the bidirectional long short-term memory network is used to output temporal features.

[0096] Convolutional layers excel at capturing local visual patterns, providing spatial feature representations of handwritten characters; bidirectional long short-term memory networks can fuse forward and backward information of feature sequences in the temporal dimension, generating temporal features sensitive to character order, and providing alignment gradient information for subsequent CTC decoding.

[0097] The front end of the image feature network employs several convolutional layers (such as 3×3 convolution, BatchNorm, ReLU) and pooling layers, outputting a feature map of shape [T, C]. This feature map is unfolded in the width direction and input as a sequence to a bidirectional LSTM layer, with the number of layers set to 2 and the number of hidden units set to 256. The forward and backward hidden states of the bidirectional LSTM are concatenated at each time step to form temporal features. (T is the time step, C is the character table size), where This is the logit value of BiLSTM for character c at time step t, used for CTC loss and differentiable decoding.

[0098] The combination of convolution and bidirectional LSTM achieves end-to-end feature mapping from raw pixels to character sequences, preserving both spatial local information and global sequence dependencies. CNNs can efficiently handle image deformation and noise, and are suitable for the diversity of handwriting, such as different writing styles or paper backgrounds. BiLSTM captures the sequential relationships between characters through bidirectional context, reducing isolated recognition errors. For example, in handwritten forms, characters may be partially occluded; BiLSTM can use contextual information to infer the missing parts, improving the recognition rate. Simultaneously, temporal features lay the foundation for CTC loss calculation, enabling the model to handle sequences of variable length. Furthermore, this network structure is highly scalable, allowing the integration of advanced modules such as attention mechanisms to adapt to more complex scenarios.

[0099] Replaceable or modified implementations: The convolutional network can be replaced with ResNet, DenseNet, or a lightweight MobileNet structure; the bidirectional LSTM can be replaced with a bidirectional GRU or a self-attention-based Transformer encoder; the feature map can be unfolded by using convolution stride adjustment or spatial pyramid pooling to adapt to input images of different resolutions.

[0100] In some embodiments, the first loss function is the CTC loss function, which is calculated based on the alignment path between the temporal features and the target text; the differentiable decoding operation is implemented using the Gumbel-Softmax decoding mechanism.

[0101] CTC loss achieves sequence learning without explicit alignment by summing over all possible alignment paths; however, traditional CTC decoding is a discrete operation and cannot directly participate in gradient propagation. Gumbel-Softmax achieves approximate discrete sampling by adding differentiable noise during discrete sampling, enabling the decoding process to provide gradients during backpropagation.

[0102] Technical solution and component functions: During CTC loss calculation, all legal alignment paths are constructed for the temporal features P_t. , The set of paths compatible with the target text is summed to obtain the negative log-likelihood. Subsequently, for each time step Gumbel noise is added; an approximate one-hot distribution is obtained by Softmax processing with temperature parameter τ; argmax is performed on the one-hot distribution to obtain a discrete decoding sequence, which is used as the input for subsequent semantic scoring.

[0103] This combination achieves the sequence alignment capability of CTC loss and the gradient transitivity of differentiable decoding, allowing the semantic scoring loss to influence the parameter updates of the feature network within the same training iteration. The first loss function is explicitly defined as the CTC loss function, and Gumbel-Softmax is used to implement the differentiable decoding operation. CTC loss is widely used in sequence recognition and can handle input-output alignment problems, while Gumbel-Softmax makes the decoding process differentiable, allowing the semantic loss to propagate back. This combination solves the problem of non-differentiability of semantic constraints in traditional methods. CTC loss does not require precise alignment, is suitable for handwritten characters with varying lengths, and reduces annotation costs. Gumbel-Softmax, by introducing noise and temperature parameters and approximating the argmax operation, makes the gradient of the decoded text computable, thus seamlessly integrating the semantic loss. This avoids training interruptions caused by non-differentiable functions, improving stability and convergence speed. For example, during training, the model can simultaneously optimize visual and semantic objectives, achieving end-to-end learning.

[0104] Alternative or modified implementations: CTC loss can be replaced with an improved Focal-CTC or a CTC with added label smoothing; temperature scheduling of Gumbel-Softmax can be achieved using linear decay, cosine annealing, or adaptive adjustment; noise generation can be achieved using a Concrete distribution or by using Straight-Through estimation to achieve approximate discrete sampling.

[0105] In some embodiments, Gumbel noise is added to the temporal features; the temporal features with added noise are processed by the Softmax function and temperature parameters to obtain an approximate one-hot distribution; the approximate one-hot distribution is subjected to an argmax operation to obtain a decoding sequence, and the decoding sequence is used as the decoded text after removing whitespace characters.

[0106] CTC loss achieves sequence learning without explicit alignment by summing over all possible alignment paths; however, traditional CTC decoding is a discrete operation and cannot directly participate in gradient propagation. Gumbel-Softmax achieves approximate discrete sampling by adding differentiable noise during discrete sampling, enabling the decoding process to provide gradients during backpropagation.

[0107] Specifically, for each time step Add Gumbel noise: ,in An approximate one-hot distribution (temperature) is obtained by using Softmax processing with a temperature parameter τ. ): ;right Executing argmax yields a discrete decoded sequence. (After removing whitespace) This will serve as the input for subsequent semantic scoring.

[0108] This mechanism provides a complete gradient path while maintaining an approximately discrete decoding process, allowing the subsequent semantic scoring loss to directly influence the learning of temporal features. It avoids local optima through randomness while preserving differentiability, making the decoded text more diverse and stable. Gumbel noise simulates the sampling process, enhancing exploration capabilities, while the temperature parameter controls the degree of approximation, allowing fuzzy decision-making in the early stages of training and gradually hardening it later to improve accuracy. For example, in handwritten digit recognition, the model can try multiple decoding paths and select the semantically optimal solution. This mechanism also reduces the risk of overfitting, improves generalization, and ensures the practicality of differentiable operations by refining the decoding steps.

[0109] Alternative or modified implementations: sampling of Gumbel noise can be replaced with sampling using a Concrete distribution; the scheduling strategy for temperature τ can be fixed, exponentially decaying, or adaptively adjusted based on gradient magnitude; the argmax step can employ Straight-Through estimation to further simplify backpropagation.

[0110] In some embodiments, in step S5, the second loss function is calculated based on the semantic score, which is output by a pre-trained semantic scoring model on the decoded text.

[0111] When using only CTC loss, the model lacks constraints on the overall semantic coherence of the output text; introducing a semantic scoring-based loss can penalize incoherent decoding results at the gradient level, thereby guiding the feature network to learn representations that are more in line with language rules.

[0112] The decoded text is input into a semantic scoring model (BERT class), and the model outputs a normalized score s∈[0,1]; the second loss function is defined as follows: =1-s (or cross-entropy form-logs), directly reflecting the degree of insufficient text coherence; this loss participates in the gradient calculation in the total loss with weight b.

[0113] By transforming semantic scores into differentiable numerical losses, semantic constraints can work in conjunction with CTC gradients in the same training step, improving the model's learning of semantic integrity. It emphasizes that the second loss function is calculated based on semantic scores, directly utilizing the output of the pre-trained semantic scoring model. This allows the text recognition model to directly benefit from semantic knowledge without integrating complex NLP modules internally. Semantic scores, as external supervision, provide an objective measure of semantic quality, making loss calculation more accurate. Simultaneously, the pre-trained model freezes parameters, ensuring score consistency and avoiding training fluctuations. For example, during optimization, the second loss function guides the model to prioritize outputting coherent text, improving user experience, simplifying the semantic integration process, and reducing implementation complexity.

[0114] Alternative or modified implementation methods: The second loss function can be the squared error (1-s)^2 or the weighted cross-entropy; the output layer of the semantic scoring model can be replaced with a two-layer perceptron or a temperature-calibrated Softmax to adapt to different annotation scales; if a multilingual scenario is used, a language recognition branch can be added before the scoring model.

[0115] In some embodiments, in step S6, the total loss function is a weighted sum of the first loss function and the second loss function, with the sum of the weight coefficients being 1; the backpropagation optimization adopts a phased training strategy: in the first phase, the character recognition model is pre-trained using only the first loss function; in the second phase, the character recognition model is jointly trained using the total loss function; during the joint training in the second phase, the parameters of the semantic scoring model remain frozen, and only the parameters of the character recognition model are optimized.

[0116] Directly performing joint training on the complete loss may cause gradient noise in the semantic scoring model to affect the convergence of the feature network. By adopting a phased strategy, first allowing the character recognition model to learn robust visual features under CTC constraints only, and then introducing semantic constraints for refinement, the overall training stability and convergence speed can be improved.

[0117] In the first phase, losses = The semantic scoring model does not participate in the forward computation; after pre-training is completed, all parameters of the semantic scoring model are frozen, and the second stage begins, with the loss... =a· +b· The weight coefficients a and b satisfy a+b=1, and are usually taken as a=0.7 and b=0.3; backpropagation only updates the weights of the convolutional network and the bidirectional LSTM.

[0118] Phased training enables rapid convergence of visual features first, followed by fine-grained correction through semantic constraints. This avoids gradient invariance issues caused by semantic scoring noise in the initial stage. The total loss function is defined as a weighted sum, and a phased training strategy is employed: the text recognition model is pre-trained first, then jointly trained, with the semantic scoring model parameters frozen. This ensures gradual training, avoids initial semantic interference, and improves stability. Pre-training allows the model to master visual features first, followed by the introduction of semantic constraints, preventing gradient conflicts. The weighted sum allows adjustment of the visual and semantic balance to adapt to different scenarios. Freezing semantic parameters reduces computational overhead and accelerates training. For example, on resource-constrained devices, weights can be adjusted to prioritize real-time performance. By optimizing the training strategy, the feasibility and efficiency of the method are improved.

[0119] Alternative or modified implementation methods: The weight coefficients can be dynamically adjusted, such as increasing linearly by b with each training round; the freezing strategy can be changed to partial freezing (freezing only the first few layers of the language model) or fine-tuning with a low learning rate; additional auxiliary losses (such as character-level cross-entropy) can be used in the first stage to further improve the quality of feature extraction.

[0120] In some embodiments, during the inference phase, for the input image, the text is directly decoded and output by the character recognition model (CTC), without calling the semantic scoring model.

[0121] The semantic score introduced during training is only used to improve the model's ability to learn coherent text. In actual deployment, the internal parameters of the character recognition model have implicit semantic constraints through joint training, so additional semantic score calculation can be omitted during inference, reducing latency.

[0122] Technical solution and component functions: The inference process includes: ① The input image is fed into a pre-trained convolutional network and bidirectional LSTM to obtain temporal features P; ② Standard CTC decoding (such as greedy or beam search) is directly performed on P to produce the final text output x^; ③ No secondary semantic scoring is performed on the decoded text.

[0123] Omitting the forward computation of the semantic scoring model significantly reduces computational resource consumption and response time during inference, meeting the needs of real-time applications. CTC decoding is performed directly using the text recognition model during the inference phase, eliminating the need to call the semantic scoring model. This significantly reduces inference latency, meets real-time requirements, simplifies the deployment process, reduces dependent modules, and makes the system more lightweight. For example, in mobile applications, the model can quickly output text without additional semantic scoring calculations. Simultaneously, the semantic knowledge acquired during training is internalized into the recognition model, ensuring output quality. Through inference optimization, the method achieves efficient deployment.

[0124] Alternative or modified implementation methods: CTC decoding can use beam search or beam-search with length penalty; if higher semantic consistency is required for a specific scenario, a lightweight rule filter can be added after inference instead of a complete semantic model.

[0125] Figure 3 This application provides a schematic diagram of the structure of a text recognition semantic model training system based on image information, as shown in one embodiment. Figure 3 As shown, the text recognition semantic model training system 300 based on image information in this embodiment includes: a data construction module 301, a semantic training module 302, a feature analysis module 303, a micro-decoding module 304, a semantic scoring module 305, and a backpropagation module 306.

[0126] The data construction module 301 is used to construct a semantic scoring dataset, which includes semantically coherent positive sample text and semantically incoherent negative sample text.

[0127] Semantic training module 302 is used to train a semantic scoring model based on the semantic scoring dataset, wherein the semantic scoring model is able to output a semantic coherence score for the input text;

[0128] Feature analysis module 303 is used to construct a text recognition model, which includes an image feature network. For an input image, temporal features are extracted through the image feature network.

[0129] The micro-decoding module 304 is used to calculate a first loss function based on the difference between the temporal features and the target text, and to perform differentiable decoding operation on the temporal features through the first loss function to determine the decoded text;

[0130] The semantic scoring module 305 is used to input the decoded text into the semantic scoring model to obtain a semantic score, and to calculate a second loss function based on the semantic score;

[0131] The backpropagation module 306 is used to combine the first loss function and the second loss function to obtain the total loss function, and to optimize the parameters of the character recognition model through backpropagation.

[0132] Optionally, in the data construction module 301, the positive sample text is derived from semantically coherent text; the negative sample text is generated by replacing characters at random positions in the positive sample text with similar-looking or homophonous characters.

[0133] For the negative sample text, its label is set based on the ratio of the number of replacement characters to the text length; the more replacement characters, the lower the label value.

[0134] Optionally, in the semantic scoring module 302, the semantic scoring model is a Bert-type model, whose input is a text sequence and whose output is a semantic score normalized by a Sigmoid layer, with the score value between 0 and 1; the loss function used when training the semantic scoring model is the binary cross-entropy loss.

[0135] Optionally, in the feature analysis module 303, the image feature network includes a convolutional neural network and a bidirectional long short-term memory network. The convolutional neural network is used to extract image features, and the bidirectional long short-term memory network is used to output temporal features.

[0136] Optionally, in the micro-decoding module 304, the first loss function is the CTC loss function, which is calculated based on the alignment path between the temporal features and the target text; the differentiable decoding operation is implemented using the Gumbel-Softmax decoding mechanism.

[0137] Optionally, in the micro-decoding module 304, the Gumbel-Softmax decoding mechanism includes:

[0138] Add Gumbel noise to the time-series features;

[0139] The time-series features after adding noise are processed using the Softmax function and temperature parameters to obtain an approximate one-hot distribution;

[0140] The argmax operation is performed on the approximate one-hot distribution to obtain the decoding sequence, and the whitespace characters are removed from the decoding sequence to obtain the decoded text.

[0141] Optionally, in the semantic scoring module 305, the second loss function is calculated based on the semantic score, which is output by a pre-trained semantic scoring model on the decoded text.

[0142] Optionally, in the backpropagation module 306, the total loss function is a weighted sum of the first loss function and the second loss function, and the sum of the weight coefficients is 1;

[0143] The backpropagation optimization employs a phased training strategy:

[0144] In the first stage, the character recognition model is pre-trained using only the first loss function;

[0145] In the second stage, the character recognition model is jointly trained using the total loss function.

[0146] During the second phase of joint training, the parameters of the semantic scoring model are kept frozen, and only the parameters of the character recognition model are optimized.

[0147] Optionally, in the feature analysis module 303, for the input image, the text is directly decoded and output through the character recognition model (CTC), without calling the semantic scoring model.

[0148] The system in this embodiment can be used to execute the methods of any of the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

Claims

1. A method for training a semantic model for text recognition based on image information, characterized in that, include: S1. Construct a semantic scoring dataset, which contains semantically coherent positive sample text and semantically incoherent negative sample text; S2. Based on the semantic scoring dataset, train a semantic scoring model, which can output a semantic coherence score for the input text; S3. Construct a text recognition model, which includes an image feature network. For an input image, extract temporal features through the image feature network. S4. Based on the difference between the temporal features and the target text, calculate the first loss function, and perform differentiable decoding on the temporal features using the first loss function to determine the decoded text; S5. Input the decoded text into the semantic scoring model to obtain a semantic score, and calculate a second loss function based on the semantic score; S6. Combine the first loss function and the second loss function to obtain the total loss function, and optimize the parameters of the character recognition model through backpropagation.

2. The method according to claim 1, characterized in that, In step S1, the positive sample text is derived from semantically coherent text; the negative sample text is generated by replacing characters at random positions in the positive sample text with similar-looking or homophonous characters. For the negative sample text, its label is set based on the ratio of the number of replacement characters to the text length; the more replacement characters, the lower the label value.

3. The method according to claim 2, characterized in that, In step S2, the semantic scoring model is a Bert-type model, whose input is a text sequence and whose output is a semantic score normalized by a Sigmoid layer, with the score value between 0 and 1; the loss function used when training the semantic scoring model is the binary cross-entropy loss.

4. The method according to claim 3, characterized in that, The image feature network includes a convolutional neural network and a bidirectional long short-term memory network. The convolutional neural network is used to extract image features, and the bidirectional long short-term memory network is used to output temporal features.

5. The method according to claim 3, characterized in that, In step S4, the first loss function is the CTC loss function, which is calculated based on the alignment path between the temporal features and the target text; the differentiable decoding operation is implemented using the Gumbel-Softmax decoding mechanism.

6. The method according to claim 5, characterized in that, The Gumbel-Softmax decoding mechanism includes: Add Gumbel noise to the time-series features; The time-series features after adding noise are processed using the Softmax function and temperature parameters to obtain an approximate one-hot distribution; The argmax operation is performed on the approximate one-hot distribution to obtain the decoding sequence, and the whitespace characters are removed from the decoding sequence to obtain the decoded text.

7. The method according to claim 5, characterized in that, In step S5, the second loss function is calculated based on the semantic score, which is output by a pre-trained semantic scoring model on the decoded text.

8. The method according to claim 5, characterized in that, In step S6, the total loss function is a weighted sum of the first loss function and the second loss function, and the sum of the weight coefficients is 1. The backpropagation optimization employs a phased training strategy: In the first stage, the character recognition model is pre-trained using only the first loss function; In the second stage, the character recognition model is jointly trained using the total loss function. During the second phase of joint training, the parameters of the semantic scoring model are kept frozen, and only the parameters of the character recognition model are optimized.

9. The method according to claim 5, characterized in that, During the inference phase, for the input image, the text is directly decoded and output by the character recognition model (CTC), without needing to call the semantic scoring model.

10. A text recognition semantic model training system based on image information, characterized in that, The method applied to any one of claims 1-9 includes: The data construction module is used to construct a semantic scoring dataset, which contains semantically coherent positive sample text and semantically incoherent negative sample text. The semantic training module is used to train a semantic scoring model based on the semantic scoring dataset. The semantic scoring model is able to output a semantic coherence score for the input text. The feature analysis module is used to construct a text recognition model, which includes an image feature network. For an input image, temporal features are extracted through the image feature network. The micro-decoding module is used to calculate a first loss function based on the difference between the temporal features and the target text, and to perform differentiable decoding on the temporal features through the first loss function to determine the decoded text; The semantic scoring module is used to input the decoded text into the semantic scoring model to obtain a semantic score, and to calculate a second loss function based on the semantic score; The backpropagation module is used to combine the first loss function and the second loss function to obtain the total loss function, and to optimize the parameters of the character recognition model through backpropagation.

Images