Construction method of handwritten chemical equation recognition model based on transformer-crnn hybrid network

Abstract

The application discloses a handwriting chemical equation recognition model construction method based on a Transformer-CRNN hybrid network and relates to the technical field of handwriting chemical equation recognition. The application discloses a handwriting chemical equation recognition model construction method based on a Transformer-CRNN hybrid network and relates to the technical field of handwriting chemical equation recognition. The handwriting chemical equation recognition model comprises a feature extraction module, a deep bidirectional recurrent neural network module, a Transformer encoder module and a connection time sequence classification module. The application discloses a handwriting chemical equation recognition model construction method based on a Transformer-CRNN hybrid network and relates to the technical field of handwriting chemical equation recognition. The application discloses a handwriting chemical equation recognition model construction method based on a Transformer-CRNN hybrid network and relates to the technical field of handwriting chemical equation recognition. The application discloses a handwriting chemical equation recognition model construction method based on a Transformer-CRNN hybrid network and relates to the technical field of handwriting chemical equation recognition.

Description

Technical Field

[0001] This invention relates to the field of handwritten chemical equation recognition technology, and in particular to a method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network. Background Technology

[0002] Handwriting recognition is an important research direction in the fields of pattern recognition and document analysis. Its core objective is to automatically convert handwritten input into machine-editable, searchable, and computable digital text. Handwritten chemical equation image recognition falls under the category of handwriting recognition. Chemical equations have significant two-dimensional structural characteristics in their expression. Their composition includes not only basic components such as element symbols, numbers, and subscripts and superscripts, but also state markers, reaction conditions, and the spatial layout of complex structural units. This structural nature makes chemical equations different from ordinary natural language text, placing higher demands on their digital storage, automatic retrieval, and intelligent analysis.

[0003] Existing methods for recognizing handwritten chemical equations can be divided into traditional methods and deep learning-based methods.

[0004] Traditional recognition methods typically involve three steps: image preprocessing, character segmentation, and character recognition.

[0005] 1) Image Preprocessing: Images of handwritten chemical equations to be recognized often contain noise, making direct recognition difficult. Therefore, image preprocessing is necessary, including operations such as binarization, thinning, normalization, and image correction. The main purpose of image preprocessing is to optimize the quality of the original image, highlight handwritten features, and reduce interference, laying the foundation for subsequent recognition.

[0006] 2) Character segmentation: Character segmentation is the process of breaking down preprocessed text into individual characters. The principle is to separate connected or linked characters based on the spatial distribution and connectivity of handwritten characters. Edge detection and connected component analysis are commonly used to segment individual symbols in chemical equations.

[0007] 3) Character Recognition: Character recognition is a crucial step in feature extraction and classification after image preprocessing and character segmentation. Its principle is to design algorithms to extract unique features of characters and then use a classifier to map these features to corresponding character categories. Classic classification techniques such as Hidden Markov Models (HMMs), Elastic Matching, and Support Vector Machines (SVMs) are commonly used to achieve character recognition.

[0008] Deep learning-based recognition methods primarily utilize Convolutional Recurrent Neural Networks (CRNNs), deep learning models specifically designed to solve sequential text recognition problems. By combining the feature extraction capabilities of Convolutional Neural Networks (CNNs) with the sequence modeling capabilities of Recurrent Neural Networks (RNNs), CRNNs achieve efficient recognition of irregular text, handwritten text, and complex layouts. Compared to the ConvNextTiny architecture, CRNNs significantly improve recognition accuracy in Chinese handwritten text and complex background text recognition tasks.

[0009] The CRNN network consists of three parts: convolutional layers, recurrent layers, and transcriptional layers.

[0010] 1) Convolutional layer: The convolutional layer uses a convolutional neural network (CNN), which can effectively extract hierarchical local features from the original image through a series of convolutional and pooling layers, including information such as the shape and capitalization of characters, to generate high-level semantic representations.

[0011] 2) Recurrent Layer: The recurrent layer employs a Recurrent Neural Network (RNN). RNNs are neural networks specifically designed for sequential data, capable of capturing the temporal dependencies within the sequence. In a CRNN, the RNN processes the feature sequences extracted by the CNN and outputs the prediction result for each character. It effectively captures the contextual relationships between characters in the sequence, using contextual cues from the image for sequence recognition. This transforms the feature sequences output by the CNN into context-aware feature representations, which is more stable and helpful than processing each symbol individually.

[0012] 3) Transcription Layer: The transcription layer typically employs Connectionist Temporal Classification (CTC), a training method for sequence models used to address the inconsistency between the input image length and the output character sequence length. It allows training without character segmentation. The prediction results for each frame can be converted into a label sequence output, which is then aligned with the label sequence to obtain the final text sequence.

[0013] Deep learning methods, with convolutional recurrent neural networks (CRNNs) at their core, improve the robustness of the system's recognition through "end-side sequence learning," thereby enhancing the accuracy of recognizing handwritten chemical equations.

[0014] Existing methods for recognizing handwritten chemical equations have the following problems: (1) Based on traditional identification methods 1) Image preprocessing: During image preprocessing, operations such as grayscale conversion and binarization will lose the color and detail information of the original image. If the preprocessing is excessive (such as strong filtering), it will blur key strokes (such as the horizontal bar of the number "7") and affect subsequent recognition.

[0015] 2) Character segmentation: In the process of character segmentation, when the input image quality is low and the handwritten chemical equation is complex, it will lead to segmentation errors, which will directly cause subsequent recognition failures. Furthermore, traditional recognition methods lack error correction mechanisms and cannot correct segmentation errors through context.

[0016] 3) Character Recognition: In the character recognition process, structural features (such as grid features and skeleton features) are difficult to cover the diversity of handwriting styles. For example, the number "2" written by different people may have straight hooks or curved hooks. Traditional feature extractors may misclassify it as different characters, and the classifier's insufficient generalization ability will also lead to a decrease in recognition accuracy. Therefore, it performs poorly in situations with diverse handwriting styles, resulting in low recognition accuracy.

[0017] (2) Deep learning-based recognition methods When Chinese characters are included in the reaction conditions of handwritten chemical equations, CRNN networks cannot capture global temporal dependencies well during the recognition process, resulting in poor recognition accuracy. This indicates that the recognition of handwritten chemical equations still faces challenges. Summary of the Invention

[0018] This invention addresses the problems existing in the prior art by proposing a method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network.

[0019] The technical means employed in this invention are as follows: A method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network includes the following steps: Step 1: Construct a handwritten chemical equation recognition model; The handwritten chemical formula recognition model includes a feature extraction module, a deep bidirectional recurrent neural network module, a Transformer encoder module, a semantic guidance module, and a connection-time classification module. The feature extraction module is used to perform feature extraction processing on the handwritten chemical equation image to obtain a high-dimensional feature sequence of the handwritten chemical equation image; The deep bidirectional recurrent neural network module is used to perform enhanced feature processing on the high-dimensional feature sequence to obtain an enhanced high-dimensional feature sequence. The Transformer encoder module is used to encode the enhanced high-dimensional feature sequence to obtain a feature sequence with contextual information. The semantic guidance module is used to perform semantic guidance processing on the enhanced high-dimensional feature sequence during the training phase to obtain a cross-entropy loss function with contextual information; The connection temporal classification module is used to perform connection temporal classification processing on feature sequences with contextual information and output the correct recognition result; Step 2: Obtain the image of the handwritten chemical equation and perform data augmentation processing on the handwritten chemical equation image to obtain the augmented handwritten chemical equation image; Step 3: Input the acquired augmented handwritten chemical equation image into the handwritten chemical equation recognition model for training to obtain the trained handwritten chemical equation recognition model.

[0020] Furthermore, the feature extraction module includes convolutional unit 1, pooling layer 1, convolutional unit 2, pooling layer 2, convolutional unit 3, convolutional unit 4, pooling layer 3, convolutional unit 5, convolutional unit 6, pooling layer 4, and convolutional unit 7; The first convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to extract the edges, stroke direction, and local texture features in the handwritten chemical equation image to obtain a low-level feature map. The first pooling layer consists of a 2×2 max pooling layer, which is used to perform downsampling processing on the low-level feature map to obtain a downsampled feature map; The second convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to enhance the local character morphology to obtain a low-level enhanced feature map; The second pooling layer consists of a 2×2 max pooling layer, which is used to perform downsampling processing on the low-level enhanced feature map to obtain the low-level enhanced downsampled feature map; The third convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function. It is used to perform nonlinear mapping processing on the downsampled feature map after the low-level enhancement to obtain the mid-level feature map of the structural patterns of element symbols, numbers, superscripts and subscripts, and reaction arrows. The convolutional unit 4 consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to perform nonlinear mapping processing on the middle layer feature map to obtain the deep layer feature map; The third pooling layer consists of a 2×2 max pooling layer, which is used to compress the size of the deep feature map to improve the robustness of the model to local deformation and differences in writing style, and to obtain the compressed deep feature map. The fifth convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function. It is used to perform feature abstraction on the complex chemical symbol combinations in the compressed deep feature image to obtain a deep feature image after feature abstraction. The convolutional unit six consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to perform feature sampling processing on the deep feature image after the first feature abstraction to obtain the deep feature image after the second feature deep abstraction. The pooling layer four consists of 2×2 max pooling layers, which are used to perform spatial downsampling processing on the deep feature image after the deep abstraction of the secondary features to obtain the downsampled deep feature image; The convolutional unit seven consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to integrate and enhance the downsampled deep feature image to obtain a high-dimensional feature sequence of the handwritten chemical equation image.

[0021] Furthermore, the deep bidirectional recurrent neural network module includes a bidirectional recurrent neural network and a deep recurrent neural network; The bidirectional recurrent neural network is used to obtain and concatenate the high-dimensional feature sequence of the handwritten chemical equation image to obtain a high-level abstract feature map by acquiring and concatenating the forward and reverse hidden states. The deep recurrent neural network is used to stack the high-level abstract feature maps to obtain an enhanced high-dimensional feature sequence.

[0022] Furthermore, the Transformer encoder module includes an input preprocessing layer, a multi-head self-attention layer, a feedforward neural network, and a residual connection and layer normalization layer; The input preprocessing layer includes a word embedding layer and a position encoding layer. The word embedding layer is used to linearly map the enhanced high-dimensional feature sequence output by the deep bidirectional recurrent neural network module to the fixed-dimensional space required by the multi-head self-attention layer. The position encoding layer is used to inject absolute or relative position information into the fixed-dimensional space to obtain a fixed-dimensional space with position information. The multi-head self-attention layer includes a linear mapping layer, a split-head processing layer, a scaled dot product attention layer, and a splicing and linear transformation layer; The linear mapping layer is used to perform linear mapping processing on the fixed-dimensional space of the location information to obtain three different vector spaces; The split processing layer is used to split the different vector spaces to obtain multiple sub-matrices; The scaling dot product attention layer is used to perform scaling dot product attention processing on each submatrix to obtain a scaled submatrix; The concatenation and linear transformation layer is used to concatenate and linearly transform multiple scaled sub-matrices to obtain a multi-head attention output matrix. The feedforward neural network includes a first linear transformation layer, a nonlinear activation function, and a second linear transformation layer; The first linear transformation layer is used to perform linear transformation on the multi-head attention output matrix to obtain a high-dimensional multi-head attention output matrix; The nonlinear activation function is used to perform nonlinear activation on the high-dimensional multi-head attention output matrix to obtain a nonlinear high-dimensional multi-head attention output matrix. The second linear transformation layer is used to perform a linear transformation on the nonlinear high-dimensional multi-head attention output matrix to obtain a low-dimensional multi-head attention output matrix with the same dimension as the multi-head attention output matrix; The residual connection and layer normalization layer are used to perform residual connection and layer normalization processing on the low-dimensional multi-head attention output matrix to obtain the feature sequence with contextual information.

[0023] Furthermore, the semantic guidance module is used to perform semantic guidance processing on the enhanced high-dimensional feature sequence during the training phase to obtain a cross-entropy loss function with contextual information. The specific process is as follows: Used to obtain the left and right strings of the target character in the enhanced high-dimensional feature sequence, and to generate the context semantic hidden representation sequence of the left string and the context semantic hidden representation sequence of the right string; Dot product attention is performed on the context semantic hidden representation sequence of the left string and the context semantic hidden representation sequence of the right string with the enhanced high-dimensional feature sequence after transformation by the learned weight matrix to obtain the attention map of the left string and the attention map of the right string. The attention maps of the left and right strings are weighted and processed with the enhanced high-dimensional feature sequence to generate features for the characters corresponding to the characters of the left and right strings, respectively. The features of the characters corresponding to the left string and the features of the characters corresponding to the right string are processed by a classifier to obtain the predicted class probability of the characters corresponding to the left string and the predicted class probability of the characters corresponding to the right string. Calculate the cross-entropy loss function between the corresponding characters in the left and right strings and their predicted class probabilities.

[0024] Furthermore, the connection time series classification module includes an extended character set module, a feature and probability mapping module, a path mapping module, and a connection time series classification loss calculation module; The extended character set module is used to extend the original chemical symbol character set to generate an extended chemical symbol character set with "blank" symbols; The feature and probability mapping module is used to map the feature sequence to the extended chemical symbol character set through a linear transformation layer, generating a feature sequence mapping space; and to calculate the feature sequence at each time step. Output the probability of each chemical symbol character and generate multiple sets of length [value missing]. Character probability sequence path; The path mapping module is used to construct a mapping function and obtain, based on the mapping function, an effective character probability sequence path that can output the target output sequence from the character probability sequence path; The connection time-series classification loss calculation module is used to calculate the sum of the loss function value obtained from the effective character probability sequence path to the target output sequence and the cross-entropy loss function output by the semantic guidance module, distributed proportionally.

[0025] Furthermore, the augmentation process includes rotating and adding noise to the handwritten chemical equation image.

[0026] Compared with existing technologies, the method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network disclosed in this invention has the following advantages: 1. This invention improves the CRNN model: A Transformer encoder module is added to the CRNN model. The multi-head attention module in the Transformer encoder module adds multiple "attention heads" to the ordinary attention mechanism, enabling the model to better learn and represent information. It can take into account both the local morphological features and global semantic information of handwritten chemical equations, avoiding the omission of key features by a single attention mechanism, capturing global dependencies, and improving the model's recognition accuracy. 2. A semantic guidance module has been added: This module can be added as a "plug-and-play" component to the existing training process. By revealing the "correct recognition result," it continuously optimizes the network model and improves the model's recognition accuracy. The semantic guidance module is removed during inference, so it does not affect the model's inference speed at all. Therefore, both its recognition speed and recognition accuracy are improved. Attached Figure Description

[0027] Figure 1 The model structure is constructed according to the method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network disclosed in this invention. Figure 2This is a structural diagram of the feature extraction module in the model constructed using the handwritten chemical equation recognition model construction method based on Transformer-CRNN hybrid network disclosed in this invention; Figure 3 This is a structural diagram of the deep bidirectional recurrent neural network module in the model constructed using the handwritten chemical equation recognition model construction method based on Transformer-CRNN hybrid network disclosed in this invention; Figure 4 This is a structural diagram of the semantic guidance module in the model constructed using the handwritten chemical equation recognition model construction method based on Transformer-CRNN hybrid network disclosed in this invention; Figure 5 This is a structural diagram of the Transformer encoder module in the model constructed using the handwritten chemical equation recognition model construction method based on the Transformer-CRNN hybrid network disclosed in this invention; Figure 6 This is a structural diagram of the multi-head self-attention layer in the Transformer encoder module of the model constructed using the handwritten chemical equation recognition model construction method based on the Transformer-CRNN hybrid network disclosed in this invention; Figure 7 This is the original image of a handwritten chemical equation in one embodiment of this application; Figure 8 To Figure 7 The augmented data image obtained by rotating the original image 90° clockwise; Figure 9 To Figure 7 The augmented data image obtained by rotating the original image 180° clockwise; Figure 10 To Figure 7 The augmented data image obtained by rotating the original image clockwise by 270°; Figure 11 To Figure 7 Augmented data images with added noise to the original image. Detailed Implementation

[0028] In the recognition of handwritten chemical equations, CRNNs cannot capture global temporal dependencies well, resulting in poor recognition accuracy. Based on this, the main contributions of this invention are as follows: (1) Improved CRNN model This invention adds a Transformer encoder module to the traditional CRNN model. Because the multi-head attention mechanism in the Transformer encoder module enables the model to better learn and represent information, it can capture global dependencies, thereby improving the model's recognition accuracy.

[0029] (2) A semantic guidance module has been added. In the training phase, this invention employs a semantic guidance module to integrate linguistic context into the feature sequence, which is then input into the Connectionist Temporal Classification (CTC) module. The core idea of ​​the semantic guidance module is to analyze the context of the target character and capture its semantic features, thereby improving the model's ability to understand and generate language. By utilizing linguistic information, the accuracy of recognition can be effectively improved. Removing the semantic guidance module during the inference phase enhances the character recognition capability of the visual model without increasing inference costs.

[0030] The method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network disclosed in this invention specifically includes the following steps: Step 1: Construct a handwritten chemical equation recognition model; The handwritten chemical formula recognition model includes a feature extraction module, a deep bidirectional recurrent neural network module, a Transformer encoder module, a semantic guidance module, and a connection-time classification module. The feature extraction module is used to perform feature extraction processing on the handwritten chemical equation image to obtain a high-dimensional feature sequence of the handwritten chemical equation image; The deep bidirectional recurrent neural network module is used to perform enhanced feature processing on the high-dimensional feature sequence to obtain an enhanced high-dimensional feature sequence. The Transformer encoder module is used to encode the enhanced high-dimensional feature sequence to obtain a feature sequence with contextual information. The semantic guidance module is used to perform semantic guidance processing on the enhanced high-dimensional feature sequence during the training phase to obtain a cross-entropy loss function with contextual information. The connection temporal classification module is used to perform connection temporal classification processing on feature sequences with contextual information to output the correct recognition result; Step 2: Obtain the image of the handwritten chemical equation and perform data augmentation processing on the handwritten chemical equation image to obtain the augmented handwritten chemical equation image; Step 3: Input the acquired augmented handwritten chemical equation image into the handwritten chemical equation recognition model for training to obtain the trained handwritten chemical equation recognition model.

[0031] Specifically, such as Figure 1As shown, the method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network disclosed in this invention consists of a feature extraction module, a bidirectional recurrent neural network (BiRNN) module, a Transformer encoder module, a semantic guidance module, and a connection-based temporal classifier module. The functions of each part are described below: like Figure 2 As shown, the feature extraction module consists of seven convolutional units (Conv+ BatchNorm + ReLU) and four pooling layers (Max Pool). Its core task is to transform the input handwritten chemical equation image into a high-dimensional feature sequence rich in semantic information. The feature extraction module is specifically as follows: The feature extraction module includes convolutional unit 1, pooling layer 1, convolutional unit 2, pooling layer 2, convolutional unit 3, convolutional unit 4, pooling layer 3, convolutional unit 5, convolutional unit 6, pooling layer 4, and convolutional unit 7. The first convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to extract the edges, stroke direction, and local texture features in the handwritten chemical equation image to obtain a low-level feature map. The first pooling layer consists of a 2×2 max pooling layer, which is used to perform downsampling processing on the low-level feature map to obtain a downsampled feature map; The second convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to enhance the local character morphology to obtain a low-level enhanced feature map; The second pooling layer consists of a 2×2 max pooling layer, which is used to perform downsampling processing on the low-level enhanced feature map to obtain the low-level enhanced downsampled feature map; The third convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function. It is used to perform nonlinear mapping processing on the downsampled feature map after the low-level enhancement to obtain the mid-level feature map of the structural patterns of element symbols, numbers, superscripts and subscripts, and reaction arrows. The convolutional unit 4 consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to perform nonlinear mapping processing on the middle layer feature map to obtain the deep layer feature map; The third pooling layer consists of a 2×2 max pooling layer, which is used to compress the size of the deep feature map to improve the robustness of the model to local deformation and differences in writing style, and to obtain the compressed deep feature map. The fifth convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function. It is used to perform feature abstraction on the complex chemical symbol combinations in the compressed deep feature image to obtain a deep feature image after feature abstraction. The convolutional unit six consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to perform feature sampling processing on the deep feature image after the first feature abstraction to obtain the deep feature image after the second feature deep abstraction. The pooling layer four consists of 2×2 max pooling layers, which are used to perform spatial downsampling processing on the deep feature image after the deep abstraction of the secondary features to obtain the downsampled deep feature image; The convolutional unit seven consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to integrate and enhance the downsampled deep feature image to obtain a high-dimensional feature sequence of the handwritten chemical equation image.

[0032] In other words, the first convolutional unit of the feature extraction module consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function. It is used to extract low-level features such as edges, stroke direction, and local textures in the image. Then, a 2×2 max pooling layer is used to downsample the feature map. The purpose is to reduce spatial resolution and computational complexity while retaining the main response information. The second convolutional unit and its corresponding pooling layer can further enhance the ability to represent local character shapes. The continuous stacking of the third and fourth convolutional units can gradually expand the receptive field through multi-layer nonlinear mapping, enabling the network to learn higher-level structural patterns such as element symbols, numbers, subscripts and superscripts, and reaction arrows. The pooling layer further compresses the feature map size, improving the model's robustness to local deformations and differences in writing style. The fifth and sixth convolutional units continue to abstract features from complex chemical symbol combinations in the image, and the pooling layer further completes spatial downsampling. The seventh convolutional unit integrates and enhances the aforementioned multi-layer features, outputting the final high-level visual representation, which is the high-dimensional feature sequence of the handwritten chemical equation image.

[0033] Furthermore, the deep bidirectional recurrent neural network module includes a bidirectional recurrent neural network and a deep recurrent neural network; The bidirectional recurrent neural network is used to obtain and concatenate the high-dimensional feature sequence of the handwritten chemical equation image to obtain a high-level abstract feature map by acquiring and concatenating the forward and reverse hidden states. The deep recurrent neural network is used to stack the high-level abstract feature maps to obtain an enhanced high-dimensional feature sequence.

[0034] Specifically, such as Figure 3As shown, the deep bidirectional recurrent neural network module is a high-level architecture of recurrent neural networks (RNNs). It combines the characteristics of bidirectional RNNs and deep RNNs, aiming to comprehensively capture contextual information and complex patterns in sequence data. The bidirectional RNN consists of two branches: a forward RNN and a backward RNN. The forward RNN unit is composed of forward hidden layers, processing each column of feature vectors from the CNN output in normal chronological order (from left to right) of the input sequence, capturing contextual information from the past to the present. The backward RNN unit processes the input sequence in reverse chronological order (from right to left), capturing contextual information from the future to the present. At each time step t, the final output of the bidirectional RNN is the concatenation of the forward and backward RNN hidden states to enhance the ability to distinguish similar characters. Its specific working process is as follows: (1) Input feature sequence (T is the total time step of the sequence) The input feature vector at time t is passed to the first forward RNN branch (forward hidden layer) and propagated forward (from left to right) to generate the forward hidden state: calculated time-by-time in the forward direction from the initial time 1 to the last time T, combining the current input feature with the forward hidden state of the previous time. The positive hidden state at the current time step is generated through a nonlinear transformation, and its mathematical representation is as follows: (1) in, Let be the hidden state of the first positive hidden layer at time t. For activation function, This is the weight matrix from the input layer to the first forward hidden layer. This is the weight matrix from the forward hidden layer to itself. This is the bias vector.

[0035] (2) Input feature sequence (T is the total time step of the sequence) The input feature vector at time t is passed to the first inverse hidden layer and backpropagated (from right to left) to generate the inverse hidden state. At each time step t, the inverse hidden state is generated. It is determined by the input at the current time step. and the reverse hidden state at the next time step. The decision is made jointly, and its mathematical representation is: (2) in, Let be the hidden state of the first inverse hidden layer at time t. This is the weight matrix from the input layer to the first inverse hidden layer. This is the weight matrix from the first inverse hidden layer to itself. This is the bias vector for the first inverse hidden layer.

[0036] (3) Bidirectional hidden state splicing and output For each time t, forward the hidden state. With reverse hidden state The mathematical representation of dimension concatenation is: (3) (4) Output of the concatenation of the first layer forward RNN and backward RNN As the input to the next layer of forward and backward RNNs, its learning process is the same as described above, thus learning higher-level abstract features. Deep RNNs stack multiple RNN layers together, with the output of the previous RNN serving as the input to the next layer. This structure allows the model to learn more abstract and complex hierarchical features in the data, thus possessing stronger expressive and learning capabilities. This paper selects two layers, and the output feature sequence B is the enhanced high-dimensional feature sequence.

[0037] In this application, the Transformer encoder is suitable for understanding or analyzing discriminative tasks centered on input content. Its self-attention mechanism can directly model the dependency relationship between any two positions, and can achieve better results in long text recognition scenarios.

[0038] The Transformer encoder is composed of multiple sub-layers with the same structure. Each sub-layer contains two core sub-modules: a multi-head self-attention mechanism and a position-wise feedforward neural network, such as... Figure 5 As shown, to ensure the trainability of deep networks, the Transformer uses residual connections and layer normalization after each sub-layer. Specifically, after the input passes through a sub-layer (Attention or FFN) to obtain the output, it is added to the original input (this is the residual connection), and then layer normalization is performed. The working process of the Transformer encoder is as follows: (1) Input processing: The embedding layer is the first step in input processing. It maps each token in the input sequence to a dense vector of fixed dimensions. The embedding vector of each token captures the semantic information of the token, so that tokens with similar semantics are close to each other in the embedding space. Position encoding introduces the position information of the token in the sequence into the model. In this way, the model can learn the sequential relationship between tokens, which is crucial for understanding the syntax and semantics of a language.

[0039] (2) Encoder body: The encoder body consists of N encoder layers stacked together. Each layer contains a multi-head self-attention sub-layer and a feedforward neural network sub-layer. Each sub-layer is followed by a residual connection and layer normalization.

[0040] Specifically, the Transformer encoder module includes a multi-head self-attention layer, a feedforward neural network, and a residual connection and layer normalization layer; Multi-Head Self-Attention (MHA) is an extension of the self-attention mechanism widely used in Transformer models. It obtains the attention distribution of different subspaces of the input sequence by running multiple independent self-attention mechanisms in parallel, thereby capturing more comprehensive potential semantic associations in the sequence.

[0041] like Figure 6 As shown, the multi-head self-attention layer consists of a linear mapping layer, a separate processing layer, a scaled dot-product attention layer, a concatenation layer, and a linear transformation layer. Its specific workflow is as follows: The linear mapping layer is used to perform linear mapping processing on the enhanced high-dimensional feature sequence to obtain three different vector spaces; Specifically, the linear mapping layer takes the vectors output by the embedding layer and the positional encoding layer and passes them through three sets of independent, learnable weight matrices. This maps the input vector to three different vector spaces, mathematically represented as: (4) in, The embedding matrix of the input sequence, For query vector, For key vectors, It is a value vector; The split processing layer is used to split the different vector spaces to obtain multiple sub-matrices; Specifically, processing separately means... The model is split into h sub-matrices, which are h independent “attention heads” (this application uses eight attention heads, i.e., h=8). The purpose is to enable the model to learn the semantic relationships of multiple different subspaces in parallel at the same layer, thus breaking through the expression bottleneck of “single attention distribution” of single-head attention.

[0042] The scaling dot product attention layer is used to perform scaling dot product attention processing on each submatrix to obtain a scaled submatrix; Specifically, for each attention head i, in its corresponding subspace, "scaled dot product attention" is independently performed to compute its attention output, capturing different types of semantic relations, which is mathematically represented as follows: (5) in, It is a scaling factor used to alleviate the gradient saturation problem caused by excessively large dot product results. Softmax normalization converts attention scores into weights. The concatenation and linear transformation layer is used to concatenate and linearly transform multiple scaled sub-matrices to obtain a multi-head attention output matrix. Specifically, The output of each attention head The data is concatenated into a long vector, then subjected to a final linear transformation to integrate information from different heads, resulting in the final multi-head attention output, mathematically represented as: (6) in, This is a learnable weight matrix.

[0043] In the Transformer encoder, the feedforward neural network is a key sublayer that works in conjunction with the attention mechanism to introduce the necessary nonlinearity to the model, enabling it to learn and represent more complex patterns, while also "deeply processing" information to generate richer and more discriminative high-level semantic features.

[0044] A feedforward neural network consists of two linear transformation layers and an activation function (usually ReLU). The specific process is as follows: A feedforward neural network includes a first linear transformation layer, a non-linear activation function, and a second linear transformation layer; The first linear transformation layer is used to perform linear transformation on the multi-head attention output matrix to obtain a high-dimensional multi-head attention output matrix; The nonlinear activation function is used to perform nonlinear activation on the high-dimensional multi-head attention output matrix to obtain a nonlinear high-dimensional multi-head attention output matrix. The second linear transformation layer is used to perform a linear transformation on the nonlinear high-dimensional multi-head attention output matrix to obtain a low-dimensional multi-head attention output matrix with the same dimension as the multi-head attention output matrix; In other words, the first linear transformation layer of the feedforward neural network projects the input to a higher dimension; the nonlinear activation uses the ReLU activation function to introduce nonlinearity; and the second nonlinear transformation projects the high-dimensional representation back to the original dimension.

[0045] The residual connection and layer normalization layer are used to perform residual connection and layer normalization processing on the low-dimensional multi-head attention output matrix to obtain the first feature sequence with contextual information; Specifically, residual connections allow the original input information to be directly passed to deeper layers, while layer normalization ensures that the input distribution of each layer remains relatively stable. The mathematical representation of residual connection is: (7) The mathematical representation of layer normalization is: (8) in, The mean, Standard deviation and These are learnable parameters.

[0046] The Connection-Time Classification (CTC) module in this application is primarily used to address the mismatch between input data and labels. This is because the feature sequence length of the input image for handwritten chemical equation recognition (typically...) is... (time steps) and the length of the output text sequence ( The number of characters (in words) is often inconsistent, and there is no strict alignment relationship between them. To solve this problem of variable-length sequence recognition, this application introduces a connection-time classification module. By introducing blank labels and a path probability modeling method of "multi-time-step output merging," the connection-time classification module can automatically learn the alignment relationship from the feature sequence to the label sequence, thereby achieving effective recognition without the need for precise segmentation and alignment annotation. This not only reduces the difficulty of data annotation but also makes model training more consistent with real-world application scenarios. Its workflow is as follows: In this application, the Semantic Guidance Module (SGM) is a method for modeling target characters using the surrounding string context. This method effectively guides the model to capture linguistic context. During the training phase, this module guides the visual model to integrate contextual information into the feature sequence; however, it is removed during the inference phase, thus improving recognition accuracy without increasing inference costs. Figure 4 As shown, the specific working process of the semantic guidance module is as follows: Used to obtain the left and right strings of the target character in the enhanced high-dimensional feature sequence, and to generate the context semantic hidden representation sequence of the left string and the context semantic hidden representation sequence of the right string; Specifically, during the training phase, for a given text image and its character label sequence... ,in Indicates the first One character. Its context information is defined as the target character. The left-hand string and the string on the right Next, with the left string Let's take an example to describe it.

[0047] (1) The character mapping in the code is a string embedding, which is mathematically represented as:

[0048] in, The feature dimension is consistent with the dimension of feature sequence B (enhanced high-dimensional feature sequence). Indicates the size of the context window.

[0049] The feature map of the left string is obtained by performing dot product attention calculation on the context semantic hidden representation sequence of the left string and the enhanced high-dimensional feature sequence; Specifically, these left-hand string embeddings are encoded to generate a hidden representation. It represents the left string. The hidden representation of the context semantics is mathematically represented as: (10) in, It is a predefined tag. It is a learnable weight matrix. This represents the softmax function. Presentation layer normalization operation.

[0050] Hidden representation With characteristic sequence B The left-hand string feature map is obtained by using dot product attention calculation. It reflects the correlation between the left-side context and various locations of visual features, and its mathematical representation is: (11) in, It consists of a learnable weight matrix and Obtained by transformation.

[0051] The left-side string feature map and the enhanced high-dimensional feature sequence are weighted to generate the left-side string context enhancement feature; Specifically, the left-hand string feature map Weight the feature sequence B to generate the corresponding characters. The characteristics of are mathematically represented as follows: (12) in, For feature maps, This is the weight matrix. For characteristic sequences, Enhance the context features of the left-hand string. Similarly, enhance the right-hand context string. Perform the same process to generate the corresponding features. .

[0052] The left-side string context enhancement features and the right-side string context enhancement features are classified to obtain the probability distribution of the left target character and the probability distribution of the right target character, respectively. Specifically, after obtaining the context-enhanced features on the left and right sides, the semantic guidance module uses a classifier to predict the probability distributions of the target characters on the left and right sides, respectively, which are mathematically represented as follows: , (13) in, Predict the character probability distributions for the left and right sides, respectively. It is a learnable weight matrix. For feature dimension, Number of character categories (including whitespace).

[0053] Calculate the cross-entropy loss function between the corresponding characters of the left and right strings and their predicted class probabilities; Specifically, during training, the semantic guidance module guides the model to learn correct character recognition through the cross-entropy loss function, which is mathematically represented as: (14) in For sequence length, for The loss function between; for The loss function between The loss function is semantically guided.

[0054] The connection time-series classification module described in this application includes an extended character set module, a feature and probability mapping module, a path mapping module, and a connection time-series classification loss calculation module; The extended character set module is used to extend the original chemical symbol character set to generate an extended chemical symbol character set with "blank" symbols; Specifically, the connection time-series classification module first expands the original chemical symbol character set. Let the original chemical symbol character set be... (Including element symbols, numbers, operators, etc.), the connection time series classification module introduces a special "blank" symbol. (Indicates "no output" or "character interval"), forming an extended chemical symbol character set. The introduction of whitespace symbols can, on the one hand, represent invalid output time steps and intervals between marker characters, and on the other hand, separate repetitive characters that exist in the target sequence itself, preventing them from being merged and eliminated during the decoding stage. This is the key to solving the problem of no pre-alignment between the input feature sequence and the label sequence.

[0055] The feature and probability mapping module is used to input the feature sequence into the linear transformation layer and map the feature dimensions to an extended chemical symbol character set. The corresponding dimension is used to calculate the feature sequence at each time step. The conditional probabilities of all symbols (including whitespace) in the corresponding extended character set are generated, and multiple sets of length are obtained. The character probability sequence path is output as a probability sequence, which serves as the input for subsequent path mapping and loss calculation; Specifically, after obtaining the feature sequence C output by the Transformer encoder, the temporal classification module first maps it to the space Z of the extended chemical symbol character set through a linear transformation layer, which is mathematically represented as:

[0056] in, and The weight matrix and bias terms are learnable. To expand the chemical symbol character set.

[0057] Next, the time-series classification module takes the feature sequence as its input. At each time step The probability of each character is output, resulting in a probability matrix. :

[0058] in, Representing the characteristic sequence At time step Predicted as character The conditional probability, Indicates time step At that time, the character The unnormalized score, Extended chemical symbol character set The sum of the scores of all characters in the text.

[0059] The path mapping module is used to convert the probability sequence into the final recognition result sequence through the rule of "merging consecutive identical characters and removing whitespace symbols". That is, the path mapping step is used to construct a mapping function and obtain an effective character probability sequence path that can output the target output sequence from the character probability sequence path based on the mapping function.

[0060] Specifically, the path mapping module at each time step Independently predict the probability of a character, resulting in a length of A sequence of character probabilities is called a "path", denoted as . ,in However, the path length is fixed, while the final required text sequence length is variable. Therefore, the connection-time classification module defines a many-to-one mapping function. This is used to compress the path into the final output sequence (target output sequence), a mapping function. The rules are as follows: Step 1: Merge consecutive identical characters in the path into one.

[0061] For example: in the path "aa-ε-bb-ε-c", "-" is used to separate time steps, after which... After mapping, duplicate characters are merged, and the path becomes "a-ε-b-ε-c".

[0062] Step 2: Delete all whitespace characters .

[0063] For example, removing whitespace from the "a-ε-b-ε-c" obtained in the previous step yields the final sequence "abc".

[0064] The connection time-series classification loss calculation module is used to calculate the loss function value of the target output sequence obtained from the valid character probability sequence path. This loss value is obtained by aggregating the sum of the probabilities of all paths that can be mapped to the real label sequence. This solves the alignment problem of inconsistent length between the model output sequence and the target label sequence. That is, the path probabilities of all valid time-series paths that can be mapped to the real target label sequence are accumulated to obtain the total conditional probability corresponding to the real label, and the negative logarithm is further calculated to obtain the CTC loss value. Specifically, during the training phase, for a given sequence of true labels... The goal of the connection time series classification module is to maximize all those that can be mapped. get The sum of the probabilities of the paths is equivalent to minimizing their negative log-likelihood, which can be mathematically described as:

[0065] in, Represents all that can be mapped by the mapping function Obtain the target sequence The set of paths is given. Directly enumerating all possible paths would be computationally expensive. Therefore, dynamic programming is used in the actual implementation for computation. This is achieved by minimizing the connection-time classification loss. In this way, the model learns to extract ordered text information from disordered feature sequences, that is, the model learns to recognize the correct character sequences in chemical formulas.

[0066] The total loss of the model during training. Loss due to CTC and the loss of semantic guidance CE Together they constitute, and their mathematical representation is: (18) in, It is a hyperparameter that balances the two loss terms, and it is set to 0.05 in the experiments in this chapter.

[0067] During the training phase, the model's total loss is a weighted combination of the connection temporal classification loss and the cross-entropy loss function output by the semantic guidance module, in order to simultaneously optimize sequence alignment accuracy and language semantic expressive power; during the inference phase, the connection temporal classification module acts as a decoder to convert the output probability sequence into the final recognition result.

[0068] Furthermore, the augmentation process includes rotating and adding noise to the handwritten chemical equation image. Specifically, to improve the model's generalization ability and robustness, data augmentation was performed on the handwritten chemical equation using the following method: (1) such as Figure 7 As shown, the original image 1 is read and rotated by 90 degrees, 180 degrees, and 270 degrees respectively to obtain the corresponding rotated augmented images 2, 3, and 4, as shown. Figure 8 , Figure 9 and Figure 10 As shown: (2) Add noise to image 1 to obtain augmented image 5, as shown. Figure 11 As shown, the above process increases the amount of data, which in turn improves the model's generalization ability and robustness.

[0069] The model constructed using the Transformer-CRNN hybrid network-based handwritten chemical equation recognition model disclosed in this invention enables offline training of the handwritten chemical equation recognition algorithm. Specifically, the augmented handwritten chemical equation images are input into the Transformer-CRNN hybrid network-based handwritten chemical equation recognition model constructed in this application, and the model is trained offline. In this embodiment, a self-built dataset is used for training. The training set contains 2366 handwritten chemical equations, and after data augmentation, there are a total of 4732 chemical equations. Furthermore, this invention employs the CTC loss function and a semantically guided loss function, uses full training, conducts 164 training rounds with a batch size of 64, and uses AdamW as the optimizer.

[0070] The model constructed using the method disclosed in this invention for building a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network can also be used for online recognition of handwritten chemical equations. like Figure 1 As shown, the handwritten chemical equation image is input into the handwritten chemical equation recognition model built through an offline training process for recognition, and the recognition result is output.

[0071] The beneficial effects of this method are: 1. This invention improves the CRNN model: A Transformer encoder module is added to the CRNN model. The multi-head attention module in the Transformer encoder module adds multiple "attention heads" to the ordinary attention mechanism, enabling the model to better learn and represent information. It can take into account both the local morphological features and global semantic information of handwritten chemical equations, avoiding the omission of key features by a single attention mechanism, capturing global dependencies, and improving the model's recognition accuracy. 2. A semantic guidance module has been added: This module can be added as a "plug-and-play" component to the existing training process. By revealing the "correct recognition result," it continuously optimizes the network model and improves the model's recognition accuracy. The semantic guidance module is removed during inference, so it does not affect the model's inference speed at all. Therefore, both its recognition speed and recognition accuracy are improved.

[0072] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network, characterized in that, Includes the following steps: Step 1: Construct a handwritten chemical equation recognition model; The handwritten chemical equation recognition model includes a feature extraction module, a deep bidirectional recurrent neural network module, a Transformer encoder module, a semantic guidance module, and a connection-time classification module. The feature extraction module is used to perform feature extraction processing on the handwritten chemical equation image to obtain a high-dimensional feature sequence of the handwritten chemical equation image; The deep bidirectional recurrent neural network module is used to perform enhanced feature processing on the high-dimensional feature sequence to obtain an enhanced high-dimensional feature sequence. The Transformer encoder module is used to encode the enhanced high-dimensional feature sequence to obtain a feature sequence with contextual information. The semantic guidance module is used to perform semantic guidance processing on the enhanced high-dimensional feature sequence during the training phase to obtain a cross-entropy loss function with contextual information; The connection temporal classification module is used to perform connection temporal classification processing on feature sequences with contextual information and output the correct recognition result; Step 2: Obtain the image of the handwritten chemical equation and perform data augmentation processing on the handwritten chemical equation image to obtain the augmented handwritten chemical equation image; Step 3: Input the acquired augmented handwritten chemical equation image into the handwritten chemical equation recognition model for training to obtain the trained handwritten chemical equation recognition model.

2. The method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network according to claim 1, characterized in that: The feature extraction module includes convolutional unit 1, pooling layer 1, convolutional unit 2, pooling layer 2, convolutional unit 3, convolutional unit 4, pooling layer 3, convolutional unit 5, convolutional unit 6, pooling layer 4, and convolutional unit 7; The first convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to extract the edges, stroke direction, and local texture features in the handwritten chemical equation image to obtain a low-level feature map. The first pooling layer consists of a 2×2 max pooling layer, which is used to perform downsampling processing on the low-level feature map to obtain a downsampled feature map; The second convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to enhance the local character morphology to obtain a low-level enhanced feature map; The second pooling layer consists of a 2×2 max pooling layer, which is used to perform downsampling processing on the low-level enhanced feature map to obtain the low-level enhanced downsampled feature map; The third convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function. It is used to perform nonlinear mapping processing on the downsampled feature map after the low-level enhancement to obtain the mid-level feature map of the structural patterns of element symbols, numbers, superscripts and subscripts, and reaction arrows. The convolutional unit 4 consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to perform nonlinear mapping processing on the middle layer feature map to obtain the deep layer feature map; The third pooling layer consists of a 2×2 max pooling layer, which is used to compress the size of the deep feature map to improve the robustness of the model to local deformation and differences in writing style, and to obtain the compressed deep feature map. The fifth convolutional unit consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function. It is used to perform feature abstraction on the complex chemical symbol combinations in the compressed deep feature image to obtain a deep feature image after feature abstraction. The convolutional unit six consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to perform feature sampling processing on the deep feature image after the first feature abstraction to obtain the deep feature image after the second feature deep abstraction. The pooling layer four consists of 2×2 max pooling layers, which are used to perform spatial downsampling processing on the deep feature image after the deep abstraction of the secondary features to obtain the downsampled deep feature image; The convolutional unit seven consists of a 3×3 convolutional layer, a normalization layer, and a ReLU activation function, and is used to integrate and enhance the downsampled deep feature image to obtain a high-dimensional feature sequence of the handwritten chemical equation image.

3. The method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network according to claim 2, characterized in that: The deep bidirectional recurrent neural network module includes a bidirectional recurrent neural network and a deep recurrent neural network; The bidirectional recurrent neural network is used to obtain and concatenate the high-dimensional feature sequence of the handwritten chemical equation image to obtain a high-level abstract feature map by acquiring and concatenating the forward and reverse hidden states. The deep recurrent neural network is used to stack the high-level abstract feature maps to obtain an enhanced high-dimensional feature sequence.

4. The method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network according to claim 3, characterized in that: The Transformer encoder module includes an input preprocessing layer, a multi-head self-attention layer, a feedforward neural network, and a residual connection and layer normalization layer. The input preprocessing layer includes a word embedding layer and a position encoding layer. The word embedding layer is used to linearly map the enhanced high-dimensional feature sequence output by the deep bidirectional recurrent neural network module to the fixed-dimensional space required by the multi-head self-attention layer. The position encoding layer is used to inject absolute or relative position information into the fixed-dimensional space to obtain a fixed-dimensional space with position information. The multi-head self-attention layer includes a linear mapping layer, a split-head processing layer, a scaled dot product attention layer, and a splicing and linear transformation layer; The linear mapping layer is used to perform linear mapping processing on the fixed-dimensional space of the location information to obtain three different vector spaces; The split processing layer is used to split the different vector spaces to obtain multiple sub-matrices; The scaling dot product attention layer is used to perform scaling dot product attention processing on each submatrix to obtain a scaled submatrix; The concatenation and linear transformation layer is used to concatenate and linearly transform multiple scaled sub-matrices to obtain a multi-head attention output matrix. The feedforward neural network includes a first linear transformation layer, a nonlinear activation function, and a second linear transformation layer; The first linear transformation layer is used to perform linear transformation on the multi-head attention output matrix to obtain a high-dimensional multi-head attention output matrix; The nonlinear activation function is used to perform nonlinear activation on the high-dimensional multi-head attention output matrix to obtain a nonlinear high-dimensional multi-head attention output matrix. The second linear transformation layer is used to perform a linear transformation on the nonlinear high-dimensional multi-head attention output matrix to obtain a low-dimensional multi-head attention output matrix with the same dimension as the multi-head attention output matrix; The residual connection and layer normalization layer are used to perform residual connection and layer normalization processing on the low-dimensional multi-head attention output matrix to obtain the feature sequence with contextual information.

5. The method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network according to claim 4, characterized in that: The semantic guidance module is used to perform semantic guidance processing on the enhanced high-dimensional feature sequence during the training phase to obtain a cross-entropy loss function with contextual information. The specific process is as follows: Used to obtain the left and right strings of the target character in the enhanced high-dimensional feature sequence, and to generate the context semantic hidden representation sequence of the left string and the context semantic hidden representation sequence of the right string; Dot product attention is performed on the context semantic hidden representation sequence of the left string and the context semantic hidden representation sequence of the right string with the enhanced high-dimensional feature sequence after transformation by the learned weight matrix to obtain the attention map of the left string and the attention map of the right string. The attention maps of the left and right strings are weighted and processed with the enhanced high-dimensional feature sequence to generate features for the characters corresponding to the characters of the left and right strings, respectively. The features of the characters corresponding to the left string and the features of the characters corresponding to the right string are processed by a classifier to obtain the predicted class probability of the characters corresponding to the left string and the predicted class probability of the characters corresponding to the right string. Calculate the cross-entropy loss function between the corresponding characters in the left and right strings and their predicted class probabilities.

6. The method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network according to claim 4, characterized in that: The connection-time classification module includes an extended character set module, a feature and probability mapping module, a path mapping module, and a connection-time classification loss calculation module. The extended character set module is used to extend the original chemical symbol character set to generate an extended chemical symbol character set with "blank" symbols; The feature and probability mapping module is used to map the feature sequence to the extended chemical symbol character set through a linear transformation layer to generate a feature sequence mapping space; Calculate the feature sequence at each time step Output the probability of each chemical symbol character and generate multiple sets of length [value missing]. Character probability sequence path; The path mapping module is used to construct a mapping function and obtain, based on the mapping function, an effective character probability sequence path that can output the target output sequence from the character probability sequence path; The connection time-series classification loss calculation module is used to calculate the sum of the loss function value obtained from the effective character probability sequence path to the target output sequence and the cross-entropy loss function output by the semantic guidance module, distributed proportionally.

7. The method for constructing a handwritten chemical equation recognition model based on a Transformer-CRNN hybrid network according to claim 6, characterized in that: The augmentation process includes rotating the handwritten chemical equation image and adding noise.

Images