Formula recognition method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure provides a formula recognition method and device, equipment, storage medium and program product, relates to computer vision, deep learning, image processing and other artificial intelligence technical fields, and can be applied to intelligent finance, intelligent government affairs and other scenes. The method comprises the following steps: obtaining a feature map of an image to be recognized containing a formula by using an encoding network; sequentially performing decoding operation on the features in the feature map by using a decoding network; processing the feature encoding of the feature map and the decoded formula symbol, and inputting the output current hidden layer state and the feature map into a multi-directional local attention module again to obtain a region image of the formula symbol to be decoded, and processing the region image, the feature encoding, the current hidden layer state and a line count heat map to obtain an output formula symbol recognition result; and generating a formula symbol sequence from the sequentially output formula symbol recognition results.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of computer vision, deep learning, and image processing, and particularly to a formula recognition method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Document OCR (Optical Character Recognition) functions are widely needed in education, office work, and library archives.

[0003] For a document OCR engine, recognizing formulas within a document is crucial. Unlike regular text, formulas often have complex nested structures, primarily due to their unique fractional, subscript, and square root structures. Recognizing multi-line formulas is particularly challenging. Summary of the Invention

[0004] This disclosure provides a formula recognition method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0005] In a first aspect, embodiments of this disclosure propose a formula recognition method, comprising: acquiring an image to be recognized containing multiple lines of formulas, and obtaining a feature map of the image to be recognized using an encoding network in a preset formula recognition model; performing decoding operations on the features in the feature map sequentially using a decoding network in the preset formula recognition model, wherein the features in the feature map correspond to different formula symbols respectively; inputting the feature map and the feature encoding of the decoded formula symbols obtained through the decoding operation into the decoding network to obtain the current hidden layer state; wherein the current hidden layer state corresponding to the first formula symbol is preset to an initial value; inputting the feature map and the current hidden layer state into a multi-directional local attention module in the decoding network to obtain a region image corresponding to the current formula symbol to be decoded; combining and predicting the region image, feature encoding, and current hidden layer state through the decoding network to obtain a formula symbol recognition result corresponding to the current decoding operation; and generating a formula symbol sequence based on the formula symbol recognition results sequentially output by the decoding network.

[0006] Secondly, embodiments of this disclosure propose a formula recognition device, comprising: a feature map extraction unit configured to acquire an image to be recognized containing multiple lines of formulas, and to obtain a feature map of the image to be recognized using an encoding network in a preset formula recognition model; a feature decoding unit configured to perform decoding operations on the features in the feature map sequentially according to the following sub-units using a decoding network in the preset formula recognition model, wherein the features in the feature map correspond to different formula symbols; a current hidden layer state acquisition sub-unit configured to input the feature map and the feature encoding of the decoded formula symbol obtained through the decoding operation into the decoding network to obtain the current hidden layer state; wherein the current hidden layer state corresponding to the first formula symbol is preset to an initial value; a region image determination sub-unit configured to input the feature map and the current hidden layer state into a multi-directional local attention module in the decoding network to obtain a region image corresponding to the current formula symbol to be decoded; a feature combination and prediction sub-unit configured to combine and predict the region image, feature encoding, and current hidden layer state through the decoding network to obtain a formula symbol recognition result corresponding to the current decoding operation; and a formula symbol sequence generation unit configured to generate a formula symbol sequence based on the formula symbol recognition results sequentially output by the decoding network.

[0007] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the formula recognition method as described in the first aspect.

[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the formula recognition method as described in the first aspect when executed.

[0009] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the steps of the formula recognition method as described in the first aspect.

[0010] The formula recognition scheme provided in this disclosure, based on the conventional overall network architecture consisting of an encoder-decoder type network, for long sequence data such as complex multi-line formulas, not only processes the feature map output by the encoder network and the feature encoding of the decoded formula symbols obtained through the decoding operation through the decoding network, so that the output current hidden layer state can more accurately describe the relevant feature information of the formula symbols in the current step, but also replaces the attention module of the decoding network, which usually has only a single attention weight channel, with a multi-directional local attention module with multiple attention weight channels in different directions. This allows the scheme to capture the differences of different formula symbols in different directions when recognizing multiple formula symbols contained in a local region, thereby improving the recognition accuracy of complex multi-line formulas.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0013] Figure 1 This is an exemplary system architecture to which this disclosure can be applied;

[0014] Figure 2 A flowchart of a formula recognition method provided in this embodiment of the disclosure;

[0015] Figure 3 This is a schematic diagram illustrating how a multi-directional attention weight channel provided by a multi-directional local attention module is used to determine the attention weights of different formula symbols, as provided in an embodiment of this disclosure.

[0016] Figure 4 A flowchart illustrating the processing features of a multi-directional local attention module provided in this embodiment of the disclosure;

[0017] Figure 5 A flowchart of another formula recognition method provided in this disclosure embodiment;

[0018] Figure 6 A flowchart illustrating a method for obtaining a row count heatmap using a global row attention module, provided in this embodiment of the disclosure;

[0019] Figure 7 This is a schematic diagram illustrating a process of obtaining formula recognition results by processing an image to be recognized through a formula recognition model, as provided in an embodiment of this disclosure.

[0020] Figure 8 A flowchart illustrating a method for obtaining complete text recognition results based on a sequence of formula symbols, provided in this embodiment of the disclosure;

[0021] Figure 9 A structural block diagram of a formula recognition device provided in an embodiment of this disclosure;

[0022] Figure 10 This is a schematic diagram of the structure of an electronic device suitable for performing a formula recognition method, provided as an embodiment of the present disclosure. Detailed Implementation

[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0024] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of formula recognition methods, apparatuses, electronic devices, and computer-readable storage media of the present disclosure can be applied.

[0026] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0027] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include document OCR recognition applications, model training applications, and instant messaging applications.

[0028] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0029] Server 105 can provide various services through its built-in applications. Taking a document OCR recognition application as an example, when running this application, server 105 can achieve the following: First, it receives an image containing multiple formulas from terminal devices 101, 102, and 103 via network 104. Then, it uses the encoding network in the preset formula recognition model to obtain a feature map of the image. Next, it uses the decoding network in the preset formula recognition model to decode the features in the feature map in the following manner, where each feature in the feature map corresponds to a different formula symbol: the feature map and the decoded formula symbols obtained through the decoding operation are then combined. The feature encoding is input into the gated recurrent module in the decoding network to obtain the current hidden layer state. The current hidden layer state corresponding to the first formula symbol is pre-set to an initial value. The feature map and the current hidden layer state are input into the multi-directional local attention module in the decoding network to obtain the region image corresponding to the formula symbol to be decoded. The multi-directional local attention module has multiple attention weight channels corresponding to different directions. The region image, feature encoding, and current hidden layer state are combined and predicted through the fully connected layer in the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation. Finally, a formula symbol sequence is generated based on the formula symbol recognition results output sequentially by the decoding network.

[0030] It should be noted that the image to be recognized, which contains multiple lines of formulas, can be temporarily obtained from terminal devices 101, 102, and 103 via network 104, or it can be pre-stored locally on server 105 in various ways. Therefore, when server 105 detects that this data is already stored locally (e.g., when starting to process previously stored OCR recognition tasks for documents), it can choose to directly obtain this data from locally. In this case, the exemplary system architecture 100 may also exclude terminal devices 101, 102, and 103 and network 104.

[0031] Because accurate recognition of images containing multiple lines of formulas requires significant computing resources and power, the formula recognition methods provided in the subsequent embodiments of this disclosure are generally executed by a server 105 with strong computing power and abundant computing resources. Correspondingly, the formula recognition device is also generally located within the server 105. However, it should also be noted that when terminal devices 101, 102, and 103 also possess sufficient computing power and resources, they can also perform the aforementioned calculations performed by the server 105 through document OCR recognition applications installed on them, thereby outputting the same results as the server 105. Especially when multiple terminal devices with different computing capabilities exist simultaneously, but the document OCR recognition application determines that the terminal device has strong computing power and abundant remaining computing resources, it can allow the terminal device to perform the aforementioned calculations, thereby appropriately reducing the computing pressure on the server 105. Accordingly, the formula recognition device can also be located within terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also exclude the server 105 and the network 104.

[0032] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0033] Please refer to Figure 2 , Figure 2 A flowchart of a formula recognition method provided in this disclosure embodiment, wherein process 200 includes the following steps:

[0034] Step 201: Obtain the image to be recognized containing multiple lines of formulas, and use the encoding network in the preset formula recognition model to obtain the feature map of the image to be recognized;

[0035] This step aims to identify the subject executing the method (e.g., ...) using the formula. Figure 1 The server 105 shown obtains (e.g., received from) Figure 1 The terminal devices 101, 102, and 103 shown input images containing multiple lines of formulas to be recognized, and input these images as input data into a preset formula recognition model. The formula recognition model is a network structure of an encoder-decoder network. Therefore, the images to be recognized will first be encoded by the encoding network to obtain a feature map corresponding to the images to be recognized.

[0036] The image to be identified can be a photograph taken by a user with a terminal device that has a shooting function (such as a mobile phone, tablet computer, or smart camera) of a book or document containing multiple lines of complex formulas, or it can be downloaded directly from the Internet or a cloud database, or it can be extracted from other image files. No specific limitation is made here.

[0037] The formula recognition model used in this embodiment is an encoder-decoder network, also known as an encoder-decoder network. It is a type of deep learning architecture whose core idea is to encode the input data into a latent representation (usually low-dimensional) and then decode it back to the original or target data. This type of network has the following characteristics:

[0038] 1) Encoder: The first part of an encoder-decoder network is the encoder, which is responsible for transforming the input data into a latent representation. An encoder typically consists of multiple convolutional layers, pooling layers, and fully connected layers to extract features from the raw data and map them to a low-dimensional representation;

[0039] 2) Latent Representation: The latent representation is the output of the encoder and typically has low dimensionality. This representation captures key features of the input data, mapping the data from a high-dimensional space to a more compact representation space;

[0040] 3) Decoder: The decoder is the second part of the network, responsible for decoding the latent representation into the original data or target data. A decoder typically includes multiple deconvolutional layers (or transposed convolutional layers), upsampling layers, and fully connected layers to generate the target data from the latent representation.

[0041] 4) Autoencoder: A common form of encoder-decoder network is the autoencoder, which aims to learn a compact data representation while reconstructing the input data to the greatest extent possible. An autoencoder typically consists of an encoder and a decoder, minimizing reconstruction error during training.

[0042] These types of networks are commonly used for image denoising (where the encoder encodes a noisy image into a latent representation, and the decoder decodes the latent representation into a denoised image), image generation (Generative Adversarial Networks (GANs) are an extension of the encoder-decoder structure and are often used for image generation tasks such as generating realistic images and image super-resolution), speech processing (in the field of speech processing, encoder-decoder networks can be used for tasks such as speech synthesis, speech recognition, and speech denoising), and sequence-to-sequence tasks (encoder-decoder architectures are also widely used for sequence-to-sequence tasks such as machine translation, text summarization, and speech recognition, where the encoder encodes the input sequence into a latent representation). In scenarios such as decoder decoding of latent representation into target sequence, dimensionality reduction and feature learning (encoder-decoder networks such as autoencoders can be used for data dimensionality reduction and feature learning, which helps to extract useful information from the data), and anomaly detection (encoder-decoder networks can be used for anomaly detection, where normal data is successfully reconstructed, while anomalous data cannot be reconstructed well because it is different from normal data), the embodiments of this disclosure utilize the above-mentioned characteristics of such networks to accurately identify multiple complex formulas contained in an image. That is, the input of the formula recognition model is the image to be recognized containing multiple formulas, and the output is the symbol sequence that constitutes the multiple formulas.

[0043] In addition, considering that sometimes the encoding network and the decoding network are played by different models, there may be a problem that the actual output of the encoding network does not match the required input of the decoding network. For example, the mismatch may occur in terms of feature map size and number of feature map channels. In order to meet the required input of the decoding network, it is necessary to adjust the size or number of channels of the feature map output by the encoding network. For example, when the number of channels of the feature map output by the encoding network is more than the number of channels of the input feature map required by the decoding network, the number of channels can be reduced to the required number of channels by using a first convolution including a 1×1 convolution.

[0044] In addition, before inputting the image to be recognized into the encoding network, the image can be preprocessed, which may include at least one of the following: orientation correction processing, binarization processing, grayscale processing, contrast processing (e.g., contrast enhancement processing), and sharpness processing (e.g., sharpness enhancement processing), so that the preprocessing highlights the part of the image containing multiple lines of formulas, which helps to improve the accuracy of the recognition results.

[0045] Step 202: Use a decoding network to decode the features in the feature map sequentially as follows, where each feature in the feature map corresponds to a different formula symbol:

[0046] This step involves using a decoding network to sequentially decode the features that correspond to different formula symbols (also known as performing decoding step by step, with each step corresponding to a different formula symbol), in order to obtain the formula symbol recognition result corresponding to the current decoding through each decoding operation.

[0047] Step 2021: Input the feature map and the feature encoding of the decoded formula symbols obtained through the decoding operation into the decoding network to obtain the current hidden layer state;

[0048] This step aims to have the aforementioned execution entity input the feature map and the feature encoding of the decoded formula symbols obtained through the decoding operation into the decoding network, thereby obtaining the current hidden layer state of the output.

[0049] One implementation method, including but not limited to, involves inputting the feature map and feature encoding into a gated recurrent module in the decoding network to obtain the current hidden layer state output by the gated recurrent module. This current hidden state is calculated based on the hidden state of the previous time step, the input of the current time step, and the output of the gating mechanism (update gate and reset gate). This hidden state contains the model's representation and memory information at a given time step.

[0050] It should be noted that since the decoding operation is performed sequentially and step-by-step, the encoded features of the decoded formula symbols used in the current decoding operation should, as far as possible, utilize the feature codes obtained from the previous decoding operation adjacent to the current operation. This is to improve the accuracy of the current hidden layer state obtained in this decoding operation by leveraging the closest possible contextual relationship. Furthermore, considering that the first formula symbol will not have any decoded formula symbol feature codes available for it during this decoding step, the current hidden layer state corresponding to the first formula symbol can be pre-set to an initial value.

[0051] The Gated Recurrent Unit (GRU) is a variant of the Recurrent Neural Network (RNN) used to process sequential data, such as natural language text and time series data. The main function of the GRU unit is to address the long-term dependency problem inherent in standard RNNs, and it has the following important functions and roles:

[0052] 1) Capturing sequence information: The core function of the GRU unit is to capture information in sequence data, and it can retain and pass on information from previous time steps so that it can be used for prediction or classification tasks in subsequent time steps.

[0053] 2) Avoiding the vanishing gradient problem: Compared to standard RNNs, GRU employs a more complex gating mechanism to mitigate the vanishing gradient problem. This makes the model easier to train when dealing with long sequences because it can better propagate gradient information;

[0054] 3) Gating Mechanisms: GRU includes two important gating mechanisms: the Update Gate and the Reset Gate. These gating mechanisms allow the GRU unit to decide which information to update at each time step, thus enabling it to adapt more flexibly to different sequence data;

[0055] 4) Update Gate: Determines whether the previous hidden state should be retained at the current time step. The sigmoid function controls which information should be retained (value close to 1) and which should be discarded (value close to 0).

[0056] 5) Reset Gate: Controls whether previous hidden states should be ignored at the current time step. It uses the Sigmoid function (also known as the S-shaped function) to determine which information should be forgotten (value close to 0) and which should be considered (value close to 1).

[0057] 6) Memory capacity: The gating mechanism of the GRU unit enables it to selectively remember or forget information, thereby better adapting to the characteristics of sequence data, which helps to process sequences of various lengths.

[0058] This step leverages the aforementioned characteristics and capabilities of the gated loop module. By inputting the feature map and the feature encoding of the decoded formula symbol obtained through decoding operations (e.g., assuming the current prediction is for the formula symbol in step 9, the feature encoding of the decoded formula symbol would be the feature encoding of the formula symbol in step 8), the gated loop module, with its long sequence data corresponding to multi-line formulas and its gating mechanism, can better capture and manage sequence information. This allows the output current hidden layer state to more accurately describe the relevant feature information of the formula symbol in the current step, especially considering the correlation between the current formula symbol and the decoded formula symbols. Of course, besides using the gated loop module, it can be replaced with other functional modules that have similar effects; no specific limitations are made here.

[0059] Step 2022: Input the feature map and the current hidden layer state into the multi-directional local attention module in the decoding network to obtain the region image corresponding to the formula symbol to be decoded;

[0060] Building upon step 2021, this step aims to have the aforementioned execution entity input the feature map and the current hidden layer state into the multi-directional local attention module in the decoding network, thereby obtaining the region image output by the module that corresponds to the formula symbol to be decoded.

[0061] The multi-directional local attention module mentioned in this step is different from the attention module that is usually present or standard in conventional or standard decoding networks. Conventional attention modules are often equipped with only one attention weight channel. Therefore, they can only assign weights to all features of the input module according to the attention mechanism through a single direction. However, since complex multi-line formulas often have multiple formula symbols in close proximity in a certain local area, if a conventional attention module with only one attention weight channel is used to process them, these different formula symbols may all have similar weight parameters from its perspective, thus failing to identify the differences between different formula symbols and leading to formula symbol recognition errors. The multi-directional local attention module provided in this disclosure is designed to address the above-mentioned problems that are prone to occur in the recognition of complex multi-line formulas by conventional attention modules. It is equipped with multiple attention weight channels corresponding to different directions, so that the complex multi-line formulas are processed through different attention weight channels from different directions, paying more attention to the local area. This allows it to find the relevant weight parameters that highlight the differences between different formulas from multiple directions, thereby improving the recognition accuracy of each formula symbol in complex multi-line formulas.

[0062] Specifically, a preferred example could be that the multi-directional local attention module has attention weight channels corresponding to the four directions of up, down, left, and right, respectively. See [reference needed] for an example. Figure 3 The diagram shown is as follows. Figure 3 The left side shows the attention weight recognition results of a conventional attention module equipped with only a single attention weight channel for different formula symbols under grayscale coverage. It can be seen that because these three formula symbols are close in position and similar in size, they are identified as having the same attention weight. Figure 3 On the right is the recognition result of the multi-directional local attention module equipped with four different attention weight channels (up, down, left, and right) for different formula symbols under the same grayscale coverage area. It can be seen that these three formula symbols are recognized with significant differences in different directions. Based on these amplified differences, they can be better identified as different formula symbols, thereby improving the recognition accuracy.

[0063] Based on this, this step inputs the feature map and the current hidden layer state output by the gated recurrent unit into the multi-directional local attention module. This fully combines the provided attention recognition and weight adjustment mechanism for multiple different directions to determine the context of the formula symbol to be decoded in the current step. In other words, it obtains the region image corresponding to the formula symbol to be decoded in the current step, so as to further improve the recognition accuracy of the formula symbol to be decoded in the current step and reduce the influence of formula symbols in other regions of the image while narrowing the scope of attention.

[0064] Step 2023: Combine and predict the region image, feature encoding, and current hidden layer state through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation.

[0065] Building upon step 2022, this step aims to have the aforementioned executing entity perform feature mapping and combination of the region image, feature encoding, and current hidden layer state through the fully connected layer in the decoding network, and then perform prediction processing on the combined features to obtain the output formula symbol sequence.

[0066] One specific implementation method could be:

[0067] First, the region image, the feature encoding, and the current hidden layer state are processed through the first fully connected layer in the decoding network for feature mapping and combination of the mapped features to obtain combined features. Then, the combined features are processed through the second fully connected layer in the decoding network for feature prediction to obtain the formula symbol recognition result corresponding to the current decoding operation.

[0068] Step 203: Generate a formula symbol sequence based on the formula symbol recognition results output sequentially by the decoding network.

[0069] Building upon step 2023, this step aims to have the aforementioned executing entity organize the formula symbol recognition results output by the decoding network in each sequential decoding operation into a formula symbol sequence.

[0070] The formula recognition method provided in this disclosure, based on the conventional overall network architecture consisting of an encoder-decoder type network, for long sequence data such as complex multi-line formulas, not only processes the feature map output by the encoder network and the feature encoding of the decoded formula symbols obtained through the decoding operation through the decoding network, so that the output current hidden layer state can more accurately describe the relevant feature information of the formula symbols in the current step, but also replaces the attention module of the decoding network, which usually has only a single attention weight channel, with a multi-directional local attention module with multiple attention weight channels in different directions. This allows the method to capture the differences of different formula symbols in different directions when recognizing multiple formula symbols contained in a local region, thereby improving the recognition accuracy of complex multi-line formulas.

[0071] Based on the above embodiments, to enhance the understanding of how the region image is obtained through the multi-directional local attention module, this embodiment also... Figure 4 A specific implementation is shown, and its process 400 includes the following steps:

[0072] Step 401: Input the current hidden layer state into the first fully connected layer in the multi-directional local attention module to obtain the processed features;

[0073] This step aims to have the aforementioned execution entity input the current hidden layer state into the fully connected layer, so that the input features can be mapped to a higher-level representation space using a set of weight parameters learned by the fully connected layer. By stacking multiple fully connected layers, more complex feature extraction and abstraction can be achieved. As the model depth increases, the fully connected layers can introduce nonlinear transformations through nonlinear activation functions, thereby better fitting nonlinear relationships.

[0074] Step 402: Input the feature map into multiple branch structures in the multi-directional local attention module, and use convolution kernels of different sizes in different branch structures to extract features by summing the attention weight features of each decoded formula symbol, so as to obtain the target attention weight features corresponding to the formula symbol to be decoded.

[0075] This step aims to enhance the ability of the execution entity to perceive image features of different sizes in the feature map by using multiple convolutional kernels of different sizes (i.e., enriching the receptive field by using convolutional kernels of different sizes). At the same time, it extracts features by summing the attention weight features of each previously decoded formula symbol, thereby obtaining the target attention weight features corresponding to the formula symbol to be decoded, so as to improve feature accuracy.

[0076] Among them, the convolution kernels of different sizes include at least a first-size convolution kernel larger than the size of the feature map and a second-size convolution kernel smaller than the size of the feature map. Assuming the size of the feature map is 8×8, then the first-size convolution kernel can be 11×11 and the second-size convolution kernel can be 3×3.

[0077] Step 403: Input the processed features, target attention weight features, and feature map into the stacking submodule of the multi-directional local attention module in sequence to obtain the stacked features;

[0078] Step 404: Input the superimposed features into the second fully connected layer in the multi-directional local attention module to obtain the attention weights;

[0079] Step 405: Input the attention weights and feature maps into the multiplication submodule of the multidirectional local attention module to obtain the region image corresponding to the formula symbol to be decoded.

[0080] Steps 403-405 first use the overlay submodule to overlay the processed features, attention weight features, and feature maps. Then, a fully connected layer is used to combine and interact these features, thereby learning the correlation between features and improving the accuracy of the final attention weight expression. Finally, a multiplication operation is used to ensure that the calculated attention weights are accurately applied to the corresponding positions in the feature maps. The final output is used as the region image corresponding to the formula symbol to be decoded.

[0081] This embodiment provides a specific process for a multi-directional local attention module to process input features through steps 401-405. It not only uses a first fully connected layer to process the current hidden layer state to extract and abstract more complex features, but also uses multiple convolutional kernels to enhance the ability to perceive features of different image sizes, thereby improving feature accuracy. Furthermore, it determines accurate attention weights through superposition and combination transformations, ultimately combining the attention weights with the feature map to output the region image corresponding to the currently decoded formula symbol. This allows for effective focusing on the corresponding local region to better decode the formula symbol of the current step.

[0082] Based on any of the above embodiments, in order to further improve the accuracy of recognizing multi-line formulas, Figure 5 An additional approach to enhancing the ability to perceive complex multi-line formulas is also shown, with process 500 including the following steps:

[0083] Step 501: Obtain an image to be recognized containing multiple lines of formulas, and use the encoding network in the preset formula recognition model to obtain the feature map of the image to be recognized;

[0084] Step 502: Input the feature map into the preset global row attention module in the decoding network to obtain the row count heatmap;

[0085] Based on step 501, this step aims to have the aforementioned executing entity input the feature map output by the encoding network into the preset global line attention module in the decoding network to obtain the output line count heatmap.

[0086] The preset global row attention module is a functional module specifically added to this embodiment on the basis of the conventional attention module in the conventional decoding network. It is used to predict the actual number of rows of multi-line formulas contained in the image to be identified. That is, it adds special attention to the number of formula rows at the global level, and specifically uses the method of representing the number of rows with different heat in the heat map to reduce the difficulty of prediction.

[0087] Typically, to enable the global row attention module to accurately predict the actual number of rows in an image containing formulas, one approach is to use training samples labeled with the row positions or coordinates of each formula in a sample image containing multiple rows of formulas. This is then trained in a fully supervised manner to learn the ability to predict the location of each formula in the image. Another approach is to use training samples labeled only with the number of formula rows in a sample heatmap containing multiple rows of formulas. This is then trained in a weakly supervised manner to learn the ability to predict how many rows of formulas are in the image. It should be understood that compared to the fully supervised training method that requires labeling the row positions or coordinates of each formula row, the weakly supervised method significantly reduces the labeling difficulty and workload. However, the accuracy of predicting the location of each formula row can be improved by using auxiliary heatmaps and heat distribution techniques.

[0088] One type of weakly supervised training scheme, including but not limited to:

[0089] First, select a heatmap containing multiple formulas and only label the actual number of rows as a training sample. Then, use this training sample to perform weakly supervised training on the initial global row attention module based on the loss function of mean absolute error (usually referred to as the L1 loss function) to obtain the pre-trained global row attention module.

[0090] Heatmaps are used because they are a data visualization method used to show the distribution, density, or patterns of data, typically by using color to represent different attributes or values ​​of the data in an image. Heatmaps help people understand data intuitively and reveal patterns and trends within the data. Heatmaps have the following characteristics:

[0091] 1) Color Mapping: Heatmaps use colors to represent information in the data. Typically, a color map is a gradient from low to high values, such as from blue (low values) to red (high values), or from dark to light. Different colors represent different data values ​​or density levels.

[0092] 2) Two-dimensional space: Heatmaps are typically created on a two-dimensional plane, where the X and Y axes represent two dimensions or coordinates of the data. Colors are applied to points, rectangles, or cells on the plane to reflect the data attributes associated with these two dimensions;

[0093] 3) Density and Distribution: Heatmaps are used to visualize the density and distribution of data. High-density areas are typically shown as darker colored areas, while low-density areas are shown as lighter colored areas.

[0094] Based on the aforementioned characteristics of heatmap technology, this step chooses to use heatmap technology to predict the number of formula rows contained in the image to be identified by the feature density and distribution in the feature map. This allows for more accurate identification of each formula symbol in a given number of formula rows, thereby improving the recognition accuracy of complex formulas with multiple rows.

[0095] Furthermore, in order to improve the global row attention module's ability to perceive features of images at different scales, it can be designed as a multi-branch structure, that is, each different branch structure uses convolutional kernels of different sizes to extract features.

[0096] Step 503: Input the row count heatmap and the feature encoding of the decoded formula symbols obtained through the decoding operation into the decoding network to obtain the current hidden layer state;

[0097] Based on step 502, this step aims to have the aforementioned execution entity input the row count heatmap and the feature encoding of the decoded formula symbols obtained through the decoding operation into the decoding network (e.g., the gated loop module therein), thereby obtaining the output current hidden layer state.

[0098] Step 504: Input the feature map and the current hidden layer state into the multi-directional local attention module in the decoding network to obtain the region image corresponding to the formula symbol to be decoded;

[0099] Step 505: Combine and predict the region image, feature encoding, current hidden layer state, and row count heatmap through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation;

[0100] Step 506: Generate a formula symbol sequence based on the formula symbol recognition results output sequentially by the decoding network.

[0101] The solution provided in this embodiment, based on the beneficial effects of the solutions provided in the above embodiments, further adds a global line attention module to the decoding network. This global line attention module can output a line count heatmap to the input feature map to indicate the number of predicted formula lines. In this way, the number of formula lines that should be present in the image to be identified can be clearly determined by the different heat levels in the line count heatmap. Thus, when the number of formula lines is clear, each formula symbol of a multi-line formula can be accurately determined, thereby further improving the recognition accuracy of complex multi-line formulas.

[0102] Based on the above embodiments, to enhance the understanding of how the row count heatmap is obtained through the global row attention module, this embodiment also... Figure 6 A method for obtaining a row count heatmap using a global row attention module is shown, and its process 600 includes the following steps:

[0103] Step 601: Pass the feature map sequentially through the convolutional layer and normalization layer in the preset global row attention module to obtain the transformed feature map;

[0104] By processing feature maps sequentially through convolutional and normalization layers, we can better learn and represent features in the data, while accelerating the training process and improving the model's generalization performance. The convolutional layers are responsible for feature extraction, transforming the raw input data into useful feature representations, while the normalization layers help the network train more stably, avoiding problems such as gradient explosion and gradient vanishing.

[0105] Step 602: Pass the transformed feature map through the channel attention submodule in the preset global row attention module to obtain the channel attention-enhanced feature map;

[0106] Based on step 601, this step aims to obtain a channel attention-enhanced feature map from the feature map transformed by the above-mentioned execution subject through the channel attention submodule in the preset global row attention module.

[0107] The channel attention submodule, derived from SENet (Squeeze-and-Excitation Network), is one of its core innovations, aiming to enhance the representational capabilities of deep convolutional neural networks, particularly in image classification and related computer vision tasks. Its working principle can be divided into two key steps: Squeeze and Excitation.

[0108] Squeeze: In this step, a global average pooling operation is performed on the feature map of each channel, reducing the feature value of each channel to a single scalar. This means that for each channel, SENet calculates the average value of all pixels on its feature map to obtain a numerical value representing the information of that channel. This value will be used for subsequent weight learning;

[0109] Excitation: In this step, weights for each channel are learned through two fully connected layers (typically consisting of a linear layer and an activation function layer, such as ReLU). These weights are used to enhance or weaken the feature response of each channel. The first fully connected layer takes the output of the Squeeze stage as input, produces an intermediate representation, then undergoes a non-linear transformation through the activation function, and finally passes through the second fully connected layer to obtain the weight vector for each channel. These weight vectors are used to adjust the importance of each channel in the feature map.

[0110] Based on the above principles, the channel attention module enhances the network's focus on different channels in the feature map, thereby enabling the network to better capture important feature information. Specifically, it includes the following:

[0111] 1) Adaptive Feature Weights: SENet adaptively adjusts the feature response of each channel using learned weights. This means that important channels will receive higher weights, while less important channels will receive lower weights, thereby reducing the impact of noise and redundant information;

[0112] 2) Enhance useful feature channels: The channel attention module enables the network to better highlight feature channels that are useful for the task. This helps improve the network's discriminative ability, thereby improving task performance;

[0113] 3) Reduce overfitting: The channel attention module helps the network generalize better to new samples because it can reduce overfitting to noisy or unimportant features in the training data.

[0114] In summary, this step, by using SENet's channel attention module and introducing adaptive channel weights, increases the convolutional neural network's attention to feature maps, thereby enhancing the model's representational power and generalization performance. This makes the features adjusted by channel attention more accurate and easier to determine the number of formula rows in subsequent processing steps.

[0115] Step 603: Adjust the single-channel feature map with a channel number of 1 by passing the first convolution in the preset global row attention module after channel attention enhancement.

[0116] The purpose of converting multi-channel feature maps into single-channel feature maps by using a first convolution (e.g., a convolution with a kernel size of 1×1) is to reduce the channel dimension. Furthermore, the mutual weights between channels can be adjusted by learning weights to extract more useful features and reduce redundant information. In particular, the channel attention adjustment mechanism already implemented in step 302 will further improve the performance.

[0117] The specific effects that can be achieved by performing this operation include:

[0118] 1) Reduce computational complexity: Converting the feature map to a single-channel feature map via the first convolution reduces computational cost because, in some cases, multi-channel feature maps may contain a lot of redundant or irrelevant information. Reducing the channel dimension to a single channel reduces the number of model parameters and computational cost.

[0119] 2) Feature Interaction and Integration: While transforming the channel dimensions, the first convolution can also perform cross-channel feature interaction and integration. By learning weights, the network can automatically adjust the weights of different channels to better integrate and utilize the information in each channel;

[0120] 3) Extracting useful features: By converting to a single-channel feature map and using weight adjustment, the model can focus on giving stronger responses to important features while reducing responses to unimportant features. This helps the model better distinguish feature differences between different categories and improves the model's discriminative ability.

[0121] 4) Reduce redundant information: In some cases, multiple channels in a multi-channel feature map may contain highly correlated information. By transforming through the first convolution, this related information can be merged into one channel, reducing redundancy and improving computational efficiency.

[0122] Step 604: Compress the value of each element in the single-channel feature map to a preset range using the sigmoid activation function in the preset global row attention module, and output the feature map at this time as a row count heatmap.

[0123] Building upon step 603, this step aims to have the aforementioned execution entity compress the value of each element in the single-channel feature map to a range of 0 to 1 using a sigmoid activation function (i.e., a sigmoid activation function) under a preset global row attention module. This transforms the output from a feature map into a row count heatmap represented in probabilistic form. Of course, besides the sigmoid activation function, other activation functions with similar effects can be used, which will not be listed here.

[0124] This embodiment provides a process through steps 601-604 of how the global row attention module processes the input feature map to obtain the output row count heatmap. It not only adjusts the attention weights of each channel in the feature map through the channel attention module provided by SENet, but also converts it into a single-channel feature map to perform feature interaction integration, reduce computational costs, and further extract useful features. Finally, it converts it into a probabilistic representation through a sigmoid activation function, thereby obtaining a row count heatmap that can be used to predict the number of rows in the included formula.

[0125] To enhance understanding, this disclosure also provides a specific implementation scheme based on a particular application scenario. Please refer to the example below. Figure 7 This diagram illustrates a process for processing an image to be recognized using a formula recognition model to obtain the formula recognition result. Figure 7 In the decoding network, a conventional attention module is used as an example to highlight the role of adding a global row attention module in the overall scheme:

[0126] This embodiment is based on an encoder-decoder network. The encoder network 710 converts the input image into a feature map F, and the decoder network 720 converts the feature map F into a sequence of formula symbols for output after a series of processing steps. A consistent approach is used when decoding single-line and multi-line formulas. That is, the network is first made aware of the content of each line of the formula by counting the line of formulas under weak supervision, and then the counting information is used to assist in the sequential decoding of the content of each subsequent line of formulas.

[0127] The encoding network 710 can specifically adopt DenseNet (densely connected network). For a three-channel image with input width and height of W and H, DenseNet first uses a convolutional layer with 7 kernels and a stride of 2 to downsample the input image, outputting feature maps with width and height of W / 2 and H / 2. Then, a max pooling layer with 2 kernels and a stride of 2 is used to further downsample the image, outputting feature maps with width and height of W / 4 and H / 4. Finally, a combination of multiple convolutional layers with 1 kernel and 3 kernels is used to further encode the feature maps to obtain the final output feature map F, with width and height of W / 16 and H / 16, and 684 channels.

[0128] The decoding network 720, given a feature map F with widths and heights of W / 16 and H / 16 extracted by the backbone feature extraction network DenseNet, needs to process it and output the final result. For the input feature map F, a 1×1 convolution is first used to transform it into intermediate layer features M, with widths and heights of W / 16 and H / 16 and 512 channels. The intermediate layer features M are then input into the attention module 721 and the global row attention module 722 to obtain a row count heatmap Q, with widths and heights of W / 16 and H / 16 and 1 channel. Combining the row count heatmap Q with the feature encoding E of the symbols decoded in the previous step, it is input into the GRU network 723 to obtain the current hidden layer state h_t. Finally, the feature map F, the intermediate layer features M, and the current hidden layer state h_t are input into the attention module 721 to obtain the contextual semantic information C. The contextual semantic information C, the feature encoding of the symbol decoded in the previous step E, the current hidden layer state h_t, and the row count heatmap Q are respectively mapped by a fully connected layer 724 and then combined to obtain feature T. After passing through another fully connected layer 724 for prediction, the final formula symbol sequence prediction result can be obtained.

[0129] The following section will focus on the global row attention module mentioned above:

[0130] Current research on counting is mostly fully supervised, requiring the acquisition of the location information of each instance, i.e., counting is performed through detection methods. Applying counting to multi-line formulas requires weakly supervised counting, independent of the positional annotations of each line of the formula, relying solely on the line number annotations. This embodiment designs a global line attention module with a dual-branch structure. The two branches use convolutional kernels of different sizes (one 3×3, the other 5×5) to create differences in the receptive field, thereby constructing multi-scale information modeling capabilities. The input feature map M passes through convolutional layers and normalization layers, and the resulting transformed feature map is input into the channel attention submodule to obtain enhanced features. The channel attention submodule used here is the one proposed in SENet. After obtaining the enhanced features, a 1x1 convolution is used to adjust the number of feature channels to 1. After a 1×1 convolution, the Sigmoid activation function is used to compress the value of each element in the feature map into the range of (0,1), which yields the row count heatmap Q. The row count prediction value is obtained by summing all the elements of the row count heatmap Q. The row count prediction value is then supervised by applying the L1 loss function.

[0131] Based on any of the above embodiments, this embodiment also... Figure 8 A method for obtaining complete text recognition results based on formula symbol sequences is provided, and its process 800 includes the following steps:

[0132] Step 801: Determine the number of predicted formula rows for multi-line formulas contained in the image to be identified based on the row count heatmap;

[0133] This step can be located at Figure 2 After step 203 of the illustrated embodiment and located in Figure 5 After step 506, i.e. after obtaining the formula symbol sequence in steps 203 and 506, the execution entity determines the predicted number of formula rows in the image to be identified based on the heat information in the row count heatmap. In other words, this step is the execution entity first determining how many rows of formulas are in the image.

[0134] Step 802: Reconstruct the formula by dividing the formula symbol sequence into the predicted formula lines to obtain a multi-line formula in text form;

[0135] Based on step 801, this step aims to have the aforementioned executing entity restore the formula symbols contained in the formula symbol sequence according to the determined number of formula lines (that is, to concatenate and restore the formula symbols in the same line in the same line according to the order in the sequence), to obtain a multi-line formula in text form.

[0136] Step 803: Combine the multi-line formula in text form with the other text recognition results in the non-multi-line formula part of the image to be recognized, according to the original layout, to obtain the complete text recognition result corresponding to the image to be recognized.

[0137] Based on step 802, this step aims to have the aforementioned executing entity combine the multi-line formula in text form with other text recognition results in the non-multi-line formula portion of the image to be recognized, according to the original layout (i.e., the layout between the formula portion and the non-formula portion presented in the image to be recognized), to obtain the complete text recognition result corresponding to the image to be recognized.

[0138] It should be noted that steps 801-802 can be combined with other embodiments to form a new embodiment without using step 803 in subsequent processing steps. This embodiment exists only as a preferred embodiment that exists simultaneously.

[0139] Further reference Figure 9 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a formula recognition device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0140] like Figure 9As shown, the formula recognition device 600 of this embodiment may include: a feature map extraction unit 901, a feature decoding unit 902, and a formula symbol sequence generation unit 903. The feature map extraction unit 901 is configured to acquire an image to be recognized containing multiple lines of formulas, and obtain a feature map of the image to be recognized using an encoding network in a preset formula recognition model. The feature decoding unit 902 is configured to use a decoding network in the preset formula recognition model to perform decoding operations on the features in the feature map according to the following sub-units, where the features in the feature map correspond to different formula symbols: the current hidden layer state acquisition sub-unit 9021 is configured to input the feature map and the feature encoding of the decoded formula symbols obtained through the decoding operation into the decoding network to obtain the current hidden layer state; wherein, the first formula symbol sequence generation unit 903 is pre-programmed with the first formula symbol sequence generation unit 904. The current hidden layer state corresponding to the formula symbol is set to an initial value; the region image determination subunit 9022 is configured to input the feature map and the current hidden layer state into the multi-directional local attention module in the decoding network to obtain the region image corresponding to the formula symbol to be decoded; the feature combination and prediction subunit 9023 is configured to combine and predict the region image, feature encoding, and current hidden layer state through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation; the formula symbol sequence generation unit 903 is configured to generate a formula symbol sequence based on the formula symbol recognition results output sequentially by the decoding network.

[0141] In this embodiment, the specific processing of the feature map extraction unit 901, the feature decoding unit 902, and the formula symbol sequence generation unit 903 in the formula recognition device 900, and the resulting technical effects, can be found in the following references: Figure 2 The relevant descriptions of steps 201-203 in the corresponding embodiments will not be repeated here.

[0142] In some optional implementations of this embodiment, the region image determination subunit 9022 can be further configured as follows:

[0143] The current hidden layer state is input into the first fully connected layer in the multi-directional local attention module to obtain the processed features.

[0144] The feature map is input into multiple branch structures in the multi-directional local attention module, and the attention weight features of each decoded formula symbol are extracted by using convolution kernels of different sizes in different branch structures to obtain the target attention weight features corresponding to the formula symbol to be decoded.

[0145] The processed features, target attention weight features, and feature map are sequentially input into the stacking submodule of the multi-directional local attention module to obtain the stacked features.

[0146] The superimposed features are input into the second fully connected layer in the multi-directional local attention module to obtain the attention weights.

[0147] The attention weights and feature maps are input into the multiplication submodule of the multidirectional local attention module to obtain the region image corresponding to the formula symbol to be decoded.

[0148] In some optional implementations of this embodiment, the convolution kernels of different sizes include at least a first-size convolution kernel larger than the feature map size and a second-size convolution kernel smaller than the feature map size.

[0149] In some optional implementations of this embodiment, the formula recognition device 900 may further include:

[0150] The channel number adjustment unit is configured to adjust the channel number of the feature map to the required channel number through a first convolution in response to a mismatch between the channel number of the feature map and the channel number of the decoding network, so that the decoding network can use the channel number-adjusted feature map for decoding.

[0151] In some optional implementations of this embodiment, the feature combination and prediction subunit 9023 can be further configured as follows:

[0152] The region image, feature encoding, and current hidden layer state are processed through the first fully connected layer in the decoding network to perform feature mapping and combine the mapped features to obtain combined features.

[0153] The combined features are used for feature prediction through the second fully connected layer in the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation.

[0154] In some optional implementations of this embodiment, the formula recognition device 900 may further include:

[0155] The preprocessing unit is configured to preprocess the image to be recognized; wherein the preprocessing includes at least one of: orientation correction processing, binarization processing, grayscale processing, contrast processing, and sharpness processing.

[0156] In some optional implementations of this embodiment, the formula recognition device 900 may further include:

[0157] The global row attention module processing unit is configured to input the feature map into the preset global row attention module in the decoding network to obtain the row count heatmap;

[0158] Correspondingly, the current hidden layer state acquisition subunit 9021 can be further configured as follows:

[0159] The feature encoding of the row count heatmap and the decoded formula symbols obtained through the decoding operation are input into the decoding network to obtain the current hidden layer state;

[0160] Correspondingly, the feature combination and prediction subunit 9023 can be further configured as follows:

[0161] The region image, feature encoding, current hidden layer state, and row count heatmap are combined and predicted through a decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation.

[0162] In some optional implementations of this embodiment, the global row attention module processing unit can be further configured as follows:

[0163] The feature map is passed sequentially through the convolutional layer and normalization layer in the preset global row attention module to obtain the transformed feature map;

[0164] The transformed feature map is passed through the channel attention module in the preset global row attention module to obtain the channel attention-enhanced feature map;

[0165] After channel attention enhancement, the feature map is adjusted to a single-channel feature map with 1 channel number through the first convolution in the preset global row attention module;

[0166] The single-channel feature map is compressed to a preset range by the S-shaped activation function in the preset global row attention module, and the feature map at this time is output as a row count heatmap.

[0167] In some optional implementations of this embodiment, the formula recognition device 900 may further include: a global row attention module training unit, which may be configured to;

[0168] Obtain training samples with only the actual number of rows labeled for the sample heatmap containing formulas with multiple rows;

[0169] The initial global row attention module is weakly trained using the training samples based on the loss function of mean absolute error, resulting in a pre-trained global row attention module.

[0170] In some optional implementations of this embodiment, the formula recognition device 900 may further include:

[0171] The prediction formula row number determination unit is configured to determine the prediction formula row number of multi-row formulas contained in the image to be identified based on the row count heatmap;

[0172] The formula reduction unit is configured to reduce the formula symbol sequence according to the predicted formula line number to obtain a multi-line formula in text form.

[0173] In some optional implementations of this embodiment, the formula recognition device 900 may further include:

[0174] The complete text recognition result generation unit is configured to combine multi-line formulas in text form with other text results that are not multi-line formulas in the image to be recognized, according to the original layout, to obtain the complete text recognition result corresponding to the image to be recognized.

[0175] This embodiment exists as a device embodiment corresponding to the above method embodiment. The formula recognition device provided in this embodiment, based on the conventional overall network architecture of an encoder-decoder type network, for long sequence data such as multi-line complex formulas, not only processes the feature map output by the encoder network and the feature encoding of the decoded formula symbols obtained by the decoding operation through the decoding network, so that the output current hidden layer state can more accurately describe the relevant feature information of the formula symbols in the current step, but also replaces the attention module of the decoding network, which usually has only a single attention weight channel, with a multi-directional local attention module with multiple attention weight channels in multiple different directions. This allows the device to capture the differences of different formula symbols in different directions when recognizing multiple formula symbols contained in a local area, thereby improving the recognition accuracy of multi-line complex formulas.

[0176] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the formula recognition method described in any of the above embodiments when executed.

[0177] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the formula recognition method described in any of the above embodiments when executed.

[0178] According to embodiments of this disclosure, this disclosure also provides a computer program product that, when executed by a processor, can implement the steps of the formula recognition method described in any of the above embodiments.

[0179] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0180] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0181] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0182] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the formula recognition method. For example, in some embodiments, the formula recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the formula recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the formula recognition method by any other suitable means (e.g., by means of firmware).

[0183] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0184] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0185] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0186] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0187] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0188] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0189] According to the technical solution of this disclosure, based on the conventional overall network architecture consisting of an encoder-decoder type network, for long sequence data such as multi-line complex formulas, not only is the feature map output by the encoder network and the feature encoding of the decoded formula symbols obtained by the decoding operation processed by the decoding network, so that the output current hidden layer state can more accurately describe the relevant feature information of the formula symbols in the current step, but also by replacing the attention module of the decoding network, which usually has only a single attention weight channel, with a multi-directional local attention module with multiple attention weight channels in multiple different directions, the differences between different formula symbols in different directions can be captured by the attention weight channels in multiple directions when recognizing multiple formula symbols contained in a local region, thereby improving the recognition accuracy of multi-line complex formulas.

[0190] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0191] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A formula recognition method, comprising: Obtain an image to be recognized containing multiple lines of formulas, and use the encoding network in the preset formula recognition model to obtain the feature map of the image to be recognized; The features in the feature map are decoded sequentially using the decoding network in the preset formula recognition model in the following manner, where each feature in the feature map corresponds to a different formula symbol: The feature map and the feature encoding of the decoded formula symbol obtained through the decoding operation are input into the decoding network to obtain the current hidden layer state; wherein, the current hidden layer state corresponding to the first formula symbol is set to the initial value in advance; The feature map and the current hidden layer state are input into the multi-directional local attention module in the decoding network to obtain the region image corresponding to the formula symbol to be decoded. The region image, the feature encoding, and the current hidden layer state are combined and predicted through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation; A formula symbol sequence is generated based on the formula symbol recognition results output sequentially by the decoding network; The step of inputting the feature map and the current hidden layer state into the multi-directional local attention module in the decoding network to obtain the region image corresponding to the formula symbol to be decoded includes: The current hidden layer state is input into the first fully connected layer in the multi-directional local attention module to obtain the processed features; The feature map is input into multiple branch structures in the multi-directional local attention module, and the sum of the attention weight features of each decoded formula symbol is extracted using convolution kernels of different sizes in different branch structures to obtain the target attention weight features corresponding to the formula symbol to be decoded. The processed features, the target attention weight features, and the feature map are sequentially input into the superposition submodule of the multi-directional local attention module to obtain the superimposed features. The superimposed features are input into the second fully connected layer of the multi-directional local attention module to obtain attention weights. The attention weights and the feature map are input into the multiplication submodule of the multidirectional local attention module to obtain the region image corresponding to the formula symbol to be decoded.

2. The method according to claim 1, wherein, The different sizes of convolutional kernels include at least a first-size convolutional kernel larger than the size of the feature map and a second-size convolutional kernel smaller than the size of the feature map.

3. The method according to claim 1, further comprising: In response to a mismatch between the number of channels in the feature map and the number of channels in the decoding network, the feature map is adjusted to the required number of channels through a first convolution, so that the decoding network uses the feature map with the adjusted number of channels for decoding.

4. The method according to claim 1, wherein, The step of combining and predicting the region image, the feature encoding, and the current hidden layer state through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation includes: The region image, the feature encoding, and the current hidden layer state are processed through the first fully connected layer in the decoding network to perform feature mapping and combine the mapped features to obtain combined features. The combined features are used for feature prediction through the second fully connected layer in the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation.

5. The method according to claim 1, further comprising: The image to be identified is preprocessed; wherein the preprocessing includes at least one of the following: orientation correction processing, binarization processing, grayscale processing, contrast processing, and sharpness processing.

6. The method according to claim 1, further comprising: The feature map is input into a preset global row attention module in the decoding network to obtain a row count heatmap; Correspondingly, the step of inputting the feature map and the feature encoding of the decoded formula symbols obtained through the decoding operation into the gated recurrent module in the decoding network to obtain the current hidden layer state includes: The row count heatmap and the feature encoding of the decoded formula symbols obtained through the decoding operation are input into the decoding network to obtain the current hidden layer state; Correspondingly, the step of combining and predicting the region image, the feature encoding, and the current hidden layer state through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation includes: The region image, the feature encoding, the current hidden layer state, and the row count heatmap are combined and predicted through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation.

7. The method according to claim 6, wherein, The step of inputting the feature map into the preset global row attention module in the decoding network of the preset formula recognition model to obtain the row count heatmap includes: The feature map is passed sequentially through the convolutional layer and normalization layer in the preset global row attention module to obtain the transformed feature map; The transformed feature map is passed through the channel attention module in the preset global row attention module to obtain a channel attention-enhanced feature map; The channel attention-enhanced feature map is adjusted to a single-channel feature map with a channel count of 1 through the first convolution in the preset global row attention module; The single-channel feature map is compressed to a preset range by the S-shaped activation function in the preset global row attention module, and the feature map at this time is output as the row count heatmap.

8. The method according to claim 6, wherein, The training process of the preset global row attention module includes: Obtain training samples with only the actual number of rows labeled for the sample heatmap containing formulas with multiple rows; The training samples are used to perform weakly supervised training on the initial global row attention module based on the loss function of mean absolute error, so as to obtain the pre-trained global row attention module.

9. The method according to claim 1, wherein, The step of inputting the feature map and the feature encoding of the decoded formula symbols obtained through the decoding operation into the decoding network to obtain the current hidden layer state includes: The feature map and the feature encoding of the decoded formula symbols obtained through the decoding operation are input into the gated recurrent module in the decoding network to obtain the current hidden layer state.

10. The method of claim 6, further comprising: The number of predicted formula rows in the multi-line formula contained in the image to be identified is determined based on the row count heatmap. The formula symbol sequence is restored according to the number of lines in the predicted formula to obtain a multi-line formula in text form.

11. The method of claim 10, further comprising: The multi-line formula in text form is combined with other text results in the image to be recognized that are not multi-line formulas, according to the original layout, to obtain the complete text recognition result corresponding to the image to be recognized.

12. A formula recognition device, comprising: The feature map extraction unit is configured to acquire an image to be identified containing multiple lines of formulas, and to obtain the feature map of the image to be identified by using the encoding network in the preset formula recognition model. The feature decoding unit is configured to use the decoding network in the preset formula recognition model to perform decoding operations on the features in the feature map sequentially according to the following sub-units, wherein the features in the feature map correspond to different formula symbols: The current hidden layer state acquisition subunit is configured to input the feature map and the feature encoding of the decoded formula symbol obtained by the decoding operation into the decoding network to obtain the current hidden layer state; wherein, the current hidden layer state corresponding to the first formula symbol is preset to an initial value; The region image determination subunit is configured to input the feature map and the current hidden layer state into the multi-directional local attention module in the decoding network to obtain a region image corresponding to the formula symbol to be decoded. The feature combination and prediction subunit is configured to combine and predict the region image, the feature encoding, and the current hidden layer state through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation. The formula symbol sequence generation unit is configured to generate a formula symbol sequence based on the formula symbol recognition results sequentially output by the decoding network; The region image determination subunit is further configured to: The current hidden layer state is input into the first fully connected layer in the multi-directional local attention module to obtain the processed features; The feature map is input into multiple branch structures in the multi-directional local attention module, and the sum of the attention weight features of each decoded formula symbol is extracted using convolution kernels of different sizes in different branch structures to obtain the target attention weight features corresponding to the formula symbol to be decoded. The processed features, the target attention weight features, and the feature map are sequentially input into the superposition submodule of the multi-directional local attention module to obtain the superimposed features. The superimposed features are input into the second fully connected layer of the multi-directional local attention module to obtain attention weights. The attention weights and the feature map are input into the multiplication submodule of the multidirectional local attention module to obtain the region image corresponding to the formula symbol to be decoded.

13. The apparatus according to claim 12, wherein, The different sizes of convolutional kernels include at least a first-size convolutional kernel larger than the size of the feature map and a second-size convolutional kernel smaller than the size of the feature map.

14. The apparatus of claim 12, further comprising: The channel number adjustment unit is configured to adjust the channel number of the feature map to the required channel number through a first convolution in response to a mismatch between the channel number of the feature map and the channel number of the decoding network, so that the decoding network uses the channel number-adjusted feature map for decoding.

15. The apparatus according to claim 12, wherein, The feature combination and prediction subunit are further configured as follows: The region image, the feature encoding, and the current hidden layer state are processed through the first fully connected layer in the decoding network to perform feature mapping and combine the mapped features to obtain combined features. The combined features are used for feature prediction through the second fully connected layer in the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation.

16. The apparatus of claim 12, further comprising: The preprocessing unit is configured to preprocess the image to be identified; wherein the preprocessing includes at least one of: orientation correction processing, binarization processing, grayscale processing, contrast processing, and sharpness processing.

17. The apparatus of claim 12, further comprising: The global row attention module processing unit is configured to input the feature map into a preset global row attention module in the decoding network to obtain a row count heatmap; Correspondingly, the current hidden layer state acquisition subunit is further configured to: The row count heatmap and the feature encoding of the decoded formula symbols obtained through the decoding operation are input into the decoding network to obtain the current hidden layer state; Correspondingly, the feature combination and prediction subunit is further configured as follows: The region image, the feature encoding, the current hidden layer state, and the row count heatmap are combined and predicted through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation.

18. The apparatus according to claim 17, wherein, The global row attention module processing unit is further configured to: The feature map is passed sequentially through the convolutional layer and normalization layer in the preset global row attention module to obtain the transformed feature map; The transformed feature map is passed through the channel attention module in the preset global row attention module to obtain a channel attention-enhanced feature map; The channel attention-enhanced feature map is adjusted to a single-channel feature map with a channel count of 1 through the first convolution in the preset global row attention module; The single-channel feature map is compressed to a preset range by the S-shaped activation function in the preset global row attention module, and the feature map at this time is output as the row count heatmap.

19. The apparatus of claim 18, further comprising: A global row attention module training unit, wherein the global row attention module training unit is configured to; Obtain training samples with only the actual number of rows labeled for the sample heatmap containing formulas with multiple rows; The training samples are used to perform weakly supervised training on the initial global row attention module based on the loss function of mean absolute error, so as to obtain the pre-trained global row attention module.

20. The apparatus according to claim 12, wherein, The current hidden layer state acquisition subunit is further configured to: The feature map and the feature encoding of the decoded formula symbols obtained through the decoding operation are input into the gated recurrent module in the decoding network to obtain the current hidden layer state.

21. The apparatus of claim 17, further comprising: The prediction formula row number determination unit is configured to determine the prediction formula row number of multi-row formulas contained in the image to be identified based on the row count heatmap; The formula restoration unit is configured to restore the formula symbol sequence according to the predicted formula line number to obtain a multi-line formula in text form.

22. The apparatus of claim 21, further comprising: The complete text recognition result generation unit is configured to combine the multi-line formula in the text form with other text results in the image to be recognized that are not the multi-line formula, according to the original layout, to obtain the complete text recognition result corresponding to the image to be recognized.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the formula recognition method according to any one of claims 1-11.

24. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the formula recognition method according to any one of claims 1-11.

25. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the formula recognition method according to any one of claims 1-11.

Images