Formula recognition method, device, equipment, storage medium and program product

Abstract

The present disclosure provides a formula recognition method, apparatus, device, storage medium and program product, which relates to the fields of artificial intelligence technologies such as computer vision, deep learning, and image processing, and can be applied to scenarios such as smart finance and smart government affairs. The method comprises: using an encoding network to obtain a feature map of an image to be recognized containing a formula, and using a global row attention module in a decoding network to obtain a row count heat map corresponding to the feature map; using a decoding network to sequentially decode the features in the feature map: processing the row count heat map and the feature coding of the decoded formula symbol, and further processing the current hidden layer state and the feature map attention module obtained after processing to obtain a regional image of the formula symbol to be decoded; processing the regional image, feature coding, current hidden layer state and row count heat map to obtain an output formula symbol recognition result; generating a formula symbol sequence from the formula symbol recognition results sequentially output by the decoding network.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as computer vision, deep learning, and image processing, and in particular to a formula recognition method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Document OCR (Optical Character Recognition) functions are widely needed in the fields of education, office, library and archives.

[0003] For a document OCR engine, recognizing formulas within documents is crucial. Unlike regular text, formulas often have complex spatially nested structures, primarily due to their unique structure, including fractions, superscripts, subscripts, and square roots. Multi-line formulas are particularly challenging to recognize. Summary of the Invention

[0004] The embodiments of the present disclosure provide a formula recognition method, apparatus, electronic device, computer-readable storage medium, and computer program product.

[0005] In the first aspect, an embodiment of the present disclosure proposes a formula recognition method, including: obtaining an image to be recognized containing multiple lines of formulas, and using an encoding network in a preset formula recognition model to obtain a feature map of the image to be recognized; inputting the feature map into a preset global row attention module in a decoding network in the preset formula recognition model to obtain a row count heat map; using the decoding network to perform decoding operations on the features in the feature map in sequence in the following manner, where the features in the feature map correspond to different formula symbols: inputting the row count heat map and the feature codes of the decoded formula symbols obtained by the decoding operation into the decoding network to obtain a current hidden layer state; wherein the current hidden layer state corresponding to the first formula symbol is pre-set as an initial value; inputting the feature map and the current hidden layer state into the decoding network to obtain a regional image corresponding to the current formula symbol to be decoded; combining and predicting the regional image, feature code, current hidden layer state and row count heat map through the decoding network to obtain a formula symbol recognition result corresponding to the current decoding operation; generating a formula symbol sequence according to the formula symbol recognition results output sequentially by the decoding network.

[0006] In the second aspect, the embodiment of the present disclosure proposes a formula recognition device, including: a feature map extraction unit, configured to obtain an image to be recognized containing multiple lines of formulas, and use the encoding network in the preset formula recognition model to obtain the feature map of the image to be recognized; a global row attention module processing unit, configured to input the feature map into the preset global row attention module in the decoding network in the preset formula recognition model to obtain a row count heat map; a feature decoding unit, configured to use the decoding network to perform decoding operations on the features in the feature map in sequence according to the following sub-units, and the features in the feature map correspond to different formula symbols respectively; a current hidden layer state acquisition sub-unit, configured to combine the row count heat map and the decoded state obtained by the decoding operation The feature code of the decoded formula symbol is input into the decoding network to obtain the current hidden layer state; wherein, the current hidden layer state corresponding to the first formula symbol is set as the initial value in advance; the regional image determination unit is configured to input the feature map and the current hidden layer state into the decoding network to obtain the regional image corresponding to the formula symbol to be decoded; the feature combination and prediction unit is configured to combine and predict the regional image, feature code, current hidden layer state and row count heat map through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation; the formula symbol sequence generation unit is configured to generate a formula symbol sequence according to the formula symbol recognition results sequentially output by the decoding network.

[0007] In a third aspect, an embodiment of the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the formula recognition method described in the first aspect when executing the instructions.

[0008] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the formula recognition method described in the first aspect when executed.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the steps of the formula recognition method described in the first aspect.

[0010] The formula recognition solution provided by the present disclosure is based on the conventional overall network architecture composed of an encoding-decoding type network. On the basis of the conventional attention module of the decoding network, an additional global row attention module is added. The global row attention module can output a row count heat map for the input feature map to indicate the predicted number of formula rows, thereby making it possible to clearly predict the number of formula rows that should be in the image to be recognized through the different heat levels in the row count heat map, and thus accurately determine each formula symbol of the multi-line formula when the number of formula rows is clear, thereby improving the recognition accuracy of multi-line complex formulas.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Other features, objects and advantages of the present disclosure will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following drawings:

[0013] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0014] Figure 2 A flow chart of a formula recognition method provided in an embodiment of the present disclosure;

[0015] Figure 3 A flowchart of a method for obtaining a row count heat map using a global row attention module provided in an embodiment of the present disclosure;

[0016] Figure 4 A flowchart of a method for obtaining a complete text recognition result based on a formula symbol sequence provided by an embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of a process for obtaining a formula recognition result by processing an image to be recognized through a formula recognition model provided in an embodiment of the present disclosure;

[0018] Figure 6 A structural block diagram of a formula recognition device provided in an embodiment of the present disclosure;

[0019] Figure 7 A schematic structural diagram of an electronic device suitable for executing a formula recognition method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other unless there is a conflict.

[0021] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0022] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the formula recognition method, apparatus, electronic device, and computer-readable storage medium of the present disclosure may be applied.

[0023] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0024] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed, such as document OCR recognition applications, model training applications, and instant messaging applications.

[0025] Terminal devices 101, 102, 103 and server 105 can be either hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and are not specifically limited here.

[0026] The server 105 can provide various services through various built-in applications. Taking the document OCR recognition application that can provide document OCR recognition service as an example, the server 105 can achieve the following effects when running the document OCR recognition application: first, the image to be recognized containing multiple lines of formulas transmitted by the terminal devices 101, 102, and 103 is received through the network 104, and then the encoding network in the preset formula recognition model is used to obtain the feature map of the image to be recognized; then, the feature map is input into the preset global row attention module in the decoding network in the preset formula recognition model to obtain a row count heat map; next, the decoding network is used to decode the features in the feature map in the following manner in turn. Code operation, the features in the feature map correspond to different formula symbols respectively: the row count heat map and the feature codes of the decoded formula symbols obtained by the decoding operation are input into the decoding network to obtain the current hidden layer state; wherein the current hidden layer state corresponding to the first formula symbol is set as the initial value in advance; the feature map and the current hidden layer state are input into the decoding network to obtain the regional image corresponding to the current formula symbol to be decoded; the regional image, the feature code, the current hidden layer state and the row count heat map are combined and predicted through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation; finally, a formula symbol sequence is generated according to the formula symbol recognition results outputted sequentially by the decoding network.

[0027] It should be noted that, in addition to being temporarily obtained from terminal devices 101, 102, and 103 via network 104, the image to be recognized, which contains multiple lines of formulas, can also be pre-stored locally on server 105 in various ways. Therefore, when server 105 detects that such data is already stored locally (for example, when starting to process a previously saved OCR recognition task for a document to be processed), it can choose to directly obtain such data locally. In this case, exemplary system architecture 100 may also not include terminal devices 101, 102, 103 and network 104.

[0028] Because accurately recognizing images containing multiple lines of formulas requires significant computing resources and significant computational power, the formula recognition methods provided in the subsequent embodiments of this disclosure are generally performed by a server 105 possessing significant computing power and resources. Accordingly, the formula recognition apparatus is generally also located within server 105. However, it should also be noted that, if terminal devices 101, 102, and 103 also possess sufficient computing power and resources, terminal devices 101, 102, and 103 may also utilize document OCR recognition applications installed thereon to perform the aforementioned operations delegated to server 105, thereby outputting the same results as server 105. In particular, in the presence of multiple terminal devices with varying computing power, if the document OCR recognition application determines that the terminal device it is located on possesses significant computing power and resources, it may allow the terminal device to perform the aforementioned operations, thereby appropriately alleviating the computational burden on server 105. Accordingly, the formula recognition apparatus may also be located within terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104 .

[0029] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0030] Please refer to Figure 2 , Figure 2 This is a flow chart of a formula recognition method provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0031] Step 201: obtaining an image to be recognized containing multiple lines of formulas, and obtaining a feature map of the image to be recognized using an encoding network in a preset formula recognition model;

[0032] This step aims to identify the execution subject of the method by the formula (e.g. Figure 1 The server 105 shown) obtains (e.g., receives Figure 1 The terminal devices 101, 102, and 103 shown in the figure transmit an image to be identified containing multiple lines of formulas, and input the image to be identified as input data into a preset formula recognition model. The formula recognition model is a network structure of an encoding-decoding network, so the image to be identified will first be feature-encoded by the encoding network to obtain a feature map corresponding to the image to be identified.

[0033] Among them, the image to be identified can be obtained by a user holding a terminal device with a shooting function (such as a mobile phone, tablet computer, smart camera) to take a picture of a book or document material containing multiple lines of complex formulas, or it can be downloaded directly from the Internet or a cloud database, or it can be captured from other image files, and no specific limitation is made here.

[0034] The formula recognition model used in the embodiments of the present disclosure is an encoder-decoder network, also commonly known as an encoder-decoder network. It is a type of deep learning architecture whose core idea is to encode input data into a potential representation (usually low-dimensional) and then decode it back to the original data or target data. This type of network has the following characteristics:

[0035] 1) Encoder: The first part of the encoder-decoder network is the encoder, which is responsible for converting the input data into a latent representation. The encoder typically includes multiple convolutional layers, pooling layers, and fully connected layers to extract features from the raw data and map them to a low-dimensional representation.

[0036] 2) Latent representation: The latent representation is the output of the encoder and is usually of lower dimension. This representation captures the key features of the input data and maps the data from a high-dimensional space to a more compact representation space;

[0037] 3) Decoder: The decoder is the second part of the network, responsible for decoding the latent representation into the original data or target data. The decoder typically includes multiple deconvolution layers (or transposed convolution layers), upsampling layers, and fully connected layers to generate the target data from the latent representation.

[0038] 4) Autoencoder: A common form of encoder-decoder network is the autoencoder, which aims to learn a compact data representation while maximally reconstructing the input data. An autoencoder typically consists of an encoder and a decoder, and the training process aims to minimize the reconstruction error.

[0039] This type of network is commonly used for image denoising (where the encoder encodes a noisy image into a latent representation, and the decoder decodes the latent representation into a denoised image), image generation (generative adversarial networks are an extension of the encoder-decoder structure, and are often used in image generation tasks, such as generating realistic images, image super-resolution, etc.), speech processing (in the field of speech processing, encoder-decoder networks can be used for tasks such as speech synthesis, speech recognition, and speech denoising), and sequence-to-sequence tasks (encoder-decoder architectures are also widely used in sequence-to-sequence tasks, such as machine translation, text summarization, and speech recognition, where the encoder encodes the input sequence into a latent representation, The decoder decodes the potential representation into a target sequence), dimensionality reduction and feature learning (encoding-decoding networks such as autoencoders can be used for data dimensionality reduction and feature learning, which helps to extract useful information from the data), and anomaly detection (encoding-decoding networks can be used for anomaly detection, in which normal data is successfully reconstructed, while abnormal data cannot be well reconstructed because it is different from normal data). The embodiments of the present disclosure make use of the above-mentioned characteristics of this type of network to accurately recognize multiple lines of complex formulas contained in an image, that is, the input of the formula recognition model is an image to be recognized containing multiple lines of formulas, and the output is a sequence of symbols that constitute the multiple lines of formulas.

[0040] In addition, considering that the encoding network and the decoding network are sometimes played by different models, there may be a problem that the actual output of the encoding network does not match the required input of the decoding network. For example, the mismatch may occur in the feature map size and the number of feature map channels. In order to meet the required input of the decoding network, the feature map output by the encoding network also needs to be resized or the number of channels is adjusted. For example, when the number of channels of the feature map output by the encoding network is more than the number of channels of the feature map required by the decoding network, the number of channels can be reduced to the required number of channels by using a first convolution including a 1×1 convolution.

[0041] In addition, before the image to be identified is input into the encoding network, the image to be identified can also be preprocessed, which can specifically include: direction correction, binarization, grayscale processing, contrast enhancement, and clarity improvement. At least one of the following can be used to highlight the part of the image where the multiple lines of formulas are located through preprocessing, which helps to improve the accuracy of the recognition results.

[0042] Step 202: Input the feature map into a preset global row attention module in the decoding network of the preset formula recognition model to obtain a row count heat map;

[0043] Based on step 201, this step aims to have the above-mentioned execution entity input the feature map output by the encoding network into the preset global row attention module in the decoding network in the formula recognition model to obtain a row count heat map.

[0044] Among them, the preset global row attention module is a functional module specially added in this embodiment on the basis of the conventional attention module of the conventional decoding network, which is used to predict the actual number of rows of the multi-row formula part contained in the image to be identified, that is, to increase the attention to the number of formula rows from the global level, and specifically select the method of representing the number of rows with different heat in the heat map to reduce the difficulty of prediction.

[0045] Generally, in order to enable the global row attention module to have the ability to accurately predict the actual number of rows of the formula part in the image, one approach is: use training samples in a sample image containing multiple rows of formulas that are marked with the row position or row coordinates of each row of formulas, and train the global row attention module in a fully supervised manner to learn the ability to predict the location of each row of formulas in the image; another approach is: use training samples in a sample thermal image containing multiple rows of formulas that are only marked with the number of formula rows, and train the global row attention module in a weakly supervised manner to learn the ability to predict how many rows of formulas are contained in the image. It should be understood that compared to the fully supervised training method that requires marking the row position or row coordinates of each row of formulas, the weakly supervised method of only marking the number of formula rows can obviously reduce the difficulty and workload of marking, but the accuracy of the prediction of the position of each row of formulas can also be improved through auxiliary heat maps and heat distribution technology.

[0046] A weakly supervised training scheme including but not limited to:

[0047] First, a training sample containing a heat map of multiple lines of formulas is taken, in which only the actual number of lines is annotated. Then, the initial global row attention module is weakly supervised trained based on the mean absolute error loss function (usually referred to as the L1 loss function) to obtain a preset global row attention module that has been trained.

[0048] Heatmaps are used because heatmap technology is a data visualization method used to show the distribution, density, or pattern of data, usually by using color on an image to represent different attributes or values of the data. Heatmaps help people understand data in an intuitive way and reveal patterns and trends in the data. Heatmaps have the following characteristics:

[0049] 1) Color Mapping: Heatmaps use color to represent data information. Typically, a color map is a gradient from low to high values, such as from blue (low value) to red (high value), or from dark to light. Different colors represent different data values or density levels.

[0050] 2) Two-dimensional space: Heatmaps are typically created on a two-dimensional plane, where the X-axis and Y-axis represent the two dimensions or coordinates of the data. Colors are applied to points, rectangles, or cells on the plane to reflect the data attributes associated with these two dimensions.

[0051] 3) Density and distribution: Heatmaps are used to visualize the density and distribution of data. High-density areas are usually displayed as darker areas, while low-density areas are displayed as lighter areas.

[0052] It is precisely based on the above-mentioned characteristics of heat map technology that this step chooses to use heat map technology to predict the number of formula rows contained in the image to be identified through the feature density and distribution in the feature map, and then more accurately identify each row of formula symbols when the number of formula rows is known, thereby improving the recognition accuracy of multiple rows of complex formulas.

[0053] Furthermore, in order to improve the global row attention module's ability to perceive features of images of different scales, it can also be designed as a multi-branch structure, that is, each different branch structure uses convolution kernels of different sizes to extract features.

[0054] Step 203: Using the decoding network, the features in the feature map are decoded in sequence according to the following method. The features in the feature map correspond to different formula symbols:

[0055] That is, this step is to use the decoding network to perform decoding operations on the features corresponding to different formula symbols in sequence (it can also be called step-by-step decoding operations, each step corresponds to a different formula symbol), so as to obtain the formula symbol recognition result of the formula symbol corresponding to the current decoding through each decoding operation.

[0056] Step 2031: Input the row count heat map and the feature code of the decoded formula symbol obtained by the decoding operation into the decoding network to obtain the current hidden layer state;

[0057] This step aims to have the above-mentioned execution subject input the row count heat map and the feature encoding of the decoded formula symbol obtained through the decoding operation into the decoding network, thereby obtaining the current hidden layer state of the output.

[0058] One implementation, including but not limited to, may be as follows: the row count heatmap and feature encoding are input into a gated recurrent module in the decoding network, and the current hidden state output by the gated recurrent module is obtained. The current hidden state is calculated based on the hidden state at the previous time step, the input at the current time step, and the output of the gating mechanism (update gate and reset gate). This hidden state contains the representation and memory information of the model at a given time step.

[0059] It should be noted that since the decoding operation is performed sequentially and step by step, when performing the current decoding operation, the encoding features of the decoded formula symbol used should, as far as possible, use the feature encoding obtained from the previous decoding operation adjacent to the current decoding operation, so as to utilize the closest possible context to improve the accuracy of the current hidden layer state obtained by this decoding operation. At the same time, considering that the first formula symbol will not have the feature encoding of the decoded formula symbol for use when performing the decoding operation in this step, the current hidden layer state corresponding to the first formula symbol can be set to the initial value in advance.

[0060] The Gated Recurrent Unit (GRU), also known as the Gated Recurrent Unit, is a variant of the Recurrent Neural Network (RNN) used to process sequential data, such as natural language text and time series. The GRU unit primarily addresses the long-term dependency issues of standard RNNs and has the following important functions and benefits:

[0061] 1) Capturing sequence information: The core function of the GRU unit is to capture information in sequence data. It can retain and transmit information from previous time steps so that it can be used for prediction or classification tasks in subsequent time steps.

[0062] 2) Avoiding the vanishing gradient problem: Compared to standard RNNs, GRUs have a more complex gating mechanism to mitigate the vanishing gradient problem. This makes the model easier to train when processing long sequences because it can better propagate gradient information.

[0063] 3) Gating mechanism: GRU includes two important gating mechanisms, namely the Update Gate and the Reset Gate. These gating mechanisms allow the GRU unit to decide which information to update at each time step, thus more flexibly adapting to different sequence data;

[0064] 4) Update Gate: This determines whether the previous hidden state should be retained at the current time step. The sigmoid function is used to control which information should be retained (values close to 1) and which should be discarded (values close to 0).

[0065] 5) Reset Gate: This controls whether the previous hidden state should be ignored at the current time step. The Sigmoid function (also known as the S-type function in Chinese) determines which information should be forgotten (values close to 0) and which should be considered (values close to 1).

[0066] 6) Memory ability: The gating mechanism of the GRU unit enables it to selectively remember or forget information, thereby better adapting to the characteristics of sequence data, which helps to process sequences of various lengths.

[0067] That is, this step can just make use of the above-mentioned characteristics and capabilities of the gated loop module, so that when the feature map and the feature coding of the decoded formula symbol obtained by the decoding operation (for example, assuming that the current prediction is the formula symbol of the 9th step, then the feature coding of the decoded formula symbol obtained by the decoding operation can be the feature coding of the formula symbol of the 8th step) are input into the gated loop module, with the help of its long sequence data and gating mechanism corresponding to multiple rows of formulas, the sequence information can be better captured and managed, so that the output current hidden layer state can more accurately describe the relevant feature information of the formula symbol of the current step, especially under the premise of fully considering the correlation between the formula symbol of the current step and the decoded formula symbol. Of course, in addition to using the gated loop module, it can also be replaced by other functional modules with similar effects, which are not specifically limited here.

[0068] Step 2032: Input the feature map and the current hidden layer state into the decoding network to obtain the region image corresponding to the current formula symbol to be decoded;

[0069] Based on step 2031, this step aims to have the above-mentioned execution entity input the feature map and the current hidden layer state into the conventional attention module already existing in the decoding network, and obtain the regional image output by the attention module corresponding to the formula symbol to be decoded.

[0070] The conventional attention module mentioned in this step is an attention module that should usually be present or standard in a conventional or standard decoding network. It can only provide an attention mechanism to process the feature map, and this step is to input the feature map and the current hidden layer state output by the gated recurrent unit into the conventional attention module, and then fully combine the attention mechanism to determine the context of the formula symbol to be decoded at the current step, that is, the part of the image to be recognized where the formula symbol to be decoded at the current step is located. In other words, it is to obtain the regional image corresponding to the formula symbol to be decoded at the current step, so as to further improve the recognition accuracy of the formula symbol to be decoded at the current step while narrowing the scope of attention and reduce the influence of the formula symbols in the remaining regional images.

[0071] Step 2033: The region image, feature code, current hidden layer state, and row count heat map are combined and predicted through the decoding network to obtain a formula symbol recognition result corresponding to the current decoding operation;

[0072] Based on step 2032, this step aims to have the above-mentioned execution entity perform feature mapping and combination of the mapped features on the regional image, feature coding, current hidden layer state and row count heat map through the fully connected layer in the decoding network, and then perform prediction processing on the combined features to obtain the formula symbol recognition result corresponding to the current decoding operation.

[0073] A specific implementation method may be:

[0074] First, the region image, the feature code, the current hidden layer state, and the row count heat map are feature mapped and the mapped features are combined through the first fully connected layer in the decoding network to obtain a combined feature; then, the combined feature is feature predicted through the second fully connected layer in the decoding network to obtain a formula symbol recognition result corresponding to the current decoding operation.

[0075] Step 204: Generate a formula symbol sequence according to the formula symbol recognition results sequentially output by the decoding network.

[0076] On the basis of step 2033, this step aims to organize the formula symbol recognition results output by the decoding network in each decoding operation performed in sequence into a formula symbol sequence by the above-mentioned execution subject.

[0077] The formula recognition method provided by the embodiment of the present disclosure is based on the conventional overall network architecture composed of an encoding-decoding type network. On the basis of the conventional attention module of the decoding network, a global row attention module is additionally provided. The global row attention module can output a row count heat map for indicating the predicted number of formula rows for the input feature map, thereby making it possible to clearly determine the predicted number of formula rows that should be in the image to be recognized through the different heat levels in the row count heat map, and thus accurately determine each formula symbol of the multi-line formula when the number of formula rows is clear, thereby improving the recognition accuracy of multi-line complex formulas.

[0078] On the basis of the above embodiment, in order to strengthen the understanding of how to obtain the row count heat map through the global row attention module, this embodiment also uses Figure 3 A method for obtaining a row count heatmap using a global row attention module is shown, wherein the process 300 includes the following steps:

[0079] Step 301: The feature map is sequentially passed through the convolution layer and the normalization layer in the preset global row attention module to obtain a converted feature map;

[0080] By sequentially processing feature maps through convolutional and normalization layers, we can better learn and represent features in the data, while accelerating the training process and improving the generalization performance of the model. The convolutional layer is responsible for feature extraction, converting the raw input data into useful feature representations, while the normalization layer helps the network train more stably, avoiding problems such as gradient explosion and gradient vanishing.

[0081] Step 302: Pass the converted feature map through the channel attention submodule in the preset global row attention module to obtain a channel attention enhanced feature map;

[0082] Based on step 301, this step aims to obtain a channel attention enhanced feature map by converting the feature map of the above-mentioned execution subject through the channel attention submodule in the preset global row attention module.

[0083] The channel attention submodule, derived from SENet (Squeeze-and-Excitation Network), is one of its core innovations, designed to enhance the representation capabilities of deep convolutional neural networks, particularly in image classification and related computer vision tasks. Its working principle can be divided into two key steps: Squeeze and Excitation.

[0084] Squeeze: In this step, a global average pooling operation is performed on the feature map of each channel to reduce the feature value of each channel to a single scalar. This means that for each channel, SENet calculates the average value of all pixels on its feature map to obtain a numerical value representing the information of the channel. This value will be used for subsequent weight learning;

[0085] Excitation: In this step, two fully connected layers (typically consisting of a linear layer and an activation function layer, such as ReLU) are used to learn the weights for each channel. These weights are used to enhance or weaken the feature response of each channel. The first fully connected layer takes the output of the squeeze stage as input to produce an intermediate representation, which is then transformed nonlinearly through the activation function. Finally, the second fully connected layer obtains a weight vector for each channel. These weight vectors are used to adjust the importance of each channel in the feature map.

[0086] Based on the above principles, the role of the channel attention module is to enhance the network's attention to different channels in the feature map, so that the network can better capture important feature information. Specifically, it includes the following:

[0087] 1) Adaptive feature weights: SENet adaptively adjusts the feature response of each channel through learned weights. This means that important channels will receive higher weights, while unimportant channels will receive lower weights, thereby reducing the impact of noise and redundant information;

[0088] 2) Enhance useful feature channels: The channel attention module enables the network to better highlight feature channels that are useful for the task. This helps improve the network's discriminative ability and thus improve task performance;

[0089] 3) Reduce overfitting: The channel attention module helps the network generalize better to new samples because it can reduce overfitting to noisy or unimportant features in the training data.

[0090] In general, this step improves the attention of the convolutional neural network to the feature map by introducing adaptive channel weights using the channel attention module of SENet, thereby enhancing the representation ability and generalization performance of the model, making the features after channel attention adjustment more accurate and more conducive to clarifying the number of formula lines contained in the subsequent processing steps.

[0091] Step 303: The feature map after channel attention enhancement is adjusted to a single-channel feature map with a channel number of 1 through the first convolution in the preset global row attention module;

[0092] The purpose of converting the multi-channel feature map into a single-channel feature map by using the first convolution (for example, a convolution with a convolution kernel size of 1×1 can be used specifically) is to reduce the channel dimension, and the mutual weights between the channels can be adjusted by learning weights to extract more useful features and reduce redundant information. In particular, the channel attention adjustment mechanism has been performed in step 302, which will further improve the effect.

[0093] The effects that can be achieved by performing this operation include:

[0094] 1) Reduce computational complexity: Converting to a single-channel feature map through the first convolution can reduce the amount of computation, because in some cases, multi-channel feature maps may contain a lot of redundant or irrelevant information. Reducing the channel dimension to a single channel can reduce the number of model parameters and computational cost;

[0095] 2) Feature interaction and integration: The first convolution not only transforms the channel dimension, but also allows for cross-channel feature interaction and integration. By learning weights, the network can automatically adjust the weights of different channels to better integrate and utilize the information in each channel;

[0096] 3) Extracting useful features: By converting to a single-channel feature map and using weight adjustment, the model can focus on responding more strongly to important features and weakening responses to unimportant features. This helps the model better distinguish feature differences between different categories and improves the model's discriminative ability.

[0097] 4) Reducing redundant information: In some cases, multiple channels in a multi-channel feature map may contain highly correlated information. Through the first convolution transformation, these related information can be merged into one channel, reducing redundancy and improving computational efficiency.

[0098] Step 304: Compress the value of each element in the single-channel feature map into a preset interval range through the S-type activation function in the preset global row attention module, and output the feature map at this time as a row count heat map.

[0099] Based on step 303, this step aims to compress the value of each element in the single-channel feature map to a range of 0 to 1 by the above-mentioned execution subject through the S-type activation function (i.e., Sigmoid activation function) under the preset global row attention module, thereby converting the output result from the feature map into a row count heat map represented in the form of probability. Of course, in addition to using the S-type activation function, other activation functions with similar effects can also be selected, which are not listed here one by one.

[0100] This embodiment provides an implementation process of how a global row attention module processes the input feature map to obtain the output row count heat map through steps 301 to 304. It not only adjusts the attention weights of each channel in the feature map through the channel attention module provided by SENet, but also converts it into a single-channel feature map to perform feature interaction integration, reduce computational costs, and further extract useful features. Finally, it is converted into a probabilistic representation through the S-type activation function, thereby obtaining a row count heat map that can be used to predict the number of rows contained in the formula.

[0101] On the basis of any of the above embodiments, this embodiment further Figure 4 A method for obtaining a complete text recognition result based on a formula symbol sequence is provided, wherein the process 400 includes the following steps:

[0102] Step 401: determining the number of predicted formula rows of multiple formula rows contained in the image to be recognized based on the row count heat map;

[0103] This step can be located in Figure 2After step 205 of the illustrated embodiment, that is, after obtaining the formula symbol sequence output by the formula recognition model in step 205, the above-mentioned execution entity determines the predicted number of formula rows of the multiple rows of formulas contained in the image to be identified based on the heat information in the row counting heat map, that is, in this step, the above-mentioned execution entity first determines how many rows of formulas there are in the image.

[0104] Step 402: Restore the formula symbol sequence according to the predicted number of formula lines to obtain a multi-line formula in text form;

[0105] Based on step 401, this step aims to restore the formula symbols contained in the formula symbol sequence according to the determined number of formula rows by the above-mentioned execution subject (that is, the formula symbols in the same row are spliced and restored in the same row according to the order in the sequence) to obtain a multi-line formula in text form.

[0106] Step 403: Combine the multi-line formula in text form with other text recognition results of the non-multi-line formula part in the image to be recognized according to the original typesetting layout to obtain a complete text recognition result corresponding to the image to be recognized.

[0107] Based on step 402, this step aims to have the above-mentioned execution entity combine the multi-line formulas in text form with other text recognition results of the non-multi-line formula part in the image to be recognized according to the original typesetting layout (that is, the typesetting layout between the formula part and the non-formula part presented in the image to be recognized) to obtain a complete text recognition result corresponding to the image to be recognized.

[0108] It should be noted that step 401 and step 402 can be combined with other embodiments to form a new embodiment when step 403 is not used in subsequent processing steps. This embodiment only exists as a preferred embodiment that exists simultaneously.

[0109] To deepen understanding, this disclosure also provides a specific implementation solution in combination with a specific application scenario, see Figure 5 A schematic diagram of a process of processing an image to be recognized by a formula recognition model to obtain a formula recognition result is shown below:

[0110] This embodiment is built on an encoder-decoder network. The encoder network 510 converts the input image into a feature map F. The decoder network 520, after a series of processing, converts the feature map F into a sequence of formula symbols for output. The same approach is used for decoding single-line and multi-line formulas. First, weakly supervised formula line counting is used to alert the network to the content of each line of the formula. This count information is then used to assist in the subsequent sequential decoding of each line of the formula content.

[0111] The encoding network 510 can specifically adopt DenseNet (densely connected network). For a three-channel image with input width and height of W and H, DenseNet first uses a convolutional layer with a kernel of 7 and a stride of 2 to downsample the input image, outputting a feature map with a width of W / 2 and a height of H / 2, respectively. It then uses a maximum pooling layer with a kernel of 2 and a stride of 2 to further downsample the input image, outputting a feature map with a width of W / 4 and a height of H / 4, respectively. The feature map is then further encoded using a combination of multiple convolutional layers with kernels of 1 and 3 to obtain the final output feature map F, which has a width of W / 16 and a height of H / 16, respectively, and a total of 684 channels.

[0112] The decoding network 520, given a feature map F with a width of W / 16 and a height of H / 16, extracted by the backbone feature extraction network DenseNet, processes it and outputs the final result. The input feature map F is first processed using a 1×1 convolution to convert it into an intermediate-layer feature map M with a width of W / 16 and a height of H / 16, respectively, and 512 channels. The intermediate-layer feature map M is then fed into the attention module 521 and the global row attention module 522. The global row attention module 522 generates a row count heatmap Q with a width of W / 16 and a height of H / 16, respectively, and 1 channel. The row count heatmap Q is combined with the feature code E of the symbol decoded in the previous step and fed into the GRU network 523 to generate the current hidden layer state h_t. The feature map F, the intermediate-layer feature map M, and the current hidden layer state h_t are fed into the attention module 521 to obtain the contextual semantic information C. The context semantic information C, the feature code E of the symbol decoded in the previous step, the current hidden layer state h_t, and the row count heat map Q are feature mapped by the fully connected layer 524 and combined together to obtain the feature T. Then, a prediction is performed by another fully connected layer 524 to obtain the final formula symbol sequence prediction result.

[0113] The following is a key introduction to the global row attention module mentioned above:

[0114] Most of the current research on counting belongs to fully supervised counting, which requires obtaining the location information of each instance, that is, counting through detection methods. When applying counting to multi-line formulas, it is necessary to not rely on the position annotation of each line of formula, and only perform weakly supervised counting by annotating the number of formula lines. The global row attention module designed in this embodiment adopts a dual-branch structure. The two branches form differences in receptive fields by using convolution kernels of different sizes (one is 3×3 and the other is 5×5), thereby constructing multi-scale information modeling capabilities. After the input feature map M passes through the convolution layer and the normalization layer, the converted feature map obtained is input into the channel attention submodule to obtain enhanced features. The channel attention submodule used here is the channel attention submodule proposed in SENet. After obtaining the enhanced features, a 1x1 convolution is used to adjust the number of feature channels to 1. After the 1×1 convolution, the Sigmoid activation function is used to compress the value of each element in the feature map into the range of (0, 1), and the row count heat map Q is obtained. The row count prediction value is obtained by adding up all the elements of the row count heat map Q, and the row count prediction value is supervised by applying the L1 loss function.

[0115] That is, the formula recognition method based on global row attention proposed in this embodiment can recognize formulas in images, whether they are simple formulas or multi-line formulas with complex nested structures, and can achieve a high recognition accuracy.

[0116] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a formula recognition device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0117] like Figure 6As shown, the formula recognition device 600 of this embodiment may include: a feature map extraction unit 601, a global row attention module processing unit 602, a feature decoding unit 603, and a formula symbol sequence generation unit 604. Among them, the feature map extraction unit 601 is configured to obtain an image to be recognized containing multiple rows of formulas, and use the encoding network in the preset formula recognition model to obtain the feature map of the image to be recognized; the global row attention module processing unit 602 is configured to input the feature map into the preset global row attention module in the decoding network in the preset formula recognition model to obtain a row count heat map; the feature decoding unit 603 is configured to use the decoding network to perform decoding operations on the features in the feature map in sequence according to the following sub-units, and the features in the feature map correspond to different formula symbols respectively: the current hidden layer state acquisition sub-unit 6031 is configured to decode the row count heat map and the decoded formula symbols obtained by the decoding operation. The feature code is input into the decoding network to obtain the current hidden layer state; wherein, the current hidden layer state corresponding to the first formula symbol is set as the initial value in advance; the region image determination unit 6032 is configured to input the feature map and the current hidden layer state into the decoding network to obtain the region image corresponding to the current formula symbol to be decoded; the feature combination and prediction unit 6033 is configured to combine and predict the region image, feature code, current hidden layer state and row count heat map through the decoding network to obtain the formula symbol recognition result corresponding to the current decoding operation; the formula symbol sequence generation unit 604 is configured to generate a formula symbol sequence according to the formula symbol recognition results outputted sequentially by the decoding network.

[0118] In this embodiment, the specific processing of the feature map extraction unit 601, the global row attention module processing unit 602, the feature decoding unit 603, and the formula symbol sequence generation unit 604 and the technical effects thereof can be referred to in detail. Figure 2 The relevant descriptions of steps 201-204 in the corresponding embodiment are not repeated here.

[0119] In some optional implementations of this embodiment, the formula recognition device 600 may further include:

[0120] The channel number adjustment unit is configured to adjust the channel number of the feature map to the required channel number through the first convolution in response to the channel number of the feature map not matching the channel number of the decoding network, so that the decoding network uses the feature map with the adjusted channel number for decoding.

[0121] In some optional implementations of this embodiment, the global row attention module processing unit 602 is further configured to:

[0122] The feature map is passed through the convolution layer and normalization layer in the preset global row attention module in sequence to obtain the converted feature map;

[0123] The converted feature map is passed through the channel attention module in the preset global row attention module to obtain the channel attention enhanced feature map;

[0124] The feature map after channel attention enhancement is adjusted to a single channel feature map with a channel number of 1 through the first convolution in the preset global row attention module;

[0125] The single-channel feature map is compressed into a preset range by the S-type activation function in the preset global row attention module, and the feature map at this time is output as a row count heat map.

[0126] In some optional implementations of this embodiment, the preset global row attention module adopts a multi-branch structure, and different branch structures use convolution kernels of different sizes.

[0127] In some optional implementations of this embodiment, the present invention further includes: a global row attention module training unit, the global row attention module training unit is further configured to:

[0128] Obtain training samples that only annotate the actual number of rows for sample heat maps containing multiple rows of formulas;

[0129] The training samples are used to perform weakly supervised training on the initial global row attention module based on the loss function of the mean absolute error to obtain the trained preset global row attention module.

[0130] In some optional implementations of this embodiment, the feature combination and prediction subunit 6033 may be further configured to:

[0131] The region image, feature code, current hidden layer state, and row count heat map are passed through the first fully connected layer in the decoding network to perform feature mapping and combine the mapped features to obtain the combined features;

[0132] The combined features are passed through the second fully connected layer in the decoding network for feature prediction to obtain the formula symbol recognition result corresponding to the current decoding operation.

[0133] In some optional implementations of this embodiment, the formula recognition device 600 may further include:

[0134] The preprocessing unit is configured to preprocess the image to be recognized; wherein the preprocessing includes at least one of: direction correction processing, binarization processing, grayscale processing, contrast processing, and clarity processing.

[0135] In some optional implementations of this embodiment, the formula recognition device 600 may further include:

[0136] a prediction formula line number determination unit, configured to determine the number of prediction formula lines of the multiple lines of formulas contained in the image to be recognized based on the line count heat map;

[0137] The formula restoration unit is configured to restore the formula symbol sequence according to the predicted number of formula lines to obtain a multi-line formula in text form.

[0138] In some optional implementations of this embodiment, the formula recognition device 600 may further include:

[0139] The complete text recognition result generating unit is configured to combine the multi-line formula in text form with other text recognition results of the non-multi-line formula part in the image to be recognized according to the original typesetting layout to obtain a complete text recognition result corresponding to the image to be recognized.

[0140] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The formula recognition apparatus provided by this embodiment is based on the conventional overall network architecture of the encoding-decoding type network, and by adding a global row attention module on the basis of the conventional attention module of the decoding network. The global row attention module can output a row count heat map for indicating the predicted number of formula rows for the input feature map, thereby making it possible to clearly predict the number of formula rows that should be in the image to be recognized through the different heat levels in the row count heat map, and thus accurately determine each formula symbol of the multi-line formula when the number of formula rows is clear, thereby improving the recognition accuracy of multi-line complex formulas.

[0141] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the formula recognition method described in any of the above embodiments can be implemented when the at least one processor executes them.

[0142] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the formula recognition method described in any of the above embodiments when executed.

[0143] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which, when executed by a processor, can implement the steps of the formula recognition method described in any of the above embodiments.

[0144] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0145] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0146] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0147] The computing unit 701 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the formula recognition method. For example, in some embodiments, the formula recognition method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the formula recognition method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the formula recognition method in any other appropriate manner (e.g., by means of firmware).

[0148] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0152] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0153] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship arises through computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and virtual private server (VPS) services.

[0154] According to the technical solution of the embodiment of the present disclosure, on the basis of the conventional overall network architecture composed of an encoding-decoding type network, a global row attention module is additionally provided on the basis of the conventional attention module of the decoding network. The global row attention module can output a row count heat map for indicating the predicted number of formula rows for the input feature map, thereby making it possible to clearly predict the number of formula rows that should be in the image to be identified through the different heat levels in the row count heat map, thereby making it possible to accurately determine each formula symbol of the multi-line formula when the number of formula rows is clear, thereby improving the recognition accuracy of multi-line complex formulas.

[0155] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0156] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A formula recognition method, comprising: Obtain an image to be identified that contains multiple lines of formulas, and use an encoding network in a preset formula recognition model to obtain a feature map of the image to be identified; Passing the feature map sequentially through the convolution layer and the normalization layer in the preset global row attention module in the decoding network in the preset formula recognition model to obtain a converted feature map; Passing the converted feature map through the channel attention submodule in the preset global row attention module to obtain a channel attention enhanced feature map; The channel attention enhanced feature map is adjusted to a single channel feature map with a channel number of 1 by the first convolution in the preset global row attention module; Compress the value of each element in the single-channel feature map into a preset range through the S-type activation function under the preset global row attention module, and output the feature map at this time as a row count heat map; The decoding network is used to sequentially decode the features in the feature map in the following manner, where the features in the feature map correspond to different formula symbols: Inputting the row count heat map and the feature codes of the decoded formula symbols obtained by the decoding operation into the gated loop module in the decoding network to obtain a current hidden layer state; wherein the current hidden layer state corresponding to the first formula symbol is previously set to an initial value; Inputting the feature map and the current hidden layer state into an attention module in the decoding network to obtain a region image output by the attention module corresponding to the current formula symbol to be decoded; The region image, the feature code, the current hidden layer state, and the row count heat map are subjected to feature mapping and combination of the mapped features through the fully connected layer of the decoding network, and the combined features are subjected to prediction processing to obtain a formula symbol recognition result corresponding to the current decoding operation; A formula symbol sequence is generated according to the formula symbol recognition results outputted sequentially by the decoding network.

2. The method according to claim 1, further comprising: In response to the number of channels of the feature map not matching the number of channels of the decoding network, the number of channels of the feature map is adjusted to a required number of channels through a first convolution, so that the decoding network uses the feature map with the adjusted channel number for decoding.

3. The method according to claim 1, wherein The preset global row attention module adopts a multi-branch structure, and different branch structures use convolution kernels of different sizes.

4. The method according to claim 1, wherein The training process of the preset global row attention module includes: Obtain training samples that only annotate the actual number of rows for sample heat maps containing multiple rows of formulas; The training samples are used to perform weakly supervised training on the initial global row attention module based on the loss function of the mean absolute error to obtain a trained preset global row attention module.

5. The method according to claim 1, wherein The combining and predicting the region image, the feature code, the current hidden layer state, and the row count heat map through the decoding network to obtain a formula symbol recognition result corresponding to the current decoding operation includes: Performing feature mapping on the region image, the feature code, the current hidden layer state, and the row count heat map through the first fully connected layer in the decoding network and combining the mapped features to obtain a combined feature; The combined features are passed through the second fully connected layer in the decoding network to perform feature prediction to obtain a formula symbol recognition result corresponding to the current decoding operation.

6. The method according to claim 1, further comprising: The image to be identified is preprocessed; wherein the preprocessing includes: at least one of direction correction processing, binarization processing, grayscale processing, contrast processing, and clarity processing.

7. The method according to any one of claims 1 to 6, further comprising: Determining the number of predicted formula rows of the multiple formula rows contained in the image to be identified according to the row count heat map; The formula symbol sequence is restored according to the predicted number of formula lines to obtain a multi-line formula in text form.

8. The method according to claim 7, further comprising: The multi-line formula in text form and other text recognition results of the part other than the multi-line formula in the image to be recognized are combined according to the original typesetting layout to obtain a complete text recognition result corresponding to the image to be recognized.

9. A formula recognition device comprising: a feature map extraction unit configured to obtain an image to be recognized containing multiple lines of formulas, and obtain a feature map of the image to be recognized using an encoding network in a preset formula recognition model; a global row attention module processing unit configured to sequentially pass the feature map through the convolution layer and the normalization layer in the preset global row attention module in the decoding network in the preset formula recognition model to obtain a converted feature map; Passing the converted feature map through the channel attention submodule in the preset global row attention module to obtain a channel attention enhanced feature map; The channel attention enhanced feature map is adjusted to a single channel feature map with a channel number of 1 by the first convolution in the preset global row attention module; Compress the value of each element in the single-channel feature map into a preset range through the S-type activation function under the preset global row attention module, and output the feature map at this time as a row count heat map; The feature decoding unit is configured to use the decoding network to perform decoding operations on the features in the feature map according to the following sub-units in sequence, where the features in the feature map correspond to different formula symbols respectively: a current hidden layer state acquisition subunit configured to input the row count heat map and the feature code of the decoded formula symbol obtained by the decoding operation into the gated loop module in the decoding network to obtain the current hidden layer state; wherein the current hidden layer state corresponding to the first formula symbol is set to an initial value in advance; a region image determining unit configured to input the feature map and the current hidden layer state into an attention module in the decoding network, and obtain a region image output by the attention module corresponding to the current formula symbol to be decoded; a feature combination and prediction unit configured to perform feature mapping on the region image, the feature code, the current hidden layer state, and the row count heat map through the fully connected layer of the decoding network, combine the mapped features, and perform prediction processing on the combined features to obtain a formula symbol recognition result corresponding to the current decoding operation; The formula symbol sequence generating unit is configured to generate a formula symbol sequence according to the formula symbol recognition results sequentially output by the decoding network.

10. The apparatus according to claim 9, further comprising: The channel number adjustment unit is configured to adjust the channel number of the feature map to the required channel number through a first convolution in response to the channel number of the feature map not matching the channel number of the decoding network, so that the decoding network uses the feature map with the adjusted channel number for decoding.

11. The device according to claim 9, wherein The preset global row attention module adopts a multi-branch structure, and different branch structures use convolution kernels of different sizes.

12. The apparatus according to claim 9, further comprising: A global row attention module training unit, wherein the global row attention module training unit is configured to: Obtain training samples that only annotate the actual number of rows for sample heat maps containing multiple rows of formulas; The training samples are used to perform weakly supervised training on the initial global row attention module based on the loss function of the mean absolute error to obtain a trained preset global row attention module.

13. The device according to claim 9, wherein The feature combination and prediction subunit is further configured to: Performing feature mapping on the region image, the feature code, the current hidden layer state, and the row count heat map through the first fully connected layer in the decoding network and combining the mapped features to obtain a combined feature; The combined features are passed through the second fully connected layer in the decoding network to perform feature prediction to obtain a formula symbol recognition result corresponding to the current decoding operation.

14. The apparatus according to claim 9, further comprising: The preprocessing unit is configured to preprocess the image to be identified; wherein the preprocessing includes at least one of: direction correction processing, binarization processing, grayscale processing, contrast processing, and clarity processing.

15. The apparatus according to any one of claims 9 to 14, further comprising: a prediction formula row number determination unit, configured to determine the number of prediction formula rows of the plurality of rows of formulas contained in the image to be identified according to the row count heat map; The formula restoration unit is configured to restore the formula symbol sequence according to the predicted number of formula lines to obtain a multi-line formula in text form.

16. The apparatus according to claim 15, further comprising: The complete text recognition result generating unit is configured to combine the multi-line formula in text form with other text recognition results of the non-multi-line formula part in the image to be recognized according to the original typesetting layout to obtain a complete text recognition result corresponding to the image to be recognized.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the formula recognition method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the formula recognition method according to any one of claims 1 to 8.

19. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the formula recognition method according to any one of claims 1 to 8.

Images