Formula recognition method, device, equipment and storage medium

Abstract

The present disclosure provides a formula recognition method, apparatus, device, and storage medium, which relate to the field of artificial intelligence technology, specifically to technical fields such as computer vision, OCR, and deep learning, and can be applied to scenarios such as intelligent finance and intelligent government affairs. The specific implementation solution includes: extracting features from a target image to obtain a first feature map corresponding to the target image; extracting a second feature map corresponding to a handwritten formula area and a third feature map corresponding to a printed formula area from the first feature map; using a preset handwritten formula recognition network to recognize the second feature map to obtain a handwritten formula recognition result; using a preset printed formula recognition network to recognize the third feature map to obtain a printed formula recognition result. The present disclosure can improve the recognition accuracy and efficiency of both handwritten formulas and printed formulas simultaneously.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technologies, specifically to technical fields such as computer vision, OCR, deep learning, etc., and can be applied to scenarios such as intelligent finance and intelligent government affairs. In particular, it relates to a formula recognition method, device, equipment, and storage medium. Background Art

[0002] Formula recognition is the process of converting formulas in images into editable formula texts. For example, the format of the editable formula text can be the standard LaTeX format.

[0003] Currently, the methods of formula recognition can include: traditional formula recognition methods based on symbol segmentation and symbol recognition, sequence formula recognition methods based on deep learning, structured formula recognition methods based on deep learning, etc.

[0004] However, the current formula recognition methods generally only apply to the recognition of formulas in printed fonts, or only apply to the recognition of formulas in handwritten fonts. When the image contains both formulas in printed fonts and formulas in handwritten fonts, the current formula recognition methods cannot simultaneously ensure the recognition accuracy of formulas in printed fonts and the recognition accuracy of formulas in handwritten fonts. Summary of the Invention

[0005] The present disclosure provides a formula recognition method, device, equipment, and storage medium, which can simultaneously improve the recognition accuracy and efficiency of handwritten formulas and printed formulas.

[0006] According to the first aspect of the present disclosure, a formula recognition method is provided. The method includes:

[0007] Performing feature extraction on a target image to obtain a first feature map corresponding to the target image; extracting a second feature map corresponding to the handwritten formula area and a third feature map corresponding to the printed formula area from the first feature map; using a preset handwritten formula recognition network to recognize the second feature map to obtain a handwritten formula recognition result; using a preset printed formula recognition network to recognize the third feature map to obtain a printed formula recognition result.

[0008] According to the second aspect of the present disclosure, a formula recognition device is provided. The device includes: an extraction unit and a recognition unit.

[0009] The extraction unit is configured to perform feature extraction on a target image to obtain a first feature map corresponding to the target image; extract a second feature map corresponding to the handwritten formula area and a third feature map corresponding to the printed formula area from the first feature map.

[0010] An identification unit is configured to identify the second feature map by using a preset handwritten formula recognition network to obtain a handwritten formula recognition result; and identify the third feature map by using a preset printed formula recognition network to obtain a printed formula recognition result.

[0011] According to a third aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to the first aspect.

[0012] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to the first aspect.

[0013] According to a fifth aspect of the present disclosure, there is provided a computer program product including a computer program, where the computer program, when executed by a processor, implements the method according to the first aspect.

[0014] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understandable through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0016] Figure 1 is a schematic flowchart of the formula recognition method provided by an embodiment of the present disclosure;

[0017] Figure 2 is a schematic diagram of the principle of the formula recognition method provided by an embodiment of the present disclosure;

[0018] Figure 3 is provided by an embodiment of the present disclosure Figure 1 a schematic flowchart of one implementation of S102 in;

[0019] Figure 4 is provided by an embodiment of the present disclosure Figure 1 another schematic flowchart of the implementation of S102 in;

[0020] Figure 5 is provided by an embodiment of the present disclosure Figure 1 a schematic flowchart of one implementation of S103 in;

[0021] Figure 6 is provided by an embodiment of the present disclosure Figure 1 another schematic flowchart of the implementation of S103 in;

[0022] Figure 7 A schematic flowchart of one implementation of S104 provided by an embodiment of the present disclosure Figure 1 in

[0023] Figure 8 Another schematic flowchart of the implementation of S104 provided by an embodiment of the present disclosure Figure 1 in

[0024] Figure 9 A schematic diagram of the composition of a formula recognition device provided by an embodiment of the present disclosure

[0025] Figure 10 A schematic block diagram of an exemplary electronic device 1000 that can be used to implement the embodiments of the present disclosure provided by an embodiment of the present disclosure Detailed implementation manners

[0026] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to assist in understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description

[0027] It should be understood that in various embodiments of the present disclosure, the character " / " generally represents an "or" relationship between the associated objects before and after. Terms such as "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features

[0028] Formula recognition is the process of converting formulas in pictures into editable formula texts. For example, the format of the editable formula text can be the standard LaTeX format

[0029] For example, formulas widely exist in many fields such as the education field and the office field. Existing optical character recognition (OCR) can detect and recognize ordinary texts well, but the detection and recognition effect for formulas is not ideal enough. Different from ordinary texts, formulas often have complex spatial structures and diverse writing styles, and there will be a situation where handwritten and printed formulas coexist. For example, there may be both printed formulas and handwritten formulas on a test paper in an educational scenario. This situation of mixed formulas poses a great challenge to formula detection and recognition. The types of formulas can include many formulas such as mathematical formulas and chemical formulas. The main purpose of formula recognition is to convert formulas from images or handwritten documents into text formats, languages, or symbol representations so that they can be edited and modified on a computer

[0030] Currently, the methods of formula recognition can include: traditional formula recognition methods based on symbol segmentation and symbol recognition, sequence formula recognition methods based on deep learning, structured formula recognition methods based on deep learning, etc.

[0031] For example, the traditional formula recognition method based on symbol segmentation and symbol recognition generally includes three steps: symbol segmentation, symbol recognition, and structure analysis, and outputs the recognition result. The sequence formula recognition method based on deep learning regards formula recognition as an image-to-sequence translation task, uses the attention mechanism to decode each symbol in the formula in turn, and outputs the recognition result. The structured formula recognition method based on deep learning regards each formula as a syntax tree, decodes it in the way from the parent node to the child node of the tree, and then converts the tree-shaped decoding result into a formula according to the pre-defined rules, and outputs the recognition result.

[0032] However, the current formula recognition methods generally only apply to the recognition of formulas in printed fonts, or only apply to the recognition of formulas in handwritten fonts. When the picture contains both formulas in printed fonts and formulas in handwritten fonts at the same time, the current formula recognition methods cannot ensure the recognition accuracy of formulas in printed fonts and the recognition accuracy of formulas in handwritten fonts at the same time.

[0033] For example, the existing technical solutions perform formula detection and formula recognition step by step. First, the image within the formula detection box is obtained, and after being cut out, subsequent formula recognition is carried out, resulting in low formula recognition efficiency; moreover, most of the existing formula detection and recognition solutions are designed only for printed formula recognition or only for handwritten formula recognition. However, in actual situations, in the pictures collected before formula recognition, most are cases where handwritten formulas and printed formulas coexist, resulting in problems in the existing technology in terms of limited scenarios on the one hand. On the other hand, when handwritten formulas and printed formulas coexist in the picture, the handwritten formulas and printed formulas will affect each other's formula recognition accuracy, resulting in low detection and recognition accuracy.

[0034] Under this background art, the present disclosure provides a formula recognition method that can improve the recognition accuracy and efficiency of both handwritten formulas and printed formulas at the same time.

[0035] Exemplarily, the execution subject of this formula recognition method can be a computer or a server, or it can also be other devices with data processing capabilities. There is no limitation on the execution subject of this method here.

[0036] In some embodiments, the server can be a single server, or it can also be a server cluster composed of multiple servers. In some implementation manners, the server cluster can also be a distributed cluster. The present disclosure does not limit the specific implementation manner of the server.

[0037] Figure 1 It is a schematic flowchart of the formula recognition method provided by the embodiments of the present disclosure. As Figure 1 shown, the method may include:

[0038] S101. Extract features from the target image to obtain a first feature map corresponding to the target image.

[0039] Exemplarily, the target image can be obtained by means such as optical disc reading, digital camera shooting, or scanning. The target image may only include printed formulas, may only include handwritten formulas, or may include both printed formulas and handwritten formulas. The method for extracting features from the target image may include operations such as convolution and pooling.

[0040] For example, the corresponding test paper area can be photographed using a mobile phone camera to obtain a three-channel target image with a width and height of W and H. Extracting features from the target image may include: performing operations such as convolution and pooling on the input three-channel target image with a width and height of W and H to obtain a first feature map corresponding to the target image.

[0041] S102. Extract a second feature map corresponding to the handwritten formula area and a third feature map corresponding to the printed formula area from the first feature map.

[0042] Exemplarily, the handwritten formula area and the printed formula area in the target image can be detected, and according to the detected handwritten formula area and printed formula area, corresponding areas are selected from the first feature map to obtain the second feature map and the third feature map.

[0043] When the target image only contains handwritten formulas, the first feature map extracted only contains the features of the handwritten formulas, and correspondingly, only the second feature map of the handwritten formula area can be extracted from the first feature map; when the target image only contains printed formulas, the first feature map extracted only contains the features of the printed formulas, and correspondingly, only the second feature map of the printed formula area can be extracted from the first feature map; when the target image contains both a handwritten formula area and a printed formula area, the first feature map extracted contains both the features of the handwritten formulas and the features of the printed formulas, and correspondingly, both the second feature map of the handwritten formulas and the third feature map of the printed formulas can be extracted from the first feature map.

[0044] For example, in a three-channel target image with width W and height H that contains both a handwritten formula area and a printed formula area, the first feature map extracted from the original image contains both handwritten formula features and printed formula features. Extracting from the first feature map can yield both a second feature map of the handwritten formula area and a third feature map corresponding to the printed formula area.

[0045] S103. Use a preset handwritten formula recognition network to recognize the second feature map to obtain a handwritten formula recognition result.

[0046] Exemplarily, the preset handwritten formula recognition network can be obtained by training a neural network using the feature map of a handwritten formula image. For example, the feature map of a handwritten formula image can be used as the input, and the text of the handwritten formula contained in the handwritten formula image can be used as the output to train the neural network to obtain the handwritten formula recognition network. The implementation of the handwritten formula recognition network is not limited here. Using the handwritten formula recognition network to perform formula recognition on the second feature map can obtain a handwritten formula recognition result.

[0047] For example, the handwritten formula in the target image can be "hello", and the printed formula can be "aple". After obtaining the first feature map of the target image, the second feature map of "hello" and the third feature map of "aple" are obtained from the first feature map of the target image. Using the handwritten formula recognition network can recognize the second feature map to obtain a handwritten formula recognition result, that is, the text of "hello".

[0048] S104. Use a preset printed formula recognition network to recognize the third feature map to obtain a printed formula recognition result.

[0049] Exemplarily, the preset printed formula recognition network can be obtained by training a neural network using the feature map of a printed formula image. For example, the feature map of a printed formula image can be used as the input, and the text of the printed formula contained in the printed formula image can be used as the output to train the neural network to obtain the printed formula recognition network. The implementation of the printed formula recognition network is not limited here. Using the printed formula recognition network to perform formula recognition on the third feature map can obtain a printed formula recognition result.

[0050] For example, the handwritten formula in the target image can be "hello", and the printed formula can be "aple". After obtaining the first feature map of the target image, the second feature map of "hello" and the third feature map of "aple" are obtained from the first feature map of the target image. Using the printed formula recognition network can recognize the third feature map to obtain a printed formula recognition result, that is, the text of "aple".

[0051] In the embodiments of the present disclosure, by extracting features from a target image, a first feature map corresponding to the target image is obtained. Then, a second feature map corresponding to the handwritten formula area and a third feature map corresponding to the printed formula area are extracted from the first feature map, and the second feature map and the third feature map are respectively recognized to obtain a handwritten formula recognition result and a printed formula recognition result. It realizes that the two processes of handwritten formula recognition and printed formula recognition first share the first feature map, and then are respectively recognized based on the separated second feature map and third feature map, which can make the recognition of handwritten formulas and printed formulas not affect each other when performing formula recognition on the same target image, realizes the end-to-end recognition from the target image to the formula recognition result, and improves the recognition accuracy and efficiency of handwritten formulas and printed formulas at the same time.

[0052] Optionally, the formula recognition method provided in the embodiments of the present disclosure can be used as an internal module of a complete document analysis and recognition product, or can be used as a separate formula product, which is not limited here.

[0053] Exemplarily, in the embodiments of the present disclosure, the functions of the formula recognition method can be implemented by a formula detection module, a feature extraction module, a handwritten formula recognition module, a printed formula recognition module, etc. For example, Figure 2 is a schematic diagram of the principle of the formula recognition method provided in the embodiments of the present disclosure. As Figure 2 shown, after obtaining the target image, the target image can be input into the formula detection module, and the formula detection module can extract features from the target image to obtain a first feature map corresponding to the target image. After obtaining the first feature map, the first feature map can be input into the feature extraction module, and the feature extraction module can output a second feature map corresponding to the handwritten area and a third feature map corresponding to the printed area according to the first feature map. After obtaining the second feature map and the third feature map, the second feature map can be input into the handwritten formula recognition module, and the third feature map can be input into the printed formula recognition module. The handwritten formula recognition module can output a handwritten formula recognition result according to the second feature map, and the printed formula recognition module can output a printed formula recognition result according to the third feature map.

[0054] In Figure 2 the process shown, the feature extraction module can connect the formula detection module and the two formula recognition modules (i.e., the handwritten formula recognition module and the printed formula recognition module) in series, so that the two formula recognition modules can share the first feature map and jointly promote the learning of features, thereby realizing the end-to-end recognition from the target image to the formula recognition result.

[0055] Figure 3 is a schematic diagram of an implementation process of S102 provided in the embodiments of the present disclosure. As Figure 1 shown, in some embodiments, the above S102 may include: Figure 3 shown, in some embodiments, the above S102 may include:

[0056] S301. Generate a score map and a coordinate map corresponding to the first feature map according to the first feature map. The score map is used to indicate the classification result of the pixel region of the first feature map, and the coordinate map is used to indicate the position coordinates of the pixel region of the first feature map.

[0057] Among them, the classification result includes a background region, or a handwritten formula region, or a printed formula region.

[0058] Exemplarily, a convolutional neural network can be used to perform a convolutional operation on the first feature map to change the number of channels of the first feature map and convert the first feature map into a score map and a coordinate map. Among them, the score map has three channels, representing the background class, the handwritten formula class, and the printed formula class respectively. Each pixel region in the score map has scores belonging to the background class, the handwritten formula class, and the printed formula class. The pixel region is a region containing at least one pixel point. The sum of the scores of the background class, the handwritten formula class, and the printed formula class is 1. The score map represents the classification result of each pixel point. The value of each pixel point in each pixel region of the coordinate map is the normalized distance from the pixel point to the four corner points of the bounding box. The bounding box is the bounding box obtained by processing the first feature map through a neural network. For example, the target detection model YOLO (you only look once, YOLO) can be used for coordinate normalization.

[0059] For example, a convolutional neural network can be used to perform a convolutional operation on the first feature map of the target image to convert the first feature map into a score map and a coordinate map. There is a pixel point 1 in the score map, and the scores of the three channels are: 0.1, 0.7, 0.2. There is also a pixel point 2, and the scores of the three channels are: 0.7, 0.2, 0.7. There is also a pixel point 3, and the scores of the three channels are: 0.1, 0.2, 0.7, respectively representing the probability sizes corresponding to the background class, the handwritten formula class, and the printed formula class for this pixel point, and the value of this pixel point has been positionally normalized using the YOLO model respectively.

[0060] S302. Extract a second feature map corresponding to the handwritten formula region and a third feature map corresponding to the printed formula region from the first feature map according to the coordinate map and the score map.

[0061] Exemplarily, when the score of the background class among the scores of the pixel points in the score map is the largest, it indicates that the probability of this pixel point belonging to the background area is relatively high, so this pixel point is not recognized; when the score of the handwritten formula class among the scores of the pixel points in the score map is the largest, it indicates that the probability of this pixel point belonging to the handwritten formula area is relatively high, so this pixel point should be divided into the second feature map of the handwritten formula area; when the score of the printed formula class among the scores of the pixel points in the score map is the largest, it indicates that the probability of this pixel point belonging to the printed formula area is relatively high, so this pixel point should be divided into the third feature map of the printed formula area. The divided area is determined by the coordinate positions of the pixel points included in the coordinate map of the pixel points.

[0062] For example, there is a pixel point 1 in the score map, and the scores of the three channels are: 0.1, 0.7, 0.2. There is also a pixel point 2, and the scores of the three channels are: 0.7, 0.2, 0.7. There is also a pixel point 3, and the scores of the three channels are: 0.1, 0.2, 0.7. Then pixel point 1 belongs to the second feature map, pixel point 3 belongs to the third feature map, and the divided area is determined according to the coordinate position information of the pixel points.

[0063] In this embodiment, by obtaining the score map and the coordinate map, the pixel points of the background class, handwritten formula class, and printed formula class are distinguished, which improves the recognition speed and accuracy of the handwritten area and the printed area, and further improves the accuracy and efficiency of formula recognition.

[0064] Figure 4 For the Figure 1 another implementation flow diagram of S102 provided by the embodiments of the present disclosure. As Figure 4 shown, in some embodiments, the above S102 may further include:

[0065] S401. Extract features from the target image to obtain fourth feature maps with at least two resolutions.

[0066] Exemplarily, in S401, the text detection network EAST may be used to extract features from the target image.

[0067] For example, obtain a three-channel target image with a width and height of W and H. First, use a convolutional layer with 64 kernels of size 7 and a stride of 2 to perform convolution on the input target image, and output a fourth feature Figure 1 with 64 channels and a width and height of W / 2 and H / 2 respectively. Figure 1 Input the feature Figure 2 into a max-pooling layer with a kernel of size 3 and a stride of 2 for further downsampling, and output a fourth feature Figure 2Perform convolution in a series of residual modules, and output a fourth feature map with a width and height of W / 32 and H / 32. Figure 3 .

[0068] S402. Fuse fourth feature maps with at least two resolutions to obtain a first feature map.

[0069] Exemplarily, the neural network model for feature fusion can be: a neural network structure model similar to U-Net.

[0070] For example, the convolutional layer and upsampling layer in the neural network structure model of U-Net perform convolution processing on the fourth feature Figure 1 and the fourth feature Figure 3 to respectively output fourth feature maps with a width and height of W / 4 and H / 4 Figure 4 and the fourth feature Figure 5 . Stack the fourth feature Figure 2 , the fourth feature Figure 4 and the fourth feature Figure 5 pixel by pixel, and output a first feature map with a width and height of W / 4 and H / 4.

[0071] That is to say, the formula detection module shown in the above Figure 2 can include the text detection network EAST and a neural network structure model similar to U-Net. The text detection network EAST and the neural network structure model similar to U-Net can implement the function of extracting the first feature map.

[0072] In this embodiment, by first extracting fourth feature maps with different resolutions and then fusing the fourth feature maps with different resolutions to obtain the first feature map, the character features in the first feature map can be made more abundant, further improving the accuracy and efficiency of formula recognition.

[0073] In some embodiments, before the above S103, the method may further include: aligning the second feature map to the target size. Before the above S104, the method may further include: aligning the third feature map to the target size.

[0074] Exemplarily, the alignment process can use the Region of Interest (ROI) align model. For example, align a second feature map with a width and height of W / 4 and H / 4 to a size of 8*32 using the ROI align model, or align a third feature map with a width and height of W / 4 and H / 4 to a size of 8*32 using the ROI align model.

[0075] In this embodiment, by aligning the sizes of the second feature map and the third feature map with uncertain sizes, the pixel point features in the aligned second feature map and third feature map become richer, and using a feature map with a fixed size for recognition further improves the accuracy and efficiency of formula recognition.

[0076] Figure 5 For the Figure 1 embodiment provided by the present disclosure, Figure 5 as shown in

[0077] Figure [not provided], in some embodiments, the above S103 may include:

[0078] S501. Convolve the second feature map to obtain the first intermediate layer feature map corresponding to the second feature map.

[0079] Exemplarily, a convolutional neural network model may be used to convolve the second feature map to obtain the first intermediate layer feature map corresponding to the second feature map.

[0080] For example, a 3*3 convolutional neural network may be used to convolve the second feature map to obtain the first intermediate layer feature map corresponding to the second feature map.

[0081] S502. Perform position encoding on the first intermediate layer feature map according to a preset first position encoding rule to obtain the first position encoding feature map.

[0082] Exemplarily, a position model based on Transformer may be used to perform position encoding on the first intermediate layer feature map to obtain the first position encoding feature map.

[0083] For example, a model based on Transformer may be used to perform position encoding on each pixel point in the first intermediate feature map. After all pixel points have been position-encoded, the first position encoding feature map corresponding to the first intermediate layer feature map is obtained.

[0084] S503. Fuse the first intermediate layer feature map and the first position encoding feature map to obtain the first fusion feature map.

[0085] Exemplarily, a convolutional neural network model may be used to fuse the first intermediate layer feature map and the first position encoding feature map, and then the first fusion feature map is obtained.

[0086] For example, a convolutional neural network may be used to add the first intermediate layer feature map and the first position encoding feature map pixel by pixel, and then the first fusion feature map is obtained.

[0087] Exemplarily, the preset handwritten formula recognition network can be obtained by training a neural network using the feature map of the handwritten formula image. For example, the feature map of the handwritten formula image can be used as the input, and the text of the handwritten formula included in the handwritten formula image can be used as the output to train the neural network to obtain the handwritten formula recognition network. The implementation of the handwritten formula recognition network is not limited herein. The first fusion feature map can be input into the trained handwritten formula recognition network to perform formula recognition on the first fusion feature map to obtain the handwritten formula recognition result.

[0088] For example, the handwritten formula extracted from the second feature map can be "hello". Convolution is performed on the second feature map to obtain the first intermediate layer feature map corresponding to the second feature map. After obtaining the first intermediate layer feature map, a position model based on Transformer is used to perform position encoding on the first intermediate layer feature map to obtain the first position-encoded feature map. After obtaining the first position-encoded feature map, the first intermediate layer feature map and the first position-encoded feature map are fused using a convolutional neural network to obtain the first fusion feature map. After obtaining the first fusion feature map, the trained handwritten formula recognition network is used to recognize the first fusion feature map to obtain the handwritten formula recognition result, that is, the text of "hello".

[0089] In this embodiment, convolution is performed on the second feature map to obtain the first intermediate layer feature map corresponding to the second feature map. After obtaining the first intermediate layer feature map, a position model is used to perform position encoding on the first intermediate layer feature map to obtain the first position-encoded feature map. After obtaining the first position-encoded feature map, the first intermediate layer feature map and the first position-encoded feature map are fused using a neural network to obtain the first fusion feature map. After obtaining the first fusion feature map, the first fusion feature map is recognized using the preset handwritten formula recognition network to obtain the handwritten formula recognition result. By fusing the position encoding and the intermediate layer features, the features of each pixel point in the first intermediate layer feature map are more abundant after fusing the position information, further improving the formula recognition accuracy and efficiency.

[0090] Figure 6 Another schematic flowchart of S103 provided by the embodiments of the present disclosure Figure 1 is shown in

[0091] As Figure 6 shown, in some embodiments, the above S103 may further include:

[0092] S601. For the first handwritten character in the first fusion feature map, the recognition result of the first handwritten character is output through the handwritten formula recognition network.

[0093] Exemplarily, the first fused feature map contains at least 1 handwritten character. The handwritten formula recognition network can be obtained by training a neural network using the feature map of the handwritten formula image. For example, the feature map of the handwritten formula image can be used as the input, and the text of the handwritten formula contained in the handwritten formula image can be used as the output to train the neural network to obtain the handwritten formula recognition network. The implementation of the handwritten formula recognition network is not limited here. The first fused feature map can be input into the trained handwritten formula recognition network to perform formula recognition on the first fused feature map to obtain the recognition result of the first character in the handwritten formula.

[0094] For example, the handwritten formula character in the first fused feature map can be "hello". The trained handwritten formula recognition network can be used to recognize the first fused feature map and output the recognition result of the first character in the handwritten formula, that is, the character "h".

[0095] S602. For the Nth handwritten character in the first fused feature map, determine the hidden layer state of the (N - 1)th handwritten character through a recurrent neural network. According to the hidden layer state of the (N - 1)th handwritten character, the first intermediate layer feature map, and the first position encoding feature map, determine the attention result corresponding to the Nth handwritten character. Through the handwritten formula recognition network, output the recognition result of the Nth handwritten character according to the attention result corresponding to the Nth handwritten character, where N is an integer greater than 1.

[0096] Exemplarily, the recurrent neural network can be a gated recurrent unit (GRU) recurrent neural network model in the recurrent neural network. The hidden layer state represents the features of each character (except the first character) in the handwritten formula. The attention result represents the local position of the image noticed at each step of decoding. The trained handwritten formula recognition network can be used to recognize the attention result and output the recognition result of each character (except the first character) in the handwritten formula.

[0097] For example, the handwritten formula character in the first fused feature map can be "hello". The trained handwritten formula recognition network can be used to recognize the first fused feature map and output the recognition result of the first character of the handwritten formula, that is, the character "h". After obtaining the first character "h" of the handwritten formula, the feature encoding of the character "h" can be taken out and input into the GRU recurrent neural network to calculate the hidden layer state of the handwritten formula character "e". The feature encoding can be a manually preset feature vector, and each character has a unique feature encoding. After obtaining the hidden layer state of the handwritten formula character "e", the attention result of the handwritten formula character "e" can be calculated by combining the hidden layer state of the handwritten formula character "e", the first intermediate layer feature map, and the first position encoding feature map, and the attention result of the handwritten formula character "e" is input into the trained handwritten formula recognition network to obtain the symbol recognition result of the handwritten formula character "e", that is, output the handwritten formula character "e". The recognition of each character after the handwritten formula character "e" repeats the recognition process of the character "e", and the recognition results of each subsequent handwritten formula symbol are output in sequence, that is, the handwritten formula characters "e", "l", "l", "o" are output in sequence.

[0098] S603. Generate a handwritten formula recognition result according to the recognition result of each handwritten character in the first fused feature map.

[0099] Exemplarily, the final handwritten formula recognition result can be generated according to the handwritten formula characters output by the handwritten formula recognition network each time. The handwritten formula contains at least one character, and there is no corresponding attention result for the first character.

[0100] For example, the handwritten formula character in the first fused feature map can be "hello". The trained handwritten formula recognition network can be used to recognize the first fused feature map and output the recognition result of the first character in the handwritten formula, that is, the character "h". After obtaining the first character "h" of the handwritten formula, the feature encoding of the character "h" can be taken out and input into the GRU recurrent neural network to calculate the hidden layer state of the handwritten formula character "e". After obtaining the hidden layer state of the handwritten formula character "e", the attention result of the handwritten formula character "e" can be calculated by combining the hidden layer state of the handwritten formula character "e", the first intermediate layer feature map, and the first position encoding feature map, and the attention result of the handwritten formula character "e" is input into the trained handwritten formula recognition network to obtain the character recognition result of the handwritten formula character "e", that is, output the handwritten formula character "e". The recognition of each character after the handwritten formula character "e" repeats the recognition process of the character "e", and the recognition results of each subsequent handwritten formula symbol are output in sequence, that is, the handwritten formula characters "e", "l", "l", "o" are output in sequence, that is, the recognition result of the output handwritten formula is "hello".

[0101] In this embodiment, the trained handwritten formula recognition network is used to recognize the first fused feature map, and the recognition result of the first character in the handwritten formula is output. After obtaining the first character of the handwritten formula, the feature encoding of the first character can be taken out and input into the GRU recurrent neural network to calculate the hidden layer state of the second character in the handwritten formula. After obtaining the hidden layer state of the second character in the handwritten formula, the attention result of the second character in the handwritten formula characters is calculated by combining the hidden layer state of the second character in the handwritten formula, the first intermediate layer feature map, and the first position encoding feature map, and the attention result of the second character in the handwritten formula is input into the trained handwritten formula recognition network to obtain the character recognition result of the second character in the handwritten formula. The recognition process of the second character is repeated for the third character and each subsequent character in the handwritten formula characters, and the recognition results of each subsequent handwritten formula symbol are output in turn. The feature vector of the previous character of each character (except the first character) in the handwritten formula is used to calculate the hidden layer state of the current character, and the attention feature of each character (except the first character) is calculated by fusing the hidden layer state of the current character, the first intermediate layer feature map, and the first position encoding feature map, so that the trained handwritten formula recognition network focuses its attention on the local position of each character (except the first character) in the feature map during the recognition process of each character, improving the recognition accuracy and efficiency of each handwritten character and further improving the formula recognition accuracy and efficiency.

[0102] Figure 7 For the Figure 1 schematic diagram of an implementation process of S104 provided by the embodiments of the present disclosure. As Figure 7 shown, in some embodiments, the above S104 may include:

[0103] S701: Convolve the third feature map to obtain a second intermediate layer feature map corresponding to the third feature map.

[0104] Exemplarily, a convolutional neural network model can be used to convolve the third feature map to obtain a second intermediate layer feature map corresponding to the third feature map.

[0105] For example, a 3*3 convolutional neural network can be used to convolve the third feature map to obtain a second intermediate layer feature map corresponding to the third feature map.

[0106] S702: Perform position encoding on the second intermediate layer feature map according to a preset second position encoding rule to obtain a second position encoding feature map.

[0107] Exemplarily, a position model based on Transformer can be used to perform position encoding on the second intermediate layer feature map to obtain the second position encoding feature map.

[0108] Optionally, the second position encoding rule and the first position encoding rule may be the same or different. That is, the second position encoding rule may also use the DeBERTa-style encoding rule. In this disclosure, it is taken as an example that the second position encoding rule is the same as the first position encoding rule.

[0109] For example, a Transformer-based model can be used to perform position encoding on each pixel point in the second intermediate feature map. After position encoding is performed on all pixel points, a second position encoding feature map corresponding to the second intermediate layer feature map is obtained.

[0110] S703. Fuse the second intermediate layer feature map and the second position encoding feature map to obtain a second fused feature map.

[0111] Exemplarily, a convolutional neural network model can be used to fuse the second intermediate layer feature map and the second position encoding feature map, and then a second fused feature map is obtained.

[0112] For example, a convolutional neural network can be used to add each pixel point of the second intermediate layer feature map and the second position encoding feature map, and then a second fused feature map is obtained.

[0113] S704. Use a preset printed formula recognition network to recognize the second fused feature map to obtain a printed formula recognition result.

[0114] Exemplarily, the preset printed formula recognition network can be obtained by training a neural network using the feature map of the printed formula image. For example, the feature map of the printed formula image can be used as the input, and the text of the printed formula included in the printed formula image can be used as the output to train the neural network to obtain the printed formula recognition network. The implementation of the printed formula recognition network is not limited here. The second fused feature map can be input into the trained printed formula recognition network to perform formula recognition on the second fused feature map to obtain a printed formula recognition result.

[0115] For example, the printed formula extracted from the third feature map may be "aple". Convolve the second feature map to obtain the second intermediate layer feature map corresponding to the third feature map. After obtaining the second intermediate layer feature map, use a Transformer-based position model to perform position encoding on the second intermediate layer feature map to obtain a second position encoding feature map. After obtaining the second position encoding feature map, use a convolutional neural network to fuse the second intermediate layer feature map and the second position encoding feature map to obtain a second fused feature map. After obtaining the second fused feature map, use the trained printed formula recognition network to recognize the second fused feature map to obtain a printed formula recognition result, that is, the text of "aple".

[0116] In this embodiment, by performing convolution on the third feature map, a second intermediate layer feature map corresponding to the third feature map is obtained. After obtaining the second intermediate layer feature map, a position model is used to perform position encoding on the second intermediate layer feature map to obtain a second position-encoded feature map. After obtaining the second position-encoded feature map, the second intermediate layer feature map and the second position-encoded feature map are fused using a neural network to obtain a second fused feature map. After obtaining the second fused feature map, a preset printed formula recognition network is used to recognize the second fused feature map to obtain a printed formula recognition result. By fusing the position encoding and the intermediate layer features, the features of each pixel point in the second intermediate layer feature map are richer after fusing the position information, further improving the formula recognition accuracy and efficiency.

[0117] Figure 8 For the Figure 1 Another schematic diagram of the implementation process of S104 provided by the embodiments of the present disclosure.

[0118] As Figure 8 shown, in some embodiments, the above S104 may further include:

[0119] S801. For the first printed character in the second fused feature map, the recognition result of the first printed character is output through the printed formula recognition network.

[0120] Exemplarily, the second fused feature map contains at least one printed character, and the printed formula recognition network can be obtained by training a neural network using the feature map of the printed formula image. For example, the feature map of the printed formula image can be used as the input, and the text of the printed formula included in the printed formula image can be used as the output to train the neural network to obtain the printed formula recognition network. The implementation of the printed formula recognition network is not limited herein. The second fused feature map can be input into the trained printed formula recognition network to perform formula recognition on the second fused feature map to obtain the recognition result of the first character in the printed formula.

[0121] For example, if the printed formula character in the second fused feature map is "aple", the trained printed formula recognition network can be used to recognize the second fused feature map and output the recognition result of the first character in the printed formula, that is, the character "a".

[0122] S802. For the M-th printed character in the second fused feature map, determine the hidden layer state of the (M - 1)-th printed character through a recurrent neural network. According to the hidden layer state of the (M - 1)-th printed character, the second intermediate layer feature map, and the second position encoding feature map, determine the attention result corresponding to the M-th printed character. Through the printed formula recognition network, output the recognition result of the M-th printed character according to the attention result corresponding to the M-th printed character, where M is an integer greater than 1.

[0123] Exemplarily, the recurrent neural network can be a gated recurrent unit (GRU) recurrent neural network model in the recurrent neural network. The hidden layer state represents the features of each character (except the first character) in the printed formula, and the attention result represents the local position of the image noticed at each step of decoding. The trained printed formula recognition network can be used to recognize the attention result and output the recognition result of each character (except the first character) in the printed formula.

[0124] For example, for example, the printed formula character in the second fused feature map can be "aple". The trained printed formula recognition network can be used to recognize the second fused feature map and output the first character "a" of the printed formula. After obtaining the first character "a" of the printed formula, extract the feature encoding of the character "a", which can be input into the GRU recurrent neural network to calculate the hidden layer state of the printed formula character "p". The feature encoding can be a manually preset feature vector, and each character has a unique feature encoding. After obtaining the hidden layer state of the printed formula character "p", combine the hidden layer state of the printed formula character "p", the second intermediate layer feature map, and the second position encoding feature map to calculate the attention result of the printed formula character "p", and input the attention result of the printed formula character "p" into the trained printed formula recognition network to obtain the symbol recognition result of the printed formula character "p", that is, output the printed formula character "p". The recognition process of each character after the printed formula character "p" repeats the recognition process of the character "p", and the recognition results of subsequent each printed formula symbol are output in sequence, that is, the printed formula characters "p", "l", "e" are output in sequence.

[0125] S803. Generate a printed formula recognition result according to the recognition result of each printed character in the second fused feature map.

[0126] Exemplarily, the final printed formula recognition result can be generated according to the printed formula character output by the printed formula recognition network each time. The printed formula contains at least one character, and there is no corresponding attention result for the first character.

[0127] For example, the printed formula character in the second fused feature map can be "aple". The trained printed formula recognition network can be used to recognize the second fused feature map and output the recognition result of the first character in the printed formula, that is, the character "a". After obtaining the first character "a" of the printed formula, the feature encoding of the character "a" can be taken out and input into the GRU recurrent neural network to calculate the hidden layer state of the printed formula character "p". After obtaining the hidden layer state of the printed formula character "p", the attention result of the printed formula character "p" can be calculated by combining the hidden layer state of the printed formula character "p", the second intermediate layer feature map, and the second position encoding feature map, and the attention result of the printed formula character "p" can be input into the trained printed formula recognition network to obtain the character recognition result of the printed formula character "p", that is, output the printed formula character "p". The recognition process of each character after the printed formula character "p" repeats the recognition process of the character "p", and the recognition results of each subsequent printed formula symbol are output in turn, that is, the printed formula characters "p", "l", "e" are output in turn, that is, the recognition result of the output printed formula is "aple".

[0128] In this embodiment, the trained printed formula recognition network is used to recognize the second fused feature map and output the recognition result of the first character in the printed formula. After obtaining the first character of the printed formula, the feature encoding of the first character can be taken out and input into the GRU recurrent neural network to calculate the hidden layer state of the second character in the printed formula. After obtaining the hidden layer state of the second character in the printed formula, the attention result of the second character in the printed formula characters can be calculated by combining the hidden layer state of the second character in the printed formula, the second intermediate layer feature map, and the second position encoding feature map, and the attention result of the second character in the printed formula can be input into the trained printed formula recognition network to obtain the character recognition result of the second character in the printed formula. The recognition process of the third character and each subsequent character in the printed formula characters repeats the recognition process of the second character, and the recognition results of each subsequent printed formula symbol are output in turn. The feature vector of the previous character of each character (except the first character) in the printed formula is used to calculate the hidden layer state of the current character, and the attention feature of each character (except the first character) is calculated by fusing the hidden layer state of the current character, the second intermediate layer feature map, and the second position encoding feature map, so that the trained printed formula recognition network focuses its attention on the local position of each character (except the first character) in the feature map during the recognition process of each character, improving the recognition accuracy and efficiency of each printed character, and further improving the formula recognition accuracy and efficiency.

[0129] In an exemplary embodiment, the present disclosure also provides a formula recognition device, which can be used to implement the formula recognition method as described in the foregoing embodiment.Figure 9 Schematic diagram of the composition of the formula recognition device provided by an embodiment of the present disclosure. As Figure 9 shown, the device may include: an extraction unit 901 and an identification unit 902.

[0130] The extraction unit 901 is configured to extract features from the target image to obtain a first feature map corresponding to the target image; extract a second feature map corresponding to the handwritten formula area and a third feature map corresponding to the printed formula area from the first feature map;

[0131] The identification unit 902 is configured to identify the second feature map by using a preset handwritten formula recognition network to obtain a handwritten formula recognition result; identify the third feature map by using a preset printed formula recognition network to obtain a printed formula recognition result.

[0132] Optionally, when extracting the second feature map corresponding to the handwritten formula area and the third feature map corresponding to the printed formula area from the first feature map, the extraction unit 901 is further configured to:

[0133] Generate a score map and a coordinate map corresponding to the first feature map according to the first feature map, where the score map is used to indicate the classification result of the pixel area of the first feature map, and the coordinate map is used to indicate the position coordinates of the pixel area of the first feature map; extract the second feature map corresponding to the handwritten formula area and the third feature map corresponding to the printed formula area from the first feature map according to the coordinate map and the score map.

[0134] Optionally, when extracting features from the target image to obtain a first feature map corresponding to the target image, the extraction unit 901 is further configured to:

[0135] Extract fourth feature maps with at least two resolutions from the target image; fuse the fourth feature maps with at least two resolutions to obtain the first feature map.

[0136] Optionally, before using the preset handwritten formula recognition network to identify the second feature map to obtain a handwritten formula recognition result, the extraction unit 902 is further configured to:

[0137] Align the second feature map to the target size;

[0138] Optionally, before using the preset printed formula recognition network to identify the third feature map to obtain a printed formula recognition result, the extraction unit 902 is further configured to:

[0139] Align the third feature map to the target size.

[0140] Optionally, when using the preset handwritten formula recognition network to identify the second feature map to obtain a handwritten formula recognition result, the identification unit 902 is further configured to:

[0141] Convolve the second feature map to obtain the first intermediate layer feature map corresponding to the second feature map; perform position encoding on the first intermediate layer feature map according to a preset first position encoding rule to obtain the first position encoding feature map; fuse the first intermediate layer feature map and the first position encoding feature map to obtain the first fused feature map; use a preset handwritten formula recognition network to recognize the first fused feature map to obtain the handwritten formula recognition result.

[0142] Optionally, use a preset handwritten formula recognition network to recognize the first fused feature map to obtain the handwritten formula recognition result. The recognition unit 902 is further configured to:

[0143] Output the recognition result of the first handwritten character in the first fused feature map through the handwritten formula recognition network; for the Nth handwritten character in the first fused feature map, determine the hidden layer state of the N-1th handwritten character through a recurrent neural network, and determine the attention result corresponding to the Nth handwritten character according to the hidden layer state of the N-1th handwritten character, the first intermediate layer feature map, and the first position encoding feature map, and output the recognition result of the Nth handwritten character through the handwritten formula recognition network according to the attention result corresponding to the Nth handwritten character, where N is an integer greater than 1; generate the handwritten formula recognition result according to the recognition results of each handwritten character in the first fused feature map.

[0144] Optionally, use a preset printed formula recognition network to recognize the third feature map to obtain the printed formula recognition result. The recognition unit 902 is further configured to:

[0145] Convolve the third feature map to obtain the second intermediate layer feature map corresponding to the third feature map; perform position encoding on the second intermediate layer feature map according to a preset second position encoding rule to obtain the second position encoding feature map; fuse the second intermediate layer feature map and the second position encoding feature map to obtain the second fused feature map; use a preset printed formula recognition network to recognize the second fused feature map to obtain the printed formula recognition result.

[0146] Optionally, use a preset printed formula recognition network to recognize the second fused feature map to obtain the printed formula recognition result. The recognition unit 902 is further configured to:

[0147] For the first printed character in the second fused feature map, the recognition result of the first printed character is output through the printed formula recognition network; for the Mth printed character in the second fused feature map, the hidden layer state of the (M - 1)th printed character is determined through a recurrent neural network. According to the hidden layer state of the (M - 1)th printed character, the second intermediate layer feature map, and the second position encoding feature map, the attention result corresponding to the Mth printed character is determined. Through the printed formula recognition network, the recognition result of the Mth printed character is output according to the attention result corresponding to the Mth printed character, where M is an integer greater than 1. According to the recognition results of each printed character in the second fused feature map, a printed formula recognition result is generated.

[0148] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0149] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0150] In an exemplary embodiment, the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in the above embodiments.

[0151] In an exemplary embodiment, the readable storage medium may be a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions are used to cause a computer to execute the method as described in the above embodiments.

[0152] In an exemplary embodiment, the computer program product includes a computer program, and the computer program implements the method as described in the above embodiments when executed by a processor.

[0153] Figure 10 FIG. shows a schematic block diagram of an exemplary electronic device 1000 that can be used to implement the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as, for example, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0154] As Figure 10As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to computer programs stored in a read-only memory (ROM) 1002 or computer programs loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0155] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, an optical disc, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0156] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 executes the various methods and processes described above, such as the formula recognition method. For example, in some embodiments, the formula recognition method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the formula recognition method described above can be executed. Alternatively, in other embodiments, the computing unit 1001 can be configured to execute the formula recognition method by any other appropriate means (e.g., by means of firmware).

[0157] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0158] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on the remote machine or server.

[0159] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0160] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0161] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

[0162] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0163] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this is not limited herein.

[0164] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A formula recognition method, the method comprising: Performing feature extraction on a target image to obtain a first feature map corresponding to the target image; Extracting a second feature map corresponding to a handwritten formula region and a third feature map corresponding to a printed formula region from the first feature map; Using a preset handwritten formula recognition network to recognize the second feature map to obtain a handwritten formula recognition result; Using a preset printed formula recognition network to recognize the third feature map to obtain a printed formula recognition result; Wherein, the using a preset handwritten formula recognition network to recognize the second feature map to obtain a handwritten formula recognition result includes: Performing convolution on the second feature map to obtain a first intermediate layer feature map corresponding to the second feature map; Encoding the positions of the first intermediate layer feature map according to a preset first position encoding rule to obtain a first position encoding feature map; Fusing the first intermediate layer feature map and the first position encoding feature map to obtain a first fused feature map; Outputting a recognition result of the first handwritten character through the handwritten formula recognition network for the first handwritten character in the first fused feature map; For the Nth handwritten character in the first fused feature map, determining the hidden layer state of the (N - 1)th handwritten character through a recurrent neural network, and determining the attention result corresponding to the Nth handwritten character according to the hidden layer state of the (N - 1)th handwritten character, the first intermediate layer feature map, and the first position encoding feature map, and outputting a recognition result of the Nth handwritten character through the handwritten formula recognition network according to the attention result corresponding to the Nth handwritten character, where N is an integer greater than 1; Generating the handwritten formula recognition result according to the recognition results of each handwritten character in the first fused feature map.

2. The method according to claim 1, wherein the extracting a second feature map corresponding to a handwritten formula region and a third feature map corresponding to a printed formula region from the first feature map includes: Generating a score map and a coordinate map corresponding to the first feature map according to the first feature map, the score map being used to indicate a classification result of a pixel region of the first feature map, the classification result including a background region, or a handwritten formula region, or a printed formula region, and the coordinate map being used to indicate the position coordinates of a pixel region of the first feature map; Extracting a second feature map corresponding to a handwritten formula region and a third feature map corresponding to a printed formula region from the first feature map according to the coordinate map and the score map.

3. The method according to claim 1 or 2, wherein the performing feature extraction on a target image to obtain a first feature map corresponding to the target image includes: Performing feature extraction on the target image to obtain fourth feature maps of at least two resolutions; Fusing the fourth feature maps of at least two resolutions to obtain the first feature map.

4. The method according to claim 1 or 2, before the using a preset handwritten formula recognition network to recognize the second feature map to obtain a handwritten formula recognition result, the method further includes: Align the second feature map to the target size; Before using the preset printed formula recognition network to recognize the third feature map and obtaining the printed formula recognition result, the method further includes: Align the third feature map to the target size.

5. The method according to claim 1 or 2, wherein using the preset printed formula recognition network to recognize the third feature map and obtaining the printed formula recognition result includes: Performing convolution on the third feature map to obtain a second intermediate layer feature map corresponding to the third feature map; Encoding the position of the second intermediate layer feature map according to a preset second position encoding rule to obtain a second position encoding feature map; Fusing the second intermediate layer feature map and the second position encoding feature map to obtain a second fusion feature map; Using the preset printed formula recognition network to recognize the second fusion feature map to obtain the printed formula recognition result.

6. The method according to claim 5, wherein using the preset printed formula recognition network to recognize the second fusion feature map and obtaining the printed formula recognition result includes: For the first printed character in the second fusion feature map, outputting the recognition result of the first printed character through the printed formula recognition network; For the Mth printed character in the second fusion feature map, determining the hidden layer state of the (M - 1)th printed character through a recurrent neural network, and determining the attention result corresponding to the Mth printed character according to the hidden layer state of the (M - 1)th printed character, the second intermediate layer feature map, and the second position encoding feature map, and outputting the recognition result of the Mth printed character through the printed formula recognition network according to the attention result corresponding to the Mth printed character, where M is an integer greater than 1; Generating the printed formula recognition result according to the recognition results of each printed character in the second fusion feature map.

7. A formula recognition device, the device includes: An extraction unit for extracting features from a target image to obtain a first feature map corresponding to the target image; Extracting a second feature map corresponding to the handwritten formula area and a third feature map corresponding to the printed formula area from the first feature map; A recognition unit for using a preset handwritten formula recognition network to recognize the second feature map to obtain a handwritten formula recognition result; and using a preset printed formula recognition network to recognize the third feature map to obtain a printed formula recognition result; The recognition unit is specifically used for: Performing convolution on the second feature map to obtain a first intermediate layer feature map corresponding to the second feature map; Encoding the position of the first intermediate layer feature map according to a preset first position encoding rule to obtain a first position encoding feature map; Fusing the first intermediate layer feature map and the first position encoding feature map to obtain a first fusion feature map; For the first handwritten character in the first fusion feature map, outputting the recognition result of the first handwritten character through the handwritten formula recognition network; For the Nth handwritten character in the first fused feature map, determine the hidden layer state of the (N - 1)th handwritten character through a recurrent neural network, and determine the attention result corresponding to the Nth handwritten character according to the hidden layer state of the (N - 1)th handwritten character, the first intermediate layer feature map, and the first position encoding feature map. Output the recognition result of the Nth handwritten character through the handwritten formula recognition network according to the attention result corresponding to the Nth handwritten character, where N is an integer greater than 1; Generate the handwritten formula recognition result according to the recognition results of each handwritten character in the first fused feature map.

8. The apparatus according to claim 7, wherein the extraction unit is specifically configured to: Generate a score map and a coordinate map corresponding to the first feature map according to the first feature map. The score map is used to indicate the classification result of the pixel region of the first feature map, and the classification result includes a background region, or a handwritten formula region, or a printed formula region. The coordinate map is used to indicate the position coordinates of the pixel region of the first feature map; Extract a second feature map corresponding to the handwritten formula region and a third feature map corresponding to the printed formula region from the first feature map according to the coordinate map and the score map.

9. The apparatus according to claim 7 or 8, wherein the extraction unit is specifically configured to: Perform feature extraction on the target image to obtain fourth feature maps of at least two resolutions; Fuse the fourth feature maps of the at least two resolutions to obtain the first feature map.

10. The apparatus according to claim 7 or 8, wherein the extraction unit is further configured to: Before using a preset handwritten formula recognition network to recognize the second feature map to obtain a handwritten formula recognition result, align the second feature map to a target size; And, before using a preset printed formula recognition network to recognize the third feature map to obtain a printed formula recognition result, align the third feature map to a target size.

11. The apparatus according to claim 7 or 8, wherein the recognition unit is specifically configured to: Perform convolution on the third feature map to obtain a second intermediate layer feature map corresponding to the third feature map; Perform position encoding on the second intermediate layer feature map according to a preset second position encoding rule to obtain a second position encoding feature map; Fuse the second intermediate layer feature map and the second position encoding feature map to obtain a second fused feature map; Use a preset printed formula recognition network to recognize the second fused feature map to obtain the printed formula recognition result.

12. The apparatus according to claim 11, wherein the recognition unit is specifically configured to: Output the recognition result of the first printed character through the printed formula recognition network for the first printed character in the second fused feature map; For the M-th printed character in the second fusion feature map, determine the hidden layer state of the (M-1)-th printed character through a recurrent neural network. According to the hidden layer state of the (M-1)-th printed character, the second intermediate layer feature map, and the second position encoding feature map, determine the attention result corresponding to the M-th printed character. Through the printed formula recognition network, output the recognition result of the M-th printed character according to the attention result corresponding to the M-th printed character, where M is an integer greater than 1; Generate the printed formula recognition result according to the recognition results of each printed character in the second fused feature map.

13. An electronic device, comprising: At least one processor; And a memory communicatively connected to the at least one processor; Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to any one of claims 1-6.

15. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-6.

Images