Formula recognition method and device, electronic equipment and storage medium
By combining DenseNet and generative adversarial networks, handwritten formulas are converted into printed formulas, solving the problems of low accuracy and low efficiency in recognizing handwritten mathematical formulas and achieving efficient and accurate formula recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2023-04-06
- Publication Date
- 2026-05-15
AI Technical Summary
Existing handwritten mathematical formula recognition methods suffer from low accuracy and low efficiency, especially when faced with mathematical formulas with complex spatial structures and diverse writing styles. Traditional methods have low recognition accuracy, while deep learning methods are either inaccurate in their attention or slow in their speed when decoding complex formulas.
The DenseNet network is used for feature extraction. The handwritten formula is mapped to the printed formula through an image transformation network. A generative adversarial network architecture is used to generate regular printed formula images. The generator and discriminator are combined to train the model, thereby improving the recognition accuracy and efficiency.
By converting complex handwritten formulas into regular printed formulas, the difficulty of recognition is reduced, and the accuracy and efficiency of handwritten formula recognition are significantly improved.
Smart Images
Figure CN116386062B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, specifically to computer vision, deep learning and other technical fields, and in particular to formula recognition methods, devices, electronic devices and storage media. Background Technology
[0002] Mathematical formulas are widely used in education, office work, and library archives. Unlike regular text, recognizing handwritten mathematical formulas presents technical challenges, including inaccurate recognition and low efficiency. Handwritten mathematical formulas exhibit complex spatial structures and diverse writing styles. The complex spatial structure is primarily due to the unique components of mathematical formulas, such as fractions, subscripts, superscripts, and square roots. This diversity in structure and writing style results in less than ideal recognition performance for existing handwritten mathematical formula recognition methods. Summary of the Invention
[0003] This disclosure provides a formula recognition method, apparatus, electronic device, storage medium, and computer program product.
[0004] According to a first aspect of this disclosure, a formula recognition method is provided, comprising:
[0005] The first feature map is obtained by extracting features from the handwritten formula image using the first feature extraction network.
[0006] The first feature map is converted into a second feature map by an image conversion network based on the mapping relationship between the features of the handwritten formula image and the features of the printed formula image, and the printed formula image is generated based on the second feature map.
[0007] A third feature map is obtained by extracting features from the printed formula image using a second feature extraction network.
[0008] The formula recognition result is obtained based on the third feature map.
[0009] According to a second aspect of this disclosure, a formula recognition device is provided, comprising:
[0010] The first feature extraction module is configured to extract features from the handwritten formula image through a first feature extraction network to obtain a first feature map.
[0011] The image conversion module is configured to convert the first feature map into a second feature map based on the mapping relationship between the features of the handwritten formula image and the features of the printed formula image through an image conversion network, and generate a printed formula image based on the second feature map;
[0012] The second feature extraction module is configured to extract features from the printed formula image through a second feature extraction network to obtain a third feature map.
[0013] The recognition module is configured to perform recognition based on the third feature map to obtain the formula recognition result.
[0014] According to a third aspect of this disclosure, an electronic device is provided, comprising:
[0015] At least one processor; and
[0016] A memory communicatively connected to the at least one processor; wherein,
[0017] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described in any of the above technical solutions.
[0018] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any one of the methods described above.
[0019] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in any one of the above technical solutions.
[0020] This disclosure provides a formula recognition method, apparatus, electronic device, storage medium, and computer program product that can simultaneously improve the accuracy and efficiency of formula recognition.
[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0023] Figure 1 This is a schematic diagram of the steps of the formula recognition method in the embodiments of this disclosure;
[0024] Figure 2 This is a model architecture diagram of the formula recognition method applied in the embodiments of this disclosure;
[0025] Figure 3 This is a schematic block diagram of the formula recognition device in the embodiments of this disclosure;
[0026] Figure 4 This is a block diagram of an electronic device used to implement the formula recognition method of the embodiments of this disclosure. Detailed Implementation
[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0028] Currently, handwritten mathematical formula recognition methods can be divided into three categories: The first category is traditional handwritten mathematical formula recognition methods, which generally include three steps: symbol segmentation, symbol recognition, and structural analysis. In the symbol recognition step, Hidden Markov Models, elastic matching algorithms, and support vector machines are frequently used. However, this method is limited by the ability to learn single features and overly complex grammatical rules, resulting in low recognition accuracy and efficiency when dealing with handwritten mathematical formulas in real-world scenarios. The second category is deep learning-based sequence handwritten mathematical formula recognition methods, which treat handwritten mathematical formula recognition as an image-to-sequence translation task, using an attention mechanism to sequentially decode each character in the mathematical formula. The first method involves random movement of attention within the image, which can lead to inaccurate attention when decoding complex mathematical formulas, resulting in either repeated or missed recognition of symbols. The second method is a deep learning-based structured handwritten mathematical formula recognition method. This method treats each mathematical formula as a syntax tree and decodes it using a parent-to-child approach. Then, it converts the tree-like decoding result into a mathematical formula according to predefined rules. This method requires finding the parent node of the current node and combining the features of the parent node to predict the category of the current node when decoding each symbol, which results in a slow decoding speed.
[0029] To address the technical problems of low accuracy and low efficiency in formula recognition in existing technologies, this disclosure provides a formula recognition method, such as... Figure 1 As shown, it includes:
[0030] Step S101: A first feature map is obtained by extracting features from the handwritten formula image using a first feature extraction network. For example, as shown... Figure 2 As shown, the first feature extraction network can be DenseNet (Densely Connected Convolutional Networks). The handwritten formula image 201 is input into the first feature extraction network 202, and the first feature map 203 is obtained by extracting features from the handwritten formula image 201.
[0031] Step S102: Using an image conversion network, the first feature map is converted into a second feature map based on the mapping relationship between the features of the handwritten formula image and the features of the printed formula image, and the printed formula image is generated based on the second feature map. For example... Figure 2 As shown, the image conversion network includes a feature conversion unit 205. Based on the mapping relationship between the features of handwritten formulas and the features of printed formulas, the feature conversion unit 205 maps the first feature map 203 into a second feature map 204, that is, converts it into the features of printed formulas, and then generates a printed formula image 206 based on these features. The feature conversion unit 205 can use a 1x1 convolution kernel.
[0032] Step S103: The third feature map is obtained by extracting features from the printed formula image using the second feature extraction network. For example... Figure 2 As shown, the second feature extraction network 207 can also use the same network structure as the first feature extraction network, or it can use the DenseNet network for feature extraction. The second feature extraction network 207 is used to extract features from the printed formula image 206 to obtain the third feature map 208.
[0033] Step S104: Based on the third feature map 208, the formula recognition result 210 is obtained. After the third feature map corresponding to the printed formula image 206 is extracted based on the second feature extraction network 207, it can be decoded by the decoder 209 to obtain the formula recognition result 210.
[0034] By using the above technical solution, image conversion is used as a prerequisite task for formula recognition. The trained model learns to map complex and varied handwritten formulas to printed formulas, and then the printed formula images are recognized. This greatly reduces the difficulty of recognizing complex handwritten formulas and improves the accuracy and efficiency of handwritten formula recognition.
[0035] As an optional implementation, converting a first feature map corresponding to a handwritten formula into a second feature map corresponding to a printed formula using an image conversion network, and generating a printed formula image based on the second feature map, includes: converting the first feature map into a second feature map using a feature conversion unit of the image conversion network. For example... Figure 2 As shown, the printing formula image 206 is generated by generator 211 based on the second feature map.
[0036] Specifically, the diverse styles and complex structures of handwritten mathematical formulas pose a significant challenge to their recognition. To address this issue, this embodiment utilizes a generative adversarial network (GAN) architecture for formula image conversion. Specifically, given a handwritten mathematical formula image as input, the generator 211 converts it into a printed formula image with fixed fonts and a regular spatial structure. Subsequent recognition models only need to recognize this printed formula image, thus greatly reducing the difficulty of recognizing complex handwritten mathematical formulas.
[0037] As an optional implementation, the first feature extraction network includes a first convolutional layer, a first max-pooling layer, multiple second convolutional layers, and multiple third convolutional layers; the first feature map is obtained by extracting features from the handwritten formula image using the first feature extraction network, including:
[0038] The first convolutional layer is used to process the handwritten formula image. The first convolutional layer can be a convolutional layer with a kernel size of 7 and a stride size of 2. For example, if the handwritten formula image is a three-channel image with a width of W and a height of H, the first convolutional layer will first process it and then output a feature map with a width of W / 2 and a height of H / 2.
[0039] The handwritten formula image is downsampled using a first max pooling layer. This first max pooling layer can be a kernel of 3 with a stride of 2, downsampling the feature map with widths of W / 2 and heights of H / 2.
[0040] The first feature map is obtained by convolutional processing of the handwritten formula image using a first convolutional module consisting of multiple second and third convolutional layers connected in series. The second convolutional layer can have 1 kernel, and the third convolutional layer can have 3 kernels. Further processing of the feature map using a combination of multiple convolutional layers with 1 and 3 kernels yields the final output first feature map, with a width of W / 16 and a height of H / 16, and 684 channels.
[0041] As an optional implementation, the second feature extraction network includes a fourth convolutional layer, a second max pooling layer, multiple fifth convolutional layers, and multiple sixth convolutional layers. Extracting features from the printing formula image using the second feature extraction network to obtain a third feature map includes: performing convolution processing on the printing formula image using the fourth convolutional layer; downsampling the printing formula image using the second max pooling layer; and performing convolution processing on the printing formula image using a second convolutional module consisting of multiple fifth and multiple sixth convolutional layers connected in series to obtain the third feature map.
[0042] Specifically, the fourth convolutional layer can also be a convolutional layer with a kernel of 7 and a stride of 2; the second max pooling layer can be a max pooling layer with a kernel of 3 and a stride of 2; the fifth convolutional layer can have a kernel of 1, and the sixth convolutional layer can have a kernel of 3. The second feature extraction network can use the same network structure as the first feature extraction network, and its feature extraction process is the same as that of the first feature extraction network, except that the object of feature extraction is changed to the printed formula image, which will not be described in detail below.
[0043] As an optional implementation, the formula recognition result obtained based on the third printing formula feature map includes:
[0044] The first intermediate layer features are obtained by extracting features from the third feature map through the seventh convolutional layer. The seventh convolutional layer can be a 1x1 convolutional kernel. The number of channels in the third feature map can be changed by the 1x1 convolutional kernel, while the width and height remain unchanged.
[0045] The first intermediate layer features are positionally encoded, and the positional encoding is added to the first intermediate layer features to obtain the second intermediate layer features. After obtaining the positional encoding, it is added to the first intermediate layer features. This operation can enhance the model's ability to distinguish different locations in the image.
[0046] The second intermediate layer features are decoded sequentially to obtain multiple symbols corresponding to the handwritten formula image, which are then used as the formula recognition result.
[0047] As an optional implementation, the process of sequentially decoding the features of the second intermediate layer to obtain multiple symbols corresponding to the handwritten formula image as the formula recognition result includes: extracting the feature code corresponding to each symbol obtained by decoding at each step; inputting the feature code into a gated recurrent neural network (GRU) to obtain the current hidden layer state through the gated recurrent neural network; and decoding the next symbol in combination with the current hidden layer state.
[0048] In this embodiment, to enhance the model's attention to the symbols decoded in the previous step and thus improve the accurate localization of the current symbol, at each decoding step, the feature encoding of the previously decoded symbol is input into the GRU network to obtain the current hidden layer state. Combining the intermediate layer features, positional encoding, and the current hidden layer state, the model's attention result can be calculated. The model's attention result represents the local image location noticed at each decoding step. Extracting this local feature and inputting it into the fully connected layer yields the symbol predicted by the current decoding step. In short, during the decoding of each symbol, the model's current hidden layer state is obtained simultaneously using the previously decoded symbol. This current hidden layer state is also used as input information when decoding the next symbol, allowing the decoder to predict the next symbol more accurately by combining the current hidden layer state.
[0049] As an optional implementation, after generating the printing formula image based on the second feature map, the method further includes: Figure 2 As shown, the discriminator 212 judges the conversion effect of the printed formula image and updates the image conversion network according to the conversion effect.
[0050] In this embodiment, a combination of generator 211 and discriminator 212 is used to form the generative model. In practice, it was found that using only generator 211 for image conversion is not very effective. This is because to correctly convert formula images, the model needs to understand the content of the image to a certain extent. Simple image conversion is only a low-semantic task at the image level and cannot enable the model to have a high-semantic-level understanding ability. Therefore, this embodiment combines formula recognition and image generation tasks simultaneously. This has two advantages: 1) If the printed formula image generated by the generator is of poor quality, the decoding process of formula recognition will have difficulty predicting the correct result, which in turn can encourage the generator to generate formula images that are as realistic as possible. 2) By sharing the first feature extraction network and the feature conversion unit, the image generation and formula recognition tasks can jointly promote feature learning, obtaining more robust features, thereby improving the performance of both tasks.
[0051] This disclosure also provides a formula recognition device 300, such as Figure 3 As shown, it includes:
[0052] The first feature extraction module 301 is configured to extract features from a handwritten formula image using a first feature extraction network to obtain a first feature map. For example, as shown... Figure 2As shown, the first feature extraction network can be DenseNet. The handwritten formula image 201 is input into the first feature extraction network 202, and the first feature map 203 is obtained by extracting features from the handwritten formula image 201.
[0053] Image conversion module 302 is configured to convert a first feature map into a second feature map using an image conversion network based on the mapping relationship between handwritten formula image features and printed formula image features, and to generate a printed formula image based on the second feature map. For example... Figure 2 As shown, the feature conversion unit 205 can map the first feature map 203 into the second feature map 204 based on the mapping relationship between the features of the handwritten formula and the features of the printed formula, that is, convert it into the features of the printed formula, and then generate the printed formula image 206 based on the features.
[0054] The second feature extraction module 303 is configured to extract features from the printed formula image using a second feature extraction network to obtain a third feature map. For example... Figure 2 As shown, the second feature extraction network 207 can also use the same network structure as the first feature extraction network, or it can use the DenseNet network for feature extraction. The second feature extraction network 207 is used to extract features from the printed formula image 206 to obtain the third feature map 208.
[0055] The recognition module 304 is configured to obtain the formula recognition result based on the third feature map. After the third feature map corresponding to the printed formula image 206 is extracted based on the second feature extraction network 207, it can be decoded by the decoder 209 to obtain the formula recognition result 210.
[0056] Through the above technical solution, the formula recognition device uses image conversion as a prerequisite for formula recognition. By training a model to learn how to map complex and varied handwritten formulas to printed formulas, and then recognizing the printed formula images, the difficulty of recognizing complex handwritten formulas is greatly reduced, and the accuracy and efficiency of handwritten formula recognition are improved.
[0057] As an optional implementation, the image conversion module includes: a feature conversion unit configured to convert a first feature map into a second feature map; and a generator configured to generate a printing formula image based on the second feature map. Figure 2 As shown, the printing formula image 206 is generated by generator 211 based on the second feature map.
[0058] Specifically, the diverse styles and complex structures of handwritten mathematical formulas pose a significant challenge to their recognition. To address this issue, this embodiment utilizes a generative adversarial network (GAN) architecture for formula image conversion. Specifically, given a handwritten mathematical formula image as input, the generator 211 converts it into a printed formula image with fixed fonts and a regular spatial structure. Subsequent recognition models only need to recognize this printed formula image, thus greatly reducing the difficulty of recognizing complex handwritten mathematical formulas.
[0059] As an optional implementation, the first feature extraction network includes: a first convolutional layer configured to perform convolution processing on the handwritten formula image; a first max pooling layer configured to downsample the handwritten formula image; and a first convolutional module comprising multiple second convolutional layers and multiple third convolutional layers connected in series, configured to perform convolution processing on the handwritten formula image to obtain a first feature map. The first convolutional layer can be a convolutional layer with 7 kernels and a stride of 2. For example, if the handwritten formula image is a three-channel image with width W and height H, the first convolutional layer first processes the image to output a feature map with width and height of W / 2 and H / 2, respectively. Then, a max pooling layer with 3 kernels and a stride of 2 downsamples the feature map with width and height of W / 2 and H / 2, respectively. Finally, a combination of multiple convolutional layers with 1 kernel and 3 kernels further processes the feature map to obtain the final output first feature map, with width and height of W / 16 and H / 16, and 684 channels.
[0060] As an optional implementation, the second feature extraction network includes: a fourth convolutional layer configured to perform convolution processing on the printed formula image; a second max pooling layer configured to downsample the printed formula image; and a second convolutional module comprising multiple fifth convolutional layers and multiple sixth convolutional layers connected in series, configured to perform convolution processing on the printed formula image to obtain a third feature map. The fourth convolutional layer can also be a convolutional layer with 7 kernels and a stride of 2; the second max pooling layer can be a max pooling layer with 3 kernels and a stride of 2; the fifth convolutional layer can have a kernel of 1, and the sixth convolutional layer can have a kernel of 3. The second feature extraction network can use the same network structure as the first feature extraction network, and its feature extraction process is the same as the first feature extraction network, only the object of feature extraction is changed to the printed formula image, which will not be described further below.
[0061] As an optional implementation, the identification module includes:
[0062] The seventh convolutional layer is configured to extract features from the third feature map to obtain the first intermediate layer features. The seventh convolutional layer can have a 1x1 convolutional kernel, which changes the number of channels in the third feature map while keeping the width and height unchanged.
[0063] The location encoding unit is configured to perform location encoding on the first intermediate layer features.
[0064] The position addition unit is configured to add the positional encoding to the first intermediate layer features to obtain the second intermediate layer features. After obtaining the positional encoding, it is added to the first intermediate layer features. This operation can enhance the model's ability to distinguish different locations in the image.
[0065] The decoding unit is configured to decode the features of the second intermediate layer in sequence to obtain multiple symbols corresponding to the handwritten formula image as the formula recognition result.
[0066] As an optional implementation, the process of sequentially decoding the features of the second intermediate layer to obtain multiple symbols corresponding to the handwritten formula image as the formula recognition result includes: extracting the feature code corresponding to each symbol obtained by decoding at each step; inputting the feature code into the GRU network to obtain the current hidden layer state through the GRU network; and decoding the next symbol in combination with the current hidden layer state.
[0067] In this embodiment, to enhance the model's attention to the symbols decoded in the previous step and thus improve the accurate localization of the current symbol, at each decoding step, the feature encoding of the previously decoded symbol is input into the GRU network to obtain the current hidden layer state. Combining the intermediate layer features, positional encoding, and the current hidden layer state, the model's attention result can be calculated. The model's attention result represents the local image location noticed at each decoding step. Extracting this local feature and inputting it into the fully connected layer yields the symbol predicted by the current decoding step. In short, during the decoding of each symbol, the model's current hidden layer state is obtained simultaneously using the previously decoded symbol. This current hidden layer state is also used as input information when decoding the next symbol, allowing the decoder to predict the next symbol more accurately by combining the current hidden layer state.
[0068] As an optional implementation, the formula recognition device further includes a discriminator configured to judge the conversion effect of the printed formula image and update the image conversion network based on the conversion effect. In this embodiment, a combination of generator 211 and discriminator 212 is used to form the generative model. In practice, it has been found that if only generator 211 is used for image conversion, the effect is not ideal. This is because if the formula image needs to be correctly converted, the model needs to understand the content of the image to a certain extent. Simple image conversion is only a low-semantic task at the image level and cannot enable the model to have a high-semantic level understanding ability. Therefore, in this embodiment, the two tasks of formula recognition and image generation are combined and performed simultaneously. This has two advantages: 1) If the printed formula image generated by the generator is of poor quality, the decoding process of formula recognition will have difficulty predicting the correct result, which can in turn promote the generator to generate the most realistic formula image possible. 2) By sharing the first feature extraction network and the feature conversion unit, the two tasks of image generation and formula recognition can jointly promote feature learning, obtain more robust features, and thus improve the performance of the two tasks of image generation and formula recognition.
[0069] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0070] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0071] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0072] like Figure 4As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.
[0073] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0074] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning objective function algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the formula recognition method. For example, in some embodiments, the formula recognition method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the formula recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the formula recognition method by any other suitable means (e.g., by means of firmware).
[0075] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0076] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0077] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0078] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0079] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0080] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0081] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0082] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A formula recognition method, comprising: The first feature map is obtained by extracting features from the handwritten formula image using the first feature extraction network. The first feature map is converted into a second feature map by an image conversion network based on the mapping relationship between the features of the handwritten formula image and the features of the printed formula image, and the printed formula image is generated based on the second feature map. The first feature map is converted into the second feature map by the feature conversion unit of the image conversion network; the generator generates the printing formula image with fixed font and regular spatial structure based on the second feature map; The discriminator determines the conversion effect of the printed formula image and updates the image conversion network based on the conversion effect; wherein the generator and the discriminator constitute an adversarial network and share the first feature extraction network and the feature conversion unit; A third feature map is obtained by extracting features from the printed formula image using a second feature extraction network. The formula recognition result is obtained based on the third feature map.
2. The method according to claim 1, wherein, The first feature extraction network includes a first convolutional layer, a first max pooling layer, multiple second convolutional layers, and multiple third convolutional layers; the step of extracting features from the handwritten formula image using the first feature extraction network to obtain the first feature map includes: The handwritten formula image is processed by convolution using the first convolutional layer. The handwritten formula image is downsampled using the max pooling layer. The first feature map is obtained by performing convolution processing on the handwritten formula image using a first convolution module consisting of multiple second convolutional layers and multiple third convolutional layers connected in series.
3. The method according to any one of claims 1 or 2, wherein, The second feature extraction network includes a fourth convolutional layer, a second max pooling layer, multiple fifth convolutional layers, and multiple sixth convolutional layers; The step of extracting features from the printing formula image using a second feature extraction network to obtain a third feature map includes: The fourth convolutional layer is used to perform convolution processing on the printing formula image; The printing formula image is downsampled using the second max-pooling layer; The third feature map is obtained by performing convolution processing on the printed formula image using a second convolution module consisting of multiple fifth convolutional layers and multiple sixth convolutional layers connected in series.
4. The method according to claim 1, wherein, The formula recognition result obtained based on the third feature map includes: The first intermediate layer features are obtained by extracting features from the third feature map through the seventh convolutional layer; Position encoding is performed on the features of the first intermediate layer; The positional encoding is added to the first intermediate layer feature to obtain the second intermediate layer feature; The second intermediate layer feature decoding sequentially obtains multiple symbols corresponding to the handwritten formula image as the formula recognition result.
5. The method according to claim 4, wherein, The step of sequentially obtaining multiple symbols corresponding to the handwritten formula image from the second intermediate layer feature decoding as the formula recognition result includes: For each symbol obtained from decoding, extract the feature code corresponding to the symbol; The feature encoding is input into a gated recurrent neural network, and the current hidden layer state is obtained through the gated recurrent neural network; The next symbol is obtained by decoding the current hidden layer state.
6. A formula recognition device, comprising: The first feature extraction module is configured to extract features from the handwritten formula image through a first feature extraction network to obtain a first feature map. The image conversion module is configured to convert the first feature map into a second feature map based on the mapping relationship between the features of the handwritten formula image and the features of the printed formula image through an image conversion network, and generate a printed formula image based on the second feature map; The image conversion module includes: a feature conversion unit configured to convert the first feature map into a second feature map; and a generator configured to generate the printing formula image based on the second feature map. A discriminator is configured to determine the conversion effect of the printed formula image and update the image conversion network based on the conversion effect; wherein the generator and the discriminator constitute an adversarial network and share the first feature extraction network and the feature conversion unit; The second feature extraction module is configured to extract features from the printed formula image through a second feature extraction network to obtain a third feature map. The recognition module is configured to perform recognition based on the third feature map to obtain the formula recognition result.
7. The apparatus according to claim 6, wherein the first feature extraction network comprises: The first convolutional layer is configured to perform convolution processing on the handwritten formula image; The first max pooling layer is configured to downsample the handwritten formula image; The first convolutional module, comprising multiple second convolutional layers and multiple third convolutional layers connected in series, is configured to perform convolution processing on the handwritten formula image to obtain the first feature map.
8. The apparatus according to any one of claims 6 or 7, wherein the second feature extraction network comprises: The fourth convolutional layer is configured to perform convolution processing on the printed formula image; The second max-pooling layer is configured to downsample the printed formula image; The second convolutional module, consisting of multiple fifth convolutional layers and multiple sixth convolutional layers connected in series, is configured to perform convolution processing on the printed formula image to obtain the third feature map.
9. The apparatus according to claim 6, wherein the identification module comprises: The seventh convolutional layer is configured to extract features from the third feature map to obtain the first intermediate layer features; A position encoding unit is configured to perform position encoding on the first intermediate layer features; The position addition unit is configured to add the position code to the first intermediate layer feature to obtain the second intermediate layer feature; The decoding unit is configured to decode the second intermediate layer features sequentially to obtain multiple symbols corresponding to the handwritten formula image as the formula recognition result.
10. The apparatus according to claim 9, wherein the decoding unit sequentially decodes the second intermediate layer features to obtain a plurality of symbols corresponding to the handwritten formula image as the formula recognition result, comprising: For each symbol obtained from decoding, extract the feature code corresponding to the symbol; The feature encoding is input into a gated recurrent neural network, and the current hidden layer state is obtained through the gated recurrent neural network; The next symbol is obtained by decoding the current hidden layer state.
11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.
13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.