Mathematical formula recognition method and electronic device

Abstract

The embodiment of the present application discloses a kind of mathematical formula identification method and electronic equipment.The method comprises: by listening to clipboard, the image captured in the clipboard is obtained, and the mathematical formula content and the context content of the mathematical formula content are extracted from the image, which is to be carried out mathematical formula identification;The mathematical formula content is identified, and the mathematical formula is obtained;From the context content, the key content related to the mathematical formula is extracted, and the key content is identified, and the semantic information of the mathematical formula is obtained;The mathematical formula and the semantic information are used as the mathematical formula identification result of the image.The technical scheme of the embodiment of the present application solves the problem that the readability of mathematical formula identification result is not strong.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image recognition, and particularly relate to a mathematical formula recognition method and an electronic device. BACKGROUND

[0002] Related technologies for recognizing mathematical formulas from images are involved in web development, scientific paper writing, teacher board writing recognition, and student photographing problem solving, etc.

[0003] However, the readability of the current mathematical formula recognition result is not strong, and needs to be solved urgently. SUMMARY

[0004] Embodiments of the present application provide a mathematical formula recognition method and an electronic device, which solve the problem of poor readability of mathematical formula recognition results.

[0005] According to an aspect of the present application, a mathematical formula recognition method can include:

[0006] By listening to the clipboard, an image captured in the clipboard for mathematical formula recognition is obtained, and mathematical formula content and context content of the mathematical formula content are extracted from the image;

[0007] The mathematical formula content is recognized to obtain a mathematical formula;

[0008] Key content related to the mathematical formula is extracted from the context content, and the key content is recognized to obtain semantic information of the mathematical formula;

[0009] The mathematical formula and the semantic information are taken as a mathematical formula recognition result of the image.

[0010] According to another aspect of the present application, an electronic device can include:

[0011] At least one processor; and

[0012] A memory in communication connection with the at least one processor; wherein,

[0013] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to implement the mathematical formula recognition method provided by any embodiment of the present application when executed.

[0014] The technical scheme of the embodiment of the present application can obtain the mathematical formula content and the context content of the mathematical formula content from the image by listening to the clipboard, capturing the image to be recognized in the mathematical formula in the clipboard, and extracting the mathematical formula content and the context content of the mathematical formula content from the image. The mathematical formula content used to obtain the mathematical formula can be obtained, and the context content can be obtained. The mathematical formula content is recognized again to obtain the mathematical formula, the recognition of the mathematical formula itself is realized, the key content related to the mathematical formula is extracted from the context content, and the key content is recognized to obtain the semantic information of the mathematical formula which is helpful to improve the readability of the mathematical formula recognition result. Finally, the mathematical formula and the semantic information are taken as the mathematical formula recognition result of the image, and the recognition of the mathematical formula is realized. The above technical scheme takes the mathematical formula and the semantic information which is helpful to improve the readability of the mathematical formula recognition result as the mathematical formula recognition result, and can improve the readability of the mathematical formula recognition result, thereby solving the problem of low readability of the mathematical formula recognition result.

[0015] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is a flowchart of a mathematical formula recognition method according to an embodiment of the present application;

[0018] Figure 2 is a flowchart of another mathematical formula recognition method according to an embodiment of the present application;

[0019] Figure 3 is a flowchart of another mathematical formula recognition method according to an embodiment of the present application;

[0020] Figure 4 is a flowchart of an optional example of another mathematical formula recognition method according to an embodiment of the present application;

[0021] Figure 5a is a schematic diagram of a clipboard in another optional example of another mathematical formula recognition method according to an embodiment of the present application;

[0022] Figure 5bis a schematic diagram of another cutting board in another optional example of the mathematical formula recognition method according to the embodiment of the present application;

[0023] Figure 5c is a schematic diagram of another cutting board in another optional example of the mathematical formula recognition method according to the embodiment of the present application;

[0024] Figure 6 is a structural block diagram of a mathematical formula recognition device according to the embodiment of the present application;

[0025] Figure 7 is a structural schematic diagram of an electronic device implementing the mathematical formula recognition method according to the embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts should fall within the protection scope of the present application.

[0027] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. The case of "target", "original" and the like is similar, and will not be described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Figure 1 is a flowchart of a mathematical formula recognition method according to an embodiment of the present application. The embodiment can be applicable to the case of mathematical formula recognition. The method can be performed by a mathematical formula recognition device provided by the embodiment of the present application, which can be realized by software and / or hardware, and can be integrated on an electronic device, which can be various user terminals or servers.

[0029] Referring to Figure 1 , the method of the embodiment of the present application specifically includes the following steps:

[0030] S110, acquire the image captured in the clipboard for mathematical formula recognition, and extract the mathematical formula content and the context content of the mathematical formula content from the image by monitoring the clipboard.

[0031] The clipboard can be understood as a clipboard capable of cutting into an image.

[0032] The image can be understood as an image to be recognized for mathematical formula.

[0033] In the embodiment of the application, the image can be acquired by monitoring the clipboard. For example, whether the content in the clipboard is a picture can be monitored. If it is a picture, the image is acquired to avoid mathematical formula recognition errors caused by acquiring non-image.

[0034] In the embodiment of the application, the image can be copied to the clipboard, and dragging the image to the clipboard can also be supported.

[0035] The mathematical formula content can be understood as content including a mathematical formula.

[0036] The context content can be understood as the content of the context of the mathematical formula content. For example, it can include the previous paragraph and the next paragraph of the mathematical formula content. For another example, it can be the content of a preset number of words, a preset number of paragraphs or a preset number of lines before the mathematical formula content, and the like.

[0037] It can be understood that in scientific research documents such as papers, mathematical formulas are often related to the surrounding text (such as variable definition). Therefore, in order to improve the readability of the mathematical formula recognition result, in the embodiment of the application, the mathematical formula content and the context content can be extracted from the image. For example, the regions corresponding to the mathematical formula content and the context content in the image can be segmented, and the mathematical formula content and the context content can be extracted from the segmented regions, so as to perform mathematical formula recognition based on the mathematical formula content and the context content, thereby improving the readability of the mathematical formula recognition result.

[0038] In the embodiment of the application, before the mathematical formula content and the context content of the mathematical formula content are extracted from the image, at least one of the pre-processing such as enhancement and labeling of the image can be performed. For example, the mathematical formula content part, the context content part and other content parts in the image can be labeled.

[0039] S120, recognize the mathematical formula content to obtain a mathematical formula.

[0040] The mathematical formula can be understood as a formula in the field of mathematics recognized from the mathematical formula content. The number of mathematical formulas can be at least one.

[0041] In the embodiments of the present application, the mathematical formula content can be recognized to obtain the mathematical formula. For example, the mathematical formula content can be recognized to obtain the mathematical formula by at least one of an online Optical Character Recognition (OCR) formula recognition service such as SimpleTex, a client-based formula recognition tool such as Mathpix, and a cloud service-based formula recognition Application Programming Interface (API) interface. Considering the above scheme of recognizing the mathematical formula in an online manner, there may be problems such as network delay, server congestion risk, the need for continuous network connection support, the inability to use offline, and the need to upload privacy-sensitive data to a third-party server. For another example, the mathematical formula content can be recognized to obtain the mathematical formula by a localized OCR model.

[0042] In S130, key content related to the mathematical formula is extracted from the context content, and the key content is recognized to obtain semantic information of the mathematical formula.

[0043] The key content can be understood as content related to the mathematical formula. For example, the key content can include at least one of keywords and / or key sentences.

[0044] The semantic information can be understood as information related to the semantics of the mathematical formula. For example, the semantic information can be at least one of a semantic annotation of the mathematical formula (such as at least one of an annotation of a parameter and an annotation of an action of the mathematical formula), a type of the mathematical formula, and an application field of the mathematical formula.

[0045] In the embodiments of the present application, the key content can be extracted from the context content. For example, the key content including a parameter in the mathematical formula and / or key content describing the mathematical formula can be extracted from the context content.

[0046] In the embodiments of the present application, the key content can be recognized to obtain the semantic information. For example, content in the key content corresponding to a recognition dimension can be recognized as the semantic information according to at least one of a preset recognition dimension (such as at least one of a parameter annotation dimension, a formula action dimension, a type of the mathematical formula dimension, and an application field of the mathematical formula dimension).

[0047] In S140, the mathematical formula and the semantic information are taken as a mathematical formula recognition result of the image.

[0048] The mathematical formula recognition result can be understood as a recognition result composed of the mathematical formula and the semantic information.

[0049] In the embodiments of the present application, the mathematical formula and the semantic information can be taken as the mathematical formula recognition result.

[0050] In the embodiment of the present application, after the mathematical formula and the semantic information are obtained as the mathematical formula recognition result of the image, the mathematical formula recognition result can be displayed.

[0051] The technical scheme of the embodiment of the present application can obtain the mathematical formula content and the context content of the mathematical formula content from the image captured in the clipboard, and can not only obtain the mathematical formula content used to obtain the mathematical formula, but also obtain the context content. Then, the mathematical formula content is recognized to obtain the mathematical formula, the recognition of the mathematical formula itself is realized, the key content related to the mathematical formula is extracted from the context content, and the key content is recognized to obtain the semantic information of the mathematical formula which is helpful to improve the readability of the mathematical formula recognition result. Finally, the mathematical formula and the semantic information are obtained as the mathematical formula recognition result of the image, and the recognition of the mathematical formula is realized. The above technical scheme can improve the readability of the mathematical formula recognition result by taking the mathematical formula and the semantic information which is helpful to improve the readability of the mathematical formula recognition result as the mathematical formula recognition result, thereby solving the problem of low readability of the mathematical formula recognition result.

[0052] An optional technical scheme can obtain the image captured in the clipboard by listening to the clipboard, including: listening to the clipboard and determining whether the image to be recognized by the mathematical formula is captured in the clipboard; if captured, obtaining the image; the method further includes: if not captured, displaying prompt information on the preset interactive interface, wherein the prompt information represents that the image is not captured in the clipboard.

[0053] The prompt information can be understood as information representing that the image is not captured in the clipboard.

[0054] In the embodiment of the present application, the clipboard can be listened to, and it is determined whether the image is captured in the clipboard. For example, the system clipboard can be listened to in real time or less than a preset refresh period (for example, less than 3 seconds) by an image grabber (Image Grabber, ImageGrab), and it is determined whether the image is captured in the clipboard, that is, the real-time image capture based on the system clipboard by the ImageGrab is realized.

[0055] In the embodiment of the present application, if captured, the image is obtained.

[0056] In the embodiment of the present application, if not captured, the prompt information is displayed on the preset interactive interface. For example, the prompt information can be displayed on the preset interactive interface by a Tkinter message box (Tkinter Message Box, messagebox).

[0057] In the embodiment of the present application, the clipboard can be monitored, and it is determined whether an image is captured in the clipboard. If the image is captured, the image is acquired. If the image is not captured, prompt information is displayed on the interactive interface. Through the linkage of the monitoring of the clipboard and the image capture, the above-mentioned scheme can provide the image capture function of the clipboard, and the prompt information can be visualized on the interactive interface in the case where the image is not captured, so that the user can clearly understand whether the image is captured.

[0058] On the basis of the above-mentioned scheme, another optional technical scheme, after the image is acquired, the mathematical formula recognition method further includes: if the size of the image exceeds a preset size, the image is reduced based on the preset size to obtain a thumbnail image, and the thumbnail image is displayed on the interactive interface; otherwise, the image is displayed on the interactive interface.

[0059] The preset size can be understood as a preset minimum size of the image without being reduced. For example, the preset size can be 300x300.

[0060] The thumbnail image can be understood as an image obtained by reducing the image based on the preset size.

[0061] In the embodiment of the present application, if the size of the image exceeds the preset size, the image is reduced based on the preset size to obtain a thumbnail image. For example, the image can be reduced based on the preset size to obtain the thumbnail image by using an image thumbnail method (image.thumbnail).

[0062] In the embodiment of the present application, if the size of the image does not exceed the preset size, the image is displayed on the interactive interface.

[0063] In the embodiment of the present application, if the size of the image exceeds the preset size, the image is reduced based on the preset size to obtain a thumbnail image, and the thumbnail image is displayed on the interactive interface. Otherwise, the image is displayed on the interactive interface. The above-mentioned technical scheme can avoid the problem that the complete image cannot be displayed due to the excessively large size of the image, so that the display optimization of the large-size image can be realized.

[0064] Another optional technical scheme, after the image captured in the clipboard and to be subjected to the mathematical formula recognition is acquired, the mathematical formula recognition method further includes: inputting the image into a degradation classifier, obtaining the degradation type of the image according to the output result of the degradation classifier; performing enhancement processing on the image by calling a generative adversarial network corresponding to the degradation type, and taking the enhanced image as the image.

[0065] The degradation classifier can be understood as a classifier for determining the degradation type of the image.

[0066] The degradation type can be understood as a type of image degradation, and the degradation type can also be understood as an abnormal type of image that affects the accuracy of mathematical formula recognition. The degradation type may, for example, include at least one of blur, low light, and compression artifacts.

[0067] In the embodiment of the present application, the image can be input into the degradation classifier, and the degradation type can be obtained according to the output result.

[0068] The generative adversarial network can be understood as a network that can perform enhancement processing on the image.

[0069] In the embodiment of the present application, the image can be enhanced by calling the generative adversarial network corresponding to the degradation type, and the enhanced image can be used as the image. For example, a generative adversarial network corresponding to each of the alternative types that can be used as the degradation type can be set in advance, and the image can be enhanced by calling the generative adversarial network corresponding to the degradation type from the generative adversarial networks corresponding to the alternative types, and the enhanced image can be used as the image.

[0070] In the embodiment of the present application, the image is input into the degradation classifier, the degradation type is obtained according to the output result, and the image is enhanced by calling the generative adversarial network, and the enhanced image is used as the image. The above scheme can improve the recognition rate of mathematical formulas of low-quality images.

[0071] Figure 2 is a flowchart of another mathematical formula recognition method provided in the embodiment of the present application. The present embodiment is optimized on the basis of the above technical solutions. In the present embodiment, the method can be applied to a terminal device to recognize mathematical formula content and obtain a mathematical formula, and the method includes: obtaining an optical character recognition model deployed on the terminal device; inputting the mathematical formula content into the optical character recognition model, and obtaining the mathematical formula according to an output result of the optical character recognition model.

[0072] Wherein, the same or corresponding terms as in the above embodiments are not repeated here.

[0073] Referring to Figure 2 The method of the present embodiment can be applied to a terminal device, and the method can specifically include the following steps:

[0074] S210, by listening to the clipboard, an image captured in the clipboard and to be subjected to mathematical formula recognition is obtained, and mathematical formula content and context content of the mathematical formula content are extracted from the image.

[0075] The terminal device can be understood as a local device for mathematical formula recognition.

[0076] S220, an optical character recognition model deployed on the terminal device is obtained.

[0077] The optical character recognition model can be understood as a model for recognizing mathematical formulas in an optical character recognition manner.

[0078] It can be understood that the optical character recognition model deployed on the terminal device can also be understood as a locally deployed optical character recognition model.

[0079] In the embodiments of the present application, the optical character recognition model can be preloaded, and in order to improve the model loading speed, the optical character recognition model can be loaded asynchronously through a thread or the like.

[0080] In the embodiments of the present application, the samples used to train the optical character recognition model can include not only printed samples, but also samples in different forms such as handwriting, so as to improve the application range of the optical character recognition model.

[0081] S230, input the mathematical formula content into the optical character recognition model, and obtain the mathematical formula according to the output result of the optical character recognition model.

[0082] In the embodiments of the present application, the mathematical formula content can be input into the optical character recognition model, and the mathematical formula can be obtained according to the output result. For example, the mathematical formula content can be input into the optical character recognition model to obtain an output result in the form of a compilable LaTeX code (for example, \frac{a}{b}), and the mathematical formula can be obtained according to the output result.

[0083] S240, extract the key content related to the mathematical formula from the context content, and recognize the key content to obtain the semantic information of the mathematical formula.

[0084] S250, the mathematical formula and the semantic information are taken as the mathematical formula recognition result of the image.

[0085] The technical scheme of the embodiments of the present application is applied to a terminal device, obtains an optical character recognition model deployed on the terminal device, then inputs mathematical formula content into the optical character recognition model, and obtains a mathematical formula according to the output result of the optical character recognition model. The above scheme can recognize a mathematical formula through an optical character recognition model deployed on a terminal device, which can avoid network delay, queuing and uploading of sensitive formula data to a cloud server, thereby improving the efficiency and security of mathematical formula recognition.

[0086] An optional technical solution, after obtaining the mathematical formula, the mathematical formula recognition method further comprises: displaying the mathematical formula on a preset interactive interface; in response to a correction instruction for the displayed mathematical formula, obtaining correction content, and taking the mathematical formula, the correction content, and a correction result of correcting the mathematical formula based on the correction content as a group of fine-tuning samples to fine-tune the optical character recognition model based on multiple groups of fine-tuning samples.

[0087] The interactive page can be understood as a page that can be used to display the mathematical formula.

[0088] In the embodiments of the present application, the mathematical formula can be displayed on the interactive interface.

[0089] The correction instruction can be understood as an instruction indicating correction of the mathematical formula.

[0090] The correction content can be understood as content for correcting the mathematical formula; the correction content may, for example, include at least one of corrected specific formula content (corrected correct formula content) and formula anomaly type (such as at least one of symbol confusion and structure error).

[0091] In the embodiments of the present application, the correction content can be obtained in response to the correction instruction; for example, the correction content can be obtained in response to a correction instruction generated by language description or gesture selection (marking an error area on the mathematical formula) for the displayed mathematical formula.

[0092] The correction result can be understood as a result obtained by correcting the mathematical formula based on the correction content; the correction result may, for example, be a correct mathematical formula after correction.

[0093] The fine-tuning sample can be understood as a sample for fine-tuning the optical character recognition model.

[0094] In the embodiments of the present application, the mathematical formula, the correction content, and the correction result can be taken as a group of fine-tuning samples to fine-tune the optical character recognition model based on multiple groups of fine-tuning samples.

[0095] In the embodiments of the present application, the mathematical formula recognition result can also be updated according to the correction result and the semantic information.

[0096] In the embodiments of the present application, the mathematical formula can be displayed on the interactive interface, and then in response to the correction instruction, the correction content can be obtained, and the mathematical formula, the correction content, and the correction result can be taken as a group of fine-tuning samples to fine-tune the optical character recognition model based on multiple groups of fine-tuning samples. The above scheme can provide an interactive correction mathematical formula function with a closed loop of instruction feedback and error correction processing, and the optical character recognition model can be fine-tuned by correction of the mathematical formula, online learning of the optical character recognition model can be realized, and the long-term accuracy of the optical character recognition model can be improved.

[0097] In another optional technical solution, the optical character recognition model comprises an encoder and a Transformer decoder, the Transformer decoder comprises a plurality of attention layers; the mathematical formula content is input into the optical character recognition model, and a mathematical formula is obtained according to an output result of the optical character recognition model, comprising: the mathematical formula content is input into the optical character recognition model, so as to encode the mathematical formula content by using the encoder to obtain mathematical formula features, and the decoder is used to select a target attention layer from the plurality of attention layers according to the complexity of the mathematical formula content, and the mathematical formula features are decoded based on the target attention layer; the mathematical formula is obtained according to the decoding result output by the optical character recognition model.

[0098] The encoder can be understood as an encoder for encoding the mathematical formula.

[0099] The Transformer decoder can be understood as a decoder for decoding the mathematical formula features by using the Transformer.

[0100] The attention layer can be understood as a sequence modeling module, the input of which is a sequence, and the output is a weighted aggregation result of the input sequence.

[0101] In the embodiment of the application, the optical character recognition model may, for example, adopt a model based on a pixel-to-LaTeX converter (Pix2Tex) Encoder-Decoder Architecture, the optical character recognition model comprising an encoder and a Transformer decoder, the encoder can extract image features, and the Transformer decoder can decode the feature sequence into a LaTeX symbol sequence.

[0102] The mathematical formula features can be understood as features of the mathematical formula content.

[0103] In the embodiment of the application, the mathematical formula content can be input into the optical character recognition model to encode the mathematical formula content by using the encoder to obtain the mathematical formula features.

[0104] The complexity can be understood as the complexity of the mathematical formula content; the complexity may, for example, be the complexity of at least one aspect of the length and the symbol density of the mathematical formula content, etc.

[0105] The target attention layer can be understood as an attention layer for decoding the mathematical formula features.

[0106] In the embodiment of the present application, the decoder can be used to select target attention layers from the plurality of attention layers according to the complexity, for example, the decoder can be used to determine the proportion of attention layers, and select target attention layers in the proportion of attention layers from the plurality of attention layers (for example, a simple mathematical formula content such as E=mc^2 only needs 50% of the target attention layers in the proportion of attention layers, and a matrix equation mathematical formula content may need all attention layers as target attention layers).

[0107] In the embodiment of the present application, the decoding of the mathematical formula feature can be based on the target attention layer, for example, the decoding of the mathematical formula feature can be based on the target attention layer and dynamically skip the calculation of part of the attention layer.

[0108] In the embodiment of the present application, the optical character recognition model includes an encoder and a Transformer decoder, the Transformer decoder includes a plurality of attention layers, the mathematical formula content is input into the optical character recognition model, the encoding of the mathematical formula content is performed by using the encoder to obtain the mathematical formula feature, the decoding of the mathematical formula feature is performed by using the decoder according to the complexity to select target attention layers from the plurality of attention layers, and the mathematical formula is obtained according to the decoding result. The above technical solution can realize adaptive calculation by pruning strategy of selecting target attention layers from the plurality of attention layers, reduce the calculation time of the optical character recognition model, and improve the efficiency of mathematical formula recognition.

[0109] Figure 3 is a flowchart of another mathematical formula recognition method provided in the embodiment of the present application. The present embodiment is optimized on the basis of the above technical solutions. In the present embodiment, the mathematical formula obtained after recognizing the mathematical formula content is represented by the LaTeΧ format, and the method further includes: converting the mathematical formula represented by the LaTeΧ format into a mathematical formula represented by a mathematical markup language format; and displaying the mathematical formula represented by the LaTeΧ format and the mathematical formula represented by the mathematical markup language format on a preset interactive interface. Wherein, the same or corresponding terms as in the above embodiments are not repeated here.

[0110] Referring to Figure 3 The method of the present embodiment can specifically include the following steps:

[0111] S310, by listening to the clipboard, an image captured in the clipboard is obtained, and the mathematical formula content and the context content of the mathematical formula content are extracted from the image.

[0112] S320, the mathematical formula content is recognized to obtain a mathematical formula represented by the LaTeΧ format.

[0113] The LaTeX format can be understood as a TeX-based markup typesetting format.

[0114] In the embodiments of the present application, the mathematical formula content can be recognized to obtain a mathematical formula represented by the LaTeX format.

[0115] S330, converting the mathematical formula represented by the LaTeX format into a mathematical formula represented by a mathematical markup language format.

[0116] The mathematical markup language format (MathML) can be understood as a standardized markup language format based on the eXtensible Markup Language (XML).

[0117] In the embodiments of the present application, the mathematical formula represented by the LaTeX format can be converted into a mathematical formula represented by the mathematical markup language format, for example, the LaTeX to MathML converter (latex2mathml) can be used to convert the mathematical formula represented by the LaTeX format with semantic preservation into a mathematical formula represented by the mathematical markup language format in compliance with the World Wide Web Consortium (W3C) standard.

[0118] S340, displaying the mathematical formula represented by the LaTeX format and the mathematical formula represented by the mathematical markup language format on a preset interactive interface.

[0119] In the embodiments of the present application, the mathematical formula represented by the LaTeX format and the mathematical formula represented by the mathematical markup language format can be displayed on the preset interactive interface, for example, the mathematical formula represented by the LaTeX format and the mathematical formula represented by the mathematical markup language format can be displayed in parallel in two text fields on the preset interactive interface, so as to visually compare the mathematical formulas in different formats.

[0120] S350, extracting key content related to the mathematical formula from the context content, and recognizing the key content to obtain semantic information of the mathematical formula.

[0121] S360, taking the mathematical formula and the semantic information as an image-based mathematical formula recognition result.

[0122] The technical scheme of the embodiment of the present application is that the mathematical formula obtained after recognizing the mathematical formula content is represented by the LaTeX format, the mathematical formula represented by the LaTeX format is converted into a mathematical formula represented by the mathematical markup language format, and the mathematical formula represented by the LaTeX format and the mathematical formula represented by the mathematical markup language format are displayed on the preset interactive interface. Compared with the related scheme supporting only a single output format, the above scheme can realize the display of mathematical formulas in multiple formats, so that the user can compare the mathematical formulas in different formats.

[0123] An optional technical scheme is that after the mathematical formula represented by the LaTeX format is converted into a mathematical formula represented by the mathematical markup language format, the mathematical formula recognition method further comprises: embedding the annotations in the mathematical formula represented by the LaTeX format into the mathematical formula represented by the mathematical markup language format.

[0124] The annotations can be understood as the annotation content of the mathematical formula represented by the LaTeX format; for example, the annotations can be the labels in the mathematical formula represented by the LaTeX format.

[0125] It can be understood that the mathematical formula represented by the LaTeX format can include annotations such as the labels of the standard MathML mathematical formula main content and the non-standard representation form. In order to ensure that the content is not lost in the mathematical formula format conversion, in the embodiment of the present application, the annotations can be embedded into the mathematical formula represented by the mathematical markup language format, for example, the annotations can be embedded into the mathematical formula represented by the mathematical markup language format by using the automatic generation technology of the nested annotation structure, that is, the double-format nested structure can be used to embed the original annotations of LaTeX (that is, the LaTeX <annotation>Tag).

[0126] In the embodiments of the present application, the mark can be embedded into a mathematical formula represented by a mathematical markup language format. The above scheme can ensure that no content is lost in the mathematical formula format conversion, that is, the semantics in the mathematical formula format conversion is maintained.

[0127] In order to better understand the technical solutions of the above embodiments of the present application, an optional example is provided. For example, referring to Figure 4 , load the pre-trained LatexOCR model into the local memory of the terminal device (for example, it can be loaded in the _init_ phase); initialize the graphical interactive interface framework (for example, it can be realized through the Tk interface (Tk Interface, tkinter)); listen to the system clipboard to obtain an image (for example, it can be realized through the ImageGrab: Grab Clipboard Image (Version 0) method version 0 (ImageGrab.grabclipboard0)); call the LatexOCR model to recognize the data formula (for example, it can be realized through the Model Inference on Image (model(image))); automatically convert the mathematical formula represented by the LaTeX format into a mathematical formula represented by the MathML format (for example, it can be realized through the LaTeX to MathML Converter (latex2mathml.converter)); display the image, the mathematical formula recognition result, the mathematical formula represented by the LaTeX format, and the mathematical formula represented by the MathML format on the interactive interface; in response to the triggering operation of the re-recognition button, refresh the content in the clipboard (for example, it can be realized by calling update_content()) and repeat the step of listening to the system clipboard to obtain an image; in response to the triggering operation of the one-key copy button, one-key copy the content displayed on the interactive interface (for example, it can be directly written to the clipboard, for example, it can be realized through the Python Clipboard (pyperclip)); perform real-time exception capture (for example, capture whether the clipboard is without an image or the model recognition is incorrect) and error handling on the exception to provide an error handling mechanism (for example, it can be realized through a try-except block to capture exceptions). The above technical solution can reduce the response time to seconds through the local model, save network transmission time compared with the online service mode, improve the recognition efficiency, avoid network fluctuation through offline operation, improve the reliability, support the generation of LaTeX and MathML two standard formats, realize multi-format compatibility, and realize one-key recognition process through single input image.

[0128] In order to better understand the technical solutions of the above embodiments of the application, another optional example is provided here. For example, referring to Figure 5a 、 Figure 5b and Figure 5c , listen to the clipboard, and determine whether an image is captured in the clipboard; if captured, obtain the image, and display a thumbnail corresponding to the generated image; convert a mathematical formula represented by a LaTeX format into a mathematical formula represented by a MathML format; display the mathematical formula represented by the LaTeX format and the mathematical formula represented by the MathML format on a preset interactive interface; in response to a triggering operation on a one-key copy button, one-key copy content corresponding to the one-key copy button displayed on the interactive interface; if not captured, display prompt information on the interactive interface.

[0129] For better understanding of the technical solutions of the above-mentioned embodiments of the present application, another optional example is provided. Illustratively, the technical solutions of the embodiments of the present application can be implemented by using the integrated system architecture of the graphical user interface (Graphical User Interface, GUI). The interactive interface in the graphical user interface can include a picture preview area (for example, it can be implemented by using a themed Tkinter label (Themed Tkinter Label, ttk.Label)), a multi-format result display area (for example, it can be implemented by using a scrolled text component (Scrolled Text Widget, ScrolledText)), and an operation control panel. The picture preview area can be used to display images, the multi-format result display area can be used to display mathematical formulas represented by a LaTeX format and mathematical formulas represented by a MathML format, and the operation control panel can provide a button group for indicating re-identification of mathematical formulas and / or copying of mathematical formula recognition results.

[0130] Figure 6 A structural block diagram of a mathematical formula recognition device provided by the embodiments of the present application is provided. The device is used to execute the mathematical formula recognition method provided by any of the above-mentioned embodiments. The device and the mathematical formula recognition method of each of the above-mentioned embodiments belong to the same inventive concept. Details not described in the embodiments of the mathematical formula recognition device can be referred to the embodiments of the mathematical formula recognition method. Referring to Figure 6 The device can specifically include: a context content extraction module 410, a mathematical formula obtaining module 420, a semantic information obtaining module 430, and a mathematical formula recognition result providing module 440.

[0131] The context content extraction module 410 is configured to obtain an image captured in the clipboard by listening to the clipboard, and extract mathematical formula content and context content of the mathematical formula content from the image.

[0132] The mathematical formula obtaining module 420 is configured to identify the mathematical formula content, and obtain a mathematical formula.

[0133] The semantic information obtaining module 430 is configured to extract key content related to the mathematical formula from the context content, and identify the key content to obtain semantic information of the mathematical formula.

[0134] The mathematical formula recognition result as module 440 is configured to take the mathematical formula and the semantic information as a mathematical formula recognition result of the image.

[0135] Optionally, the mathematical formula obtaining module 420 configured in the terminal device can include:

[0136] The optical character recognition model obtaining sub-module is configured to obtain an optical character recognition model deployed on the terminal device.

[0137] The mathematical formula obtaining sub-module is configured to input the mathematical formula content into the optical character recognition model, and obtain the mathematical formula according to an output result of the optical character recognition model.

[0138] Optionally, the device can further include:

[0139] The first mathematical formula display module is configured to display the mathematical formula on a preset interactive interface after obtaining the mathematical formula.

[0140] The fine-tuning sample as module is configured to obtain correction content in response to a correction instruction for the displayed mathematical formula, and take the mathematical formula, the correction content, and a correction result of the mathematical formula corrected based on the correction content as a group of fine-tuning samples, so as to fine-tune the optical character recognition model based on multiple groups of fine-tuning samples.

[0141] Optionally, the optical character recognition model includes an encoder and a Transformer decoder, and the Transformer decoder includes multiple attention layers.

[0142] The mathematical formula obtaining sub-module can include:

[0143] The mathematical formula feature decoding unit is configured to input the mathematical formula content into the optical character recognition model, to encode the mathematical formula content by using the encoder to obtain mathematical formula features, and to filter out a target attention layer from the multiple attention layers according to a complexity of the mathematical formula content by using the decoder, and to decode the mathematical formula features based on the target attention layer.

[0144] The mathematical formula obtaining unit is configured to obtain the mathematical formula according to a decoding result output by the optical character recognition model.

[0145] Optionally, the mathematical formula obtained after recognizing the mathematical formula content is represented by using the LaTeX format, and the device can further include:

[0146] The format conversion module is configured to convert the mathematical formula expressed in the LaTeΧ format into a mathematical formula expressed in the MathML format.

[0147] The second mathematical formula display module is configured to display the mathematical formula expressed in the LaTeΧ format and the mathematical formula expressed in the MathML format on the preset interactive interface.

[0148] Optionally, the device can further include:

[0149] The label embedding module is configured to embed the label in the mathematical formula expressed in the LaTeΧ format into the mathematical formula expressed in the MathML format after the conversion.

[0150] Optionally, the context content extraction module 410 can include:

[0151] The capture judgment sub-module is configured to listen to the clipboard and determine whether the image to be subjected to mathematical formula recognition is captured in the clipboard.

[0152] The image acquisition sub-module is configured to acquire the image if the image is captured.

[0153] The device can further include:

[0154] The prompt information display module is configured to display prompt information on the preset interactive interface if the image is not captured, wherein the prompt information indicates that the image is not captured in the clipboard.

[0155] Optionally, the device can further include:

[0156] The thumbnail display module is configured to reduce the image based on a preset size to obtain a thumbnail if the size of the image exceeds the preset size after the image is acquired, and display the thumbnail on the interactive interface.

[0157] The image display module is configured to display the image on the interactive interface otherwise.

[0158] Optionally, the device can further include:

[0159] The degradation type obtaining module is configured to input the image to be subjected to mathematical formula recognition captured in the clipboard into a degradation classifier after the image is acquired, and obtain a degradation type of the image according to an output result of the degradation classifier.

[0160] The image as module is configured to perform enhancement processing on the image by calling a generative adversarial network corresponding to the degradation type, and take the enhanced image as the image.

[0161] The mathematical formula recognition device provided by the embodiment of the present application can obtain the mathematical formula content used to obtain the mathematical formula, and the context content, and then recognize the mathematical formula content by the mathematical formula obtaining module to obtain the mathematical formula, realize the recognition of the mathematical formula itself, extract the key content related to the mathematical formula from the context content by the semantic information obtaining module, recognize the key content to obtain the semantic information of the mathematical formula which is helpful to improve the readability of the mathematical formula recognition result, and finally take the mathematical formula and the semantic information as the mathematical formula recognition result of the image by the mathematical formula recognition result as module, so as to realize the recognition of the mathematical formula. The above device takes the mathematical formula and the semantic information which is helpful to improve the readability of the mathematical formula recognition result as the mathematical formula recognition result, so as to improve the readability of the mathematical formula recognition result, thereby solving the problem of low readability of the mathematical formula recognition result.

[0162] The mathematical formula recognition device provided by the embodiment of the present application can execute the mathematical formula recognition method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0163] It should be noted that, in the embodiments of the above mathematical formula recognition device, each unit and module included is only divided according to the function logic, but is not limited to the above division, as long as the corresponding function can be realized; in addition, the specific name of each functional unit is only for the convenience of mutual differentiation, and does not limit the protection scope of the present application.

[0164] Figure 7 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown in the figures, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0165] As Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected to the at least one processor 11 in communication. The memory stores computer programs executable by the at least one processor 11, and the processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0166] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0167] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the mathematical formula recognition method.

[0168] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the methods of embodiments of the present application are performed.

[0169] In some embodiments, the mathematical formula recognition method can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 18. In some embodiments, parts or all of the computer program can be loaded and / or installed onto electronic device 10 via, e.g., ROM 12 and / or communication unit 19. When the computer program is loaded onto RAM 13 and executed by processor 11, one or more steps of the above-described mathematical formula recognition method can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the mathematical formula recognition method by way of other means, e.g., by way of firmware.

[0170] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0171] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as part of a separate software package, and partially on a remote machine or server, or entirely on a remote machine or server.

[0172] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0173] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0174] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0175] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0176] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0177] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.< / annotation>

Claims

1. A method for recognizing mathematical formulas, characterized in that, Applied to a terminal device, the method includes: By monitoring the clipboard, the image to be recognized as a mathematical formula is captured in the clipboard, and the mathematical formula content and the context of the mathematical formula content are extracted from the image. Obtain the optical character recognition model deployed on the terminal device, the optical character recognition model including an encoder and a Transformer decoder, the Transformer decoder including multiple attention layers; The mathematical formula content is input into the optical character recognition model to encode the mathematical formula content using the encoder, thereby obtaining mathematical formula features. The decoder is then used to determine the proportion of attention layers based on the complexity of the mathematical formula content. A target attention layer of the specified proportion is selected from multiple attention layers, and the mathematical formula features are decoded based on the target attention layer. The complexity is at least one aspect of the length and symbol density of the mathematical formula content. Based on the decoding results output by the optical character recognition model, a mathematical formula is obtained; Extract key content related to the mathematical formula from the context, identify the key content, and obtain the semantic information of the mathematical formula; The mathematical formula and the semantic information are used as the mathematical formula recognition result of the image.

2. The method according to claim 1, characterized in that, After obtaining the mathematical formula, the following is also included: The mathematical formula is displayed on a preset interactive interface; In response to a correction instruction for the displayed mathematical formula, the correction content is obtained, and the mathematical formula, the correction content, and the correction result after correcting the mathematical formula based on the correction content are used as a set of fine-tuning samples to fine-tune the optical character recognition model based on multiple sets of fine-tuning samples.

3. The method according to claim 1, characterized in that, The mathematical formula obtained after recognizing its content is represented using the Lattech format. The method further includes: The mathematical formula expressed in the Ratach format is converted into the mathematical formula expressed in the mathematical markup language format; The mathematical formulas expressed in the Lattech format and the mathematical markup language format are displayed on a preset interactive interface.

4. The method according to claim 3, characterized in that, After converting the mathematical formula represented in the Ratach format into the mathematical formula represented in the mathematical markup language format, the method further includes: The annotations in the mathematical formula expressed in the Ratach format are embedded into the mathematical formula expressed in the mathematical markup language format.

5. The method according to claim 1, characterized in that, The step of acquiring the image to be recognized for mathematical formulas captured in the clipboard by monitoring the clipboard includes: Monitor the clipboard and determine whether the image to be recognized as a mathematical formula has been captured in the clipboard; If captured, then acquire the image; The method further includes: If the image is not captured, a prompt message will be displayed on a preset interactive interface, indicating that the image was not captured in the clipboard.

6. The method according to claim 5, characterized in that, After acquiring the image, the process further includes: If the size of the image exceeds a preset size, the image is reduced in size based on the preset size to obtain a thumbnail, and the thumbnail is displayed on the interactive interface; Otherwise, the image is displayed on the interactive interface.

7. The method according to claim 1, characterized in that, After acquiring the image to be recognized as a mathematical formula captured in the clipboard, the method further includes: The image is input into a degradation classifier, and the degradation type of the image is obtained based on the output of the degradation classifier. The image is enhanced by invoking a generative adversarial network corresponding to the degradation type, and the enhanced image is used as the image.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the mathematical formula recognition method as described in any one of claims 1-7.

Images