Mathematical formula identification method and electronic equipment

Abstract

The embodiment of the invention discloses a mathematical formula identification method and electronic equipment. The method comprises the following steps: acquiring an image which is captured in a clipboard and is to be subjected to mathematical formula identification by monitoring the clipboard, and extracting mathematical formula contents and context contents of the mathematical formula contents from the image; identifying the content of the mathematical formula to obtain the mathematical formula; extracting key contents related to the mathematical formula from the context contents, and identifying the key contents to obtain semantic information of the mathematical formula; and taking the mathematical formula and the semantic information as a mathematical formula identification result of the image. According to the technical scheme provided by the embodiment of the invention, the problem of low readability of a mathematical formula identification result is solved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image recognition technology, and in particular to a mathematical formula recognition method and electronic device. Background Art

[0002] The technologies related to recognizing mathematical formulas from images are involved in application scenarios such as web page development, scientific research paper writing, teacher blackboard writing recognition, and students taking photos to solve problems.

[0003] However, the current mathematical formula recognition results are not very readable and need to be solved urgently. Summary of the Invention

[0004] The embodiments of the present invention provide a mathematical formula recognition method and electronic device, which solve the problem of poor readability of mathematical formula recognition results.

[0005] According to one aspect of the present invention, a mathematical formula recognition method is provided, which may include:

[0006] By monitoring the clipboard, the image to be subjected to mathematical formula recognition captured in the clipboard is obtained, and the mathematical formula content and the context content of the mathematical formula content are extracted from the image;

[0007] Identify the content of mathematical formulas and obtain the mathematical formulas;

[0008] Extract key content related to the mathematical formula from the context, identify the key content, and obtain the semantic information of the mathematical formula;

[0009] The mathematical formula and semantic information are used as the mathematical formula recognition result of the image.

[0010] According to another aspect of the present invention, there is provided an electronic device, which may include:

[0011] at least one processor; and

[0012] a memory communicatively connected to at least one processor; wherein,

[0013] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor implements the mathematical formula recognition method provided by any embodiment of the present invention when executing the computer program.

[0014] The technical solution of the embodiment of the present invention monitors the clipboard, obtains the image to be subjected to mathematical formula recognition captured in the clipboard, and extracts the mathematical formula content and the contextual content of the mathematical formula content from the image. Not only can the mathematical formula content used to obtain the mathematical formula be obtained, but also the contextual content can be obtained. The mathematical formula content is then recognized to obtain the mathematical formula, thereby realizing the recognition of the data formula itself. The key content related to the mathematical formula is then extracted from the contextual content, and the key content is recognized to obtain the semantic information of the mathematical formula that helps improve the readability of the mathematical formula recognition result. Finally, the mathematical formula and the semantic information are used as the mathematical formula recognition result of the image, thereby realizing the recognition of the mathematical formula. The above technical solution improves the readability of the mathematical formula recognition result by using the mathematical formula and the semantic information that helps improve the readability of the mathematical formula recognition result as the mathematical formula recognition result, thereby solving the problem of poor readability of the mathematical formula recognition result.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a flow chart of a mathematical formula recognition method provided according to an embodiment of the present invention;

[0018] Figure 2 is a flowchart of another mathematical formula recognition method provided according to an embodiment of the present invention;

[0019] Figure 3 is a flowchart of another mathematical formula recognition method provided by an embodiment of the present invention;

[0020] Figure 4 is a flowchart of an optional example of another mathematical formula recognition method provided by an embodiment of the present invention;

[0021] Figure 5a is a schematic diagram of a clipboard in another optional example of another mathematical formula recognition method provided by an embodiment of the present invention;

[0022] Figure 5bis a schematic diagram of another clipboard in another optional example of another mathematical formula recognition method provided according to an embodiment of the present invention;

[0023] Figure 5c is a schematic diagram of another clipboard in another optional example of another mathematical formula recognition method provided by an embodiment of the present invention;

[0024] Figure 6 This is a structural block diagram of a mathematical formula recognition device provided according to an embodiment of the present invention;

[0025] Figure 7 The figure is a schematic diagram of the structure of an electronic device for implementing the mathematical formula recognition method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The situations of "target", "original", etc. are similar and will not be repeated here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices.

[0028] Figure 1 This is a flowchart of a mathematical formula recognition method provided in an embodiment of the present invention. This embodiment is applicable to mathematical formula recognition. The method can be performed by a mathematical formula recognition device provided in an embodiment of the present invention. The device can be implemented in software and / or hardware and can be integrated into an electronic device, such as various user terminals or servers.

[0029] See also Figure 1 The method of the embodiment of the present invention specifically includes the following steps:

[0030] S110 , by monitoring the clipboard, obtaining the image to be subjected to mathematical formula recognition captured in the clipboard, and extracting the mathematical formula content and the context content of the mathematical formula content from the image.

[0031] The clipboard can be understood as a clipboard that can be used to cut images.

[0032] The image can be understood as an image to be subjected to mathematical formula recognition.

[0033] In an embodiment of the present invention, an image can be obtained by monitoring the clipboard. For example, it can be monitored whether the content in the clipboard is a picture. If it is a picture, the image is obtained to avoid mathematical formula recognition errors caused by obtaining non-images.

[0034] In the embodiment of the present invention, the image can be copied to the clipboard, and the image can also be dragged to the clipboard.

[0035] Mathematical formula content can be understood as content including mathematical formulas.

[0036] The context content can be understood as the content of the context of the mathematical formula content; the context content can, for example, include the previous and next paragraphs of the mathematical formula content, or can be the content of a preset number of words, a preset number of paragraphs, or a preset number of lines before the mathematical formula content, and so on.

[0037] It is understandable that in scientific research documents such as papers, mathematical formulas are often related to the surrounding text (such as variable definitions). Therefore, in order to improve the readability of the mathematical formula recognition results, in an embodiment of the present invention, the mathematical formula content and contextual content can be extracted from the image. For example, the areas corresponding to the mathematical formula content and contextual content in the image can be segmented, and the mathematical formula content and contextual content can be extracted from the segmented areas to perform mathematical formula recognition based on the mathematical formula content and contextual content, thereby improving the readability of the mathematical formula recognition results.

[0038] In an embodiment of the present invention, before extracting the mathematical formula content and the contextual content of the mathematical formula content from the image, the image may be pre-processed by at least one of enhancement and annotation, etc. For example, the mathematical formula content part, the contextual content part and other content parts in the image may be annotated.

[0039] S120: Identify the content of the mathematical formula and obtain the mathematical formula.

[0040] The mathematical formula can be understood as a mathematical formula identified from the content of the mathematical formula; the number of the mathematical formula can be at least one.

[0041] In an embodiment of the present invention, the content of a mathematical formula can be identified and the mathematical formula can be obtained. For example, the content of the mathematical formula can be identified and the mathematical formula can be obtained through at least one of an online optical character recognition (OCR) formula recognition service such as SimpleTex, a client-based formula recognition tool such as Mathpix, and a cloud-based formula recognition application programming interface (API) interface; considering that the above-mentioned solution of identifying mathematical formulas in an online manner may have network delays, server congestion risks, the need for continuous network connection support, the inability to use offline, and the need to upload privacy-sensitive data to a third-party server, etc., for example, the content of the mathematical formula can be identified and the mathematical formula can be obtained through a localized OCR model.

[0042] S130 : Extract key content related to the mathematical formula from the context content, identify the key content, and obtain semantic information of the mathematical formula.

[0043] The key content may be understood as content related to mathematical formulas; the key content may include, for example, keywords and / or key sentences, etc.

[0044] Semantic information can be understood as information related to the semantics of a mathematical formula; for example, semantic information can be at least one of the semantic annotations of the mathematical formula (such as annotations of parameters and annotations of the effects of the mathematical formula, etc.), the type of the mathematical formula, and the application field of the mathematical formula, etc.

[0045] In the embodiment of the present invention, key content may be extracted from the context content. For example, key content including parameters in a mathematical formula and / or key content describing the mathematical formula may be extracted from the context content.

[0046] In an embodiment of the present invention, key content can be identified and semantic information can be obtained. For example, based on preset identification dimensions (such as at least one of the annotation dimension of the parameter, the dimension of the formula effect, the type dimension of the mathematical formula, and the application field dimension of the mathematical formula, etc.), the content in the key content corresponding to the identification dimension can be identified as semantic information.

[0047] S140: Taking the mathematical formula and semantic information as a mathematical formula recognition result of the image.

[0048] The mathematical formula recognition result can be understood as a recognition result composed of a mathematical formula and semantic information.

[0049] In the embodiment of the present invention, the mathematical formula and semantic information may be used as the mathematical formula recognition result.

[0050] In the embodiment of the present invention, after the mathematical formula and semantic information are used as the mathematical formula recognition result of the image, the mathematical formula recognition result may also be displayed.

[0051] The technical solution of the embodiment of the present invention monitors the clipboard, obtains the image to be subjected to mathematical formula recognition captured in the clipboard, and extracts the mathematical formula content and the contextual content of the mathematical formula content from the image. Not only can the mathematical formula content used to obtain the mathematical formula be obtained, but also the contextual content can be obtained. The mathematical formula content is then recognized to obtain the mathematical formula, thereby realizing the recognition of the data formula itself. The key content related to the mathematical formula is then extracted from the contextual content, and the key content is recognized to obtain the semantic information of the mathematical formula that helps improve the readability of the mathematical formula recognition result. Finally, the mathematical formula and the semantic information are used as the mathematical formula recognition result of the image, thereby realizing the recognition of the mathematical formula. The above technical solution improves the readability of the mathematical formula recognition result by using the mathematical formula and the semantic information that helps improve the readability of the mathematical formula recognition result as the mathematical formula recognition result, thereby solving the problem of poor readability of the mathematical formula recognition result.

[0052] An optional technical solution, by monitoring the clipboard, obtains the image captured in the clipboard to be subjected to mathematical formula recognition, including: monitoring the clipboard and determining whether the image to be subjected to mathematical formula recognition is captured in the clipboard; if captured, obtaining the image; the method also includes: if not captured, displaying a prompt message on a preset interactive interface, wherein the prompt message indicates that the image is not captured in the clipboard.

[0053] The prompt information may be understood as information indicating that no image has been captured in the clipboard.

[0054] In an embodiment of the present invention, the clipboard can be monitored and it can be determined whether an image is captured in the clipboard. For example, an image grabber (ImageGrab) can be used to monitor the system clipboard in real time or with a refresh period less than a preset period (for example, a refresh period less than 3 seconds) and determine whether an image is captured in the clipboard, that is, to achieve real-time image capture based on the system clipboard through ImageGrab.

[0055] In an embodiment of the present invention, if captured, an image is acquired.

[0056] In an embodiment of the present invention, if the image is not captured, a prompt message is displayed on a preset interactive interface. For example, a Tkinter message box (Tkinter Message Box, messagebox) can be used to display the prompt message on the preset interactive interface.

[0057] In an embodiment of the present invention, the clipboard can be monitored and a determination can be made as to whether an image is captured in the clipboard. If so, the image is acquired; if not, a prompt message is displayed on the interactive interface. The above solution can provide a clipboard image capture function by linking clipboard monitoring and image capture, and can display a visual prompt message on the interactive interface if the image is not captured, so that the user can clearly understand whether the image is captured.

[0058] Based on the above scheme, another optional technical scheme is that after acquiring the image, the mathematical formula recognition method also includes: if the size of the image exceeds the preset size, the image is reduced based on the preset size to obtain a thumbnail, and the thumbnail is displayed on the interactive interface; otherwise, the image is displayed on the interactive interface.

[0059] The preset size may be understood as the preset minimum size of the image when the image is not reduced; the preset size may be, for example, 300x300.

[0060] A thumbnail image can be understood as an image obtained by reducing an image based on a preset size.

[0061] In an embodiment of the present invention, if the size of the image exceeds a preset size, the image is reduced based on the preset size to obtain a thumbnail. For example, an Image Thumbnail Method (image.thumbnail) may be used to reduce the image based on the preset size to obtain a thumbnail.

[0062] In an embodiment of the present invention, if the size of the image does not exceed a preset size, the image is displayed on the interactive interface.

[0063] In this embodiment of the present invention, if the image size exceeds a preset size, the image is scaled down based on the preset size to obtain a thumbnail, which is then displayed on the interactive interface. Otherwise, the image is displayed on the interactive interface. This technical solution avoids the problem of being unable to display the entire image due to excessive image size, thereby optimizing the display of large images.

[0064] Another optional technical solution, after obtaining the image to be subjected to mathematical formula recognition captured in the clipboard, the mathematical formula recognition method further includes: inputting the image into a degradation classifier, and obtaining the degradation type of the image based on the output result of the degradation classifier; enhancing the image by calling a generative adversarial network corresponding to the degradation type, and using the enhanced image as the image.

[0065] The degradation classifier may be understood as a classifier for determining the degradation type of an image.

[0066] The degradation type can be understood as the type of image degradation. The degradation type can also be understood as the abnormal type of the image that may affect the accuracy of mathematical formula recognition. The degradation type can include at least one of blur, low light, and compression artifacts.

[0067] In the embodiment of the present invention, the image may be input into a degradation classifier, and the degradation type may be obtained based on the output result.

[0068] Generative adversarial networks can be understood as networks that can enhance images.

[0069] In an embodiment of the present invention, an image can be enhanced by calling a generative adversarial network corresponding to the degradation type, and the enhanced image can be used as the image. For example, corresponding generative adversarial networks can be pre-set for alternative types that can be used as degradation types, and the image can be enhanced by calling a generative adversarial network corresponding to the degradation type from the generative adversarial networks corresponding to the alternative types, and the enhanced image can be used as the image.

[0070] In this embodiment of the present invention, an image is input into a degradation classifier, and the degradation type is determined based on the output. A generative adversarial network is then invoked to enhance the image, and the enhanced image is used as the image. This approach can improve the recognition rate of mathematical formulas in low-quality images.

[0071] Figure 2 This is a flow chart of another mathematical formula recognition method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, optionally, the method is applied to a terminal device to recognize the content of a mathematical formula and obtain the mathematical formula, including: obtaining an optical character recognition model deployed on the terminal device; inputting the mathematical formula content into the optical character recognition model; and obtaining the mathematical formula based on the output of the optical character recognition model.

[0072] The explanations of terms that are the same as or corresponding to the above embodiments are not repeated here.

[0073] See also Figure 2 The method of this embodiment is applied to a terminal device, and the method may specifically include the following steps:

[0074] S210 , by monitoring the clipboard, obtaining the image to be subjected to mathematical formula recognition captured in the clipboard, and extracting the mathematical formula content and the context content of the mathematical formula content from the image.

[0075] Here, the terminal device can be understood as a local device for mathematical formula recognition.

[0076] S220: Acquire an optical character recognition model deployed on the terminal device.

[0077] The optical character recognition model may be understood as a model for mathematical formula recognition using an optical character recognition method; the optical character recognition model may be, for example, a LaTeX OCR deep learning model.

[0078] It is understandable that the optical character recognition model deployed on the terminal device can also be understood as a locally deployed optical character recognition model.

[0079] In an embodiment of the present invention, the optical character recognition model may be preloaded. To increase the model loading speed, the optical character recognition model may be asynchronously loaded through a method such as threading.

[0080] In the embodiment of the present invention, the samples used to train the optical character recognition model may include not only printed samples, but also samples in different forms such as handwritten samples, so as to improve the applicability of the optical character recognition model.

[0081] S230: Input the mathematical formula content into an optical character recognition model, and obtain the mathematical formula according to the output result of the optical character recognition model.

[0082] In an embodiment of the present invention, the mathematical formula content can be input into the optical character recognition model, and the mathematical formula can be obtained based on the output result. For example, the mathematical formula content can be input into the optical character recognition model to obtain an output result in the form of compilable LaTeX code (for example, \frac{a}{b}), and the mathematical formula can be obtained based on the output result.

[0083] S240: Extract key content related to the mathematical formula from the context content, identify the key content, and obtain semantic information of the mathematical formula.

[0084] S250: Taking the mathematical formula and semantic information as a mathematical formula recognition result of the image.

[0085] The technical solution of an embodiment of the present invention is applied to a terminal device, obtaining an optical character recognition model deployed on the terminal device, then inputting the mathematical formula content into the optical character recognition model, and obtaining the mathematical formula based on the output of the optical character recognition model. This solution, by recognizing mathematical formulas using the optical character recognition model deployed on the terminal device, can avoid network delays, queuing, and uploading sensitive formula data to a cloud server, thereby improving the efficiency and security of mathematical formula recognition.

[0086] An optional technical solution, after obtaining the mathematical formula, the mathematical formula recognition method also includes: displaying the mathematical formula on a preset interactive interface; responding to a correction instruction for the displayed mathematical formula, obtaining correction content, and using the mathematical formula, the correction content, and the correction result after correcting the mathematical formula based on the correction content as a set of fine-tuning samples, so as to fine-tune the optical character recognition model based on multiple sets of fine-tuning samples.

[0087] The interactive page can be understood as a page that can be used to display mathematical formulas.

[0088] In an embodiment of the present invention, mathematical formulas may be displayed on an interactive interface.

[0089] The correction instruction can be understood as an instruction to correct a mathematical formula.

[0090] The correction content can be understood as the content of the correction of the mathematical formula; the correction content may include, for example, the specific formula content of the correction (correct formula content of the correction) and at least one of the formula abnormality types (such as at least one of symbol confusion and structural errors, etc.).

[0091] In an embodiment of the present invention, correction content can be obtained in response to a correction instruction. For example, correction content can be obtained in response to a correction instruction generated by language description or gesture circling (marking the error area on the mathematical formula) for a displayed mathematical formula.

[0092] The correction result can be understood as a result obtained after correcting the mathematical formula based on the correction content; the correction result can be, for example, a correct mathematical formula after correction.

[0093] Fine-tuning samples can be understood as samples used to fine-tune the optical character recognition model.

[0094] In the embodiment of the present invention, the mathematical formula, the correction content, and the correction result may be used as a set of fine-tuning samples, so as to fine-tune the optical character recognition model based on multiple sets of fine-tuning samples.

[0095] In the embodiment of the present invention, the mathematical formula recognition result may also be updated according to the correction result and the semantic information.

[0096] In an embodiment of the present invention, a mathematical formula can be displayed on an interactive interface. In response to a correction instruction, the correction content can be obtained. The mathematical formula, the correction content, and the correction result can be used as a set of fine-tuning samples to fine-tune the optical character recognition model based on multiple sets of fine-tuning samples. This solution can provide a closed-loop interactive mathematical formula correction function that integrates instruction feedback and error correction processing. By fine-tuning the optical character recognition model based on the correction of the mathematical formula, online learning of the optical character recognition model can be achieved, thereby improving the long-term accuracy of the optical character recognition model.

[0097] Another optional technical solution is that the optical character recognition model includes an encoder and a Transformer decoder, and the Transformer decoder includes multiple attention layers; the mathematical formula content is input into the optical character recognition model, and the mathematical formula is obtained based on the output result of the optical character recognition model, including: inputting the mathematical formula content into the optical character recognition model, using the encoder to encode the mathematical formula content to obtain mathematical formula features, and using the decoder to screen out a target attention layer from multiple attention layers according to the complexity of the mathematical formula content, and decoding the mathematical formula features based on the target attention layer; obtaining the mathematical formula based on the decoding result output by the optical character recognition model.

[0098] Here, the encoder can be understood as an encoder used to encode mathematical formulas.

[0099] The Transformer decoder can be understood as a decoder that uses the Transformer to decode mathematical formula features.

[0100] The attention layer can be understood as a sequence modeling module whose input is a sequence and whose output is the weighted aggregation result of the input sequence.

[0101] In an embodiment of the present invention, the optical character recognition model may, for example, adopt a model based on an encoder-decoder architecture (Encoder-Decoder Architecture) of a pixel-to-LaTeX converter (Pix2Tex). The optical character recognition model includes an encoder and a Transformer decoder. The encoder may extract image features, and the Transformer decoder may decode a feature sequence into a LaTeX symbol sequence.

[0102] Mathematical formula features can be understood as features of the content of the mathematical formula.

[0103] In an embodiment of the present invention, the mathematical formula content may be input into an optical character recognition model, so that an encoder may be used to encode the mathematical formula content to obtain mathematical formula features.

[0104] Complexity can be understood as the complexity of the content of the mathematical formula; for example, complexity can be the complexity of at least one of the length of the mathematical formula content and the density of symbols.

[0105] The target attention layer can be understood as the attention layer that decodes the features of mathematical formulas.

[0106] In an embodiment of the present invention, a decoder can be used to filter out a target attention layer from multiple attention layers based on complexity. For example, a decoder can be used to determine the attention layer ratio based on complexity, and filter out a target attention layer with a ratio of attention layers from multiple attention layers (for example, a simple mathematical formula content such as E=mc^2 only requires a target attention layer with a ratio of 50% of the attention layers, while a mathematical formula content such as a matrix equation may require all attention layers as target attention layers).

[0107] In an embodiment of the present invention, mathematical formula features can be decoded based on the target attention layer. For example, based on the target attention layer, the calculation of some attention layers can be dynamically skipped to decode the mathematical formula features.

[0108] In an embodiment of the present invention, an optical character recognition model includes an encoder and a Transformer decoder, wherein the Transformer decoder includes multiple attention layers. The content of a mathematical formula is input into the optical character recognition model, and the encoder is used to encode the content of the mathematical formula to obtain mathematical formula features. The decoder is used to screen out a target attention layer from the multiple attention layers based on complexity, and the mathematical formula features are decoded based on the target attention layer. The mathematical formula is then obtained based on the decoding results. The above technical solution can achieve adaptive calculation by using a pruning strategy to screen out the target attention layer from the multiple attention layers, thereby reducing the calculation time of the optical character recognition model and improving the efficiency of mathematical formula recognition.

[0109] Figure 3 This is a flowchart of another mathematical formula recognition method provided in an embodiment of the present invention. This embodiment is optimized based on the above-mentioned technical solutions. In this embodiment, the mathematical formula obtained after identifying the content of the mathematical formula is optionally expressed in Latach format. The method further includes: converting the mathematical formula expressed in Latach format into a mathematical formula expressed in a mathematical markup language format; and displaying the mathematical formula expressed in Latach format and the mathematical formula expressed in a mathematical markup language format on a preset interactive interface. The explanations of terms that are identical or corresponding to those in the above-mentioned embodiments are not repeated here.

[0110] See also Figure 3 The method of this embodiment may specifically include the following steps:

[0111] S310 , by monitoring the clipboard, obtaining the image to be subjected to mathematical formula recognition captured in the clipboard, and extracting the mathematical formula content and the context content of the mathematical formula content from the image.

[0112] S320: Identify the content of the mathematical formula and obtain the mathematical formula represented in the Lathe format.

[0113] Among them, LaTeX can be understood as a markup typesetting format based on TeX.

[0114] In the embodiment of the present invention, the content of the mathematical formula can be identified to obtain the mathematical formula represented by the Lathe format.

[0115] S330: Convert the mathematical formula expressed in the Latach format into a mathematical formula expressed in a mathematical markup language format.

[0116] Among them, the Mathematical Markup Language (MathML) format can be understood as a standardized markup language format based on the eXtensible Markup Language (XML).

[0117] In an embodiment of the present invention, a mathematical formula expressed in LaTeX format can be converted into a mathematical formula expressed in a mathematical markup language format. For example, a LaTeX to MathML converter (latex2mathml) can be used to convert the mathematical formula expressed in LaTeX format into a mathematical formula expressed in a mathematical markup language format that complies with the World Wide Web Consortium (W3C) standard while preserving its semantics.

[0118] S340. Displaying the mathematical formulas expressed in the Latach format and the mathematical formulas expressed in the Mathematical Markup Language format on a preset interactive interface.

[0119] In an embodiment of the present invention, mathematical formulas expressed in Latach format and mathematical formulas expressed in mathematical markup language format can be displayed on a preset interactive interface. For example, mathematical formulas expressed in Latach format and mathematical formulas expressed in mathematical markup language format can be displayed in parallel in two text fields on a preset interactive interface to facilitate visual comparison of mathematical formulas in different formats.

[0120] S350: Extract key content related to the mathematical formula from the context content, identify the key content, and obtain semantic information of the mathematical formula.

[0121] S360: Using the mathematical formula and semantic information as a mathematical formula recognition result of the image.

[0122] The technical solution of an embodiment of the present invention represents the mathematical formula obtained after identifying its content in Latach format, converts the mathematical formula represented in Latach format into a mathematical markup language format, and then displays both the mathematical formula represented in Latach format and the mathematical markup language format on a preset interactive interface. Compared to related solutions that only support a single output format, this solution can display formulas in multiple formats, allowing users to compare mathematical formulas in different formats.

[0123] An optional technical solution, after converting a mathematical formula expressed in Latach format into a mathematical formula expressed in mathematical markup language format, the mathematical formula recognition method further includes: embedding the annotations in the mathematical formula expressed in Latach format into the mathematical formula expressed in mathematical markup language format.

[0124] The annotation can be understood as the annotation content of the mathematical formula in the mathematical formula represented by the Latte format; the annotation can be, for example, a label in the mathematical formula represented by the Latte format.

[0125] It is understandable that a mathematical formula expressed in LaTeX format may include annotations such as the main content of the standard MathML mathematical formula and tags in non-standard representation forms. In order to ensure that the content is not lost during the conversion of the mathematical formula format, in an embodiment of the present invention, annotations can be embedded into the mathematical formula expressed in the mathematical markup language format. For example, the annotations can be embedded into the mathematical formula expressed in the mathematical markup language format through the automatic generation technology of the nested annotation structure, that is, a double-format nested structure can be adopted to embed the original LaTeX annotations (that is, LaTeX's <annotation>Label).

[0126] In an embodiment of the present invention, annotations can be embedded into mathematical formulas expressed in a mathematical markup language format. The above solution can ensure that the content is not lost during the mathematical formula format conversion, that is, it can ensure that the semantics are preserved during the mathematical formula format conversion.

[0127] In order to better understand the technical solution of the above embodiment of the present invention, an optional example is provided here. Figure 4 , load the pre-trained LatexOCR model into the local memory of the terminal device (for example, it can be loaded in the _init_ stage); initialize the graphical interactive interface framework (for example, it can be implemented through the Tk interface (Tk Interface, tkinter)); monitor the system clipboard to obtain images (for example, it can be implemented through ImageGrab: Grab Clipboard Image (Version 0), ImageGrab.grabclipboard0); call the LatexOCR model to recognize data formulas (for example, it can be implemented through Model Inference on Image (model(image))); automatically convert mathematical formulas expressed in LaTeX format into mathematical formulas expressed in mathematical markup language format (for example, it can be implemented through LaTeX to MathML converter (LaTeX to MathML converter) The system comprises the following steps: displaying an image, a mathematical formula recognition result, a mathematical formula expressed in LaTeX format, and a mathematical formula expressed in Mathematical Markup Language format on an interactive interface; in response to a triggering operation for a re-recognition button, refreshing the content in the clipboard (for example, by calling update_content()), and repeating the step of monitoring the system clipboard to obtain an image; in response to a triggering operation for a one-key copy button, copying the content displayed on the interactive interface with one key (for example, writing it directly to the clipboard, for example, by using the Python clipboard library (Python Clipboard, pyperclip)); capturing exceptions in real time (for example, capturing whether the clipboard has no image or a model recognition error) and handling exceptions to provide an error handling mechanism (for example, by capturing exceptions through a try-except block). This technical solution, through local models, can reduce response time to seconds, saving network transmission time and improving recognition efficiency compared to online services. It can run offline to avoid the impact of network fluctuations, with a measured availability of 99.6%, improving reliability. It supports the simultaneous generation of two standard formats, LaTeX and MathML, achieving multi-format compatibility. A single input image can realize mathematical formula recognition and conversion, thus realizing a one-click recognition process.

[0128] In order to better understand the technical solution of the above embodiment of the present invention, another optional example is provided here. Figure 5a 、 Figure 5b and Figure 5c , monitor the clipboard and determine whether an image is captured in the clipboard; if captured, obtain the image and display the thumbnail corresponding to the generated image; convert the mathematical formula expressed in the Latach format into a mathematical formula expressed in the mathematical markup language format; on a preset interactive interface, display the mathematical formula expressed in the Latach format and the mathematical formula expressed in the mathematical markup language format; in response to the triggering operation of the one-key copy button, copy the content corresponding to the one-key copy button displayed on the interactive interface with one key; if not captured, display a prompt message on the interactive interface.

[0129] In order to better understand the technical solution of the above-mentioned embodiment of the present invention, another optional example is provided here. Exemplarily, the technical solution of the embodiment of the present invention can be implemented using an integrated system architecture of a graphical user interface (GUI), and the interactive interface in the graphical user interface can include an image preview area (for example, it can be implemented by a themed Tkinter label (ttk.Label)), a multi-format result display area (for example, it can be implemented by a scrolling text component (Scrolled Text Widget, ScrolledText)) and an operation control panel; the image preview area can be used to display images, the multi-format result display area can be used to display mathematical formulas expressed in Latach format and mathematical formulas expressed in mathematical markup language format, and the operation control panel can provide a button group for instructing to re-recognize the mathematical formula and / or copy the mathematical formula recognition result.

[0130] Figure 6 This is a block diagram of the structure of a mathematical formula recognition device provided in an embodiment of the present invention. The device is used to execute the mathematical formula recognition method provided in any of the above embodiments. The device and the mathematical formula recognition method of the above embodiments belong to the same inventive concept. For details not fully described in the embodiments of the mathematical formula recognition device, please refer to the embodiments of the above mathematical formula recognition method. Figure 6 The device may specifically include: a context content extraction module 410, a mathematical formula acquisition module 420, a semantic information acquisition module 430 and a mathematical formula recognition result module 440.

[0131] The context content extraction module 410 is configured to obtain the image to be subjected to mathematical formula recognition captured in the clipboard by monitoring the clipboard, and extract the mathematical formula content and the context content of the mathematical formula content from the image;

[0132] The mathematical formula obtaining module 420 is used to identify the content of the mathematical formula and obtain the mathematical formula;

[0133] Semantic information acquisition module 430 is used to extract key content related to the mathematical formula from the context content, identify the key content, and obtain semantic information of the mathematical formula;

[0134] The mathematical formula recognition result is used as a module 440, which is used to take the mathematical formula and semantic information as the mathematical formula recognition result of the image.

[0135] Optionally, the mathematical formula obtaining module 420 configured in the terminal device may include:

[0136] An optical character recognition model acquisition submodule is used to acquire an optical character recognition model deployed on a terminal device;

[0137] The mathematical formula obtaining submodule is used to input the mathematical formula content into the optical character recognition model and obtain the mathematical formula according to the output result of the optical character recognition model.

[0138] Optionally, based on the above device, the device may further include:

[0139] A first mathematical formula display module is used to display the mathematical formula on a preset interactive interface after obtaining the mathematical formula;

[0140] The fine-tuning sample serves as a module for responding to a correction instruction for a displayed mathematical formula, obtaining correction content, and using the mathematical formula, the correction content, and the correction result after correcting the mathematical formula based on the correction content as a set of fine-tuning samples to fine-tune the optical character recognition model based on multiple sets of fine-tuning samples.

[0141] Optionally, based on the above device, the optical character recognition model includes an encoder and a Transformer decoder, and the Transformer decoder includes multiple attention layers;

[0142] The mathematical formulas are used to get submodules, which can include:

[0143] a mathematical formula feature decoding unit, configured to input the mathematical formula content into the optical character recognition model, encode the mathematical formula content using an encoder to obtain mathematical formula features, and use a decoder to select a target attention layer from multiple attention layers based on the complexity of the mathematical formula content, and decode the mathematical formula features based on the target attention layer;

[0144] The mathematical formula obtaining unit is used to obtain a mathematical formula according to the decoding result output by the optical character recognition model.

[0145] Optionally, the mathematical formula obtained after identifying the content of the mathematical formula is represented in Lathe format, and the device may further include:

[0146] a format conversion module for converting a mathematical formula expressed in Lathe format into a mathematical formula expressed in a mathematical markup language format;

[0147] The second mathematical formula display module is used to display mathematical formulas expressed in Lathe format and mathematical formulas expressed in mathematical markup language format on a preset interactive interface.

[0148] Optionally, based on the above device, the device may further include:

[0149] The annotation embedding module is used to convert the mathematical formula expressed in the Latach format into the mathematical formula expressed in the mathematical markup language format, and then embed the annotation in the mathematical formula expressed in the Latach format into the mathematical formula expressed in the mathematical markup language format.

[0150] Optionally, the context content extraction module 410 may include:

[0151] The capture judgment submodule is used to monitor the clipboard and determine whether the image to be recognized by the mathematical formula is captured in the clipboard;

[0152] An image acquisition submodule, for acquiring an image if captured;

[0153] The device may also include:

[0154] The prompt information display module is used to display a prompt information on a preset interactive interface if the image is not captured, wherein the prompt information indicates that the image is not captured in the clipboard.

[0155] Optionally, based on the above device, the device may further include:

[0156] A thumbnail display module is used to, after acquiring an image, if the size of the image exceeds a preset size, reduce the image based on the preset size to obtain a thumbnail, and display the thumbnail on the interactive interface;

[0157] The image display module is used to display images on the interactive interface.

[0158] Optionally, the device may further include:

[0159] A degradation type obtaining module is used to obtain the image to be subjected to mathematical formula recognition captured in the clipboard, input the image into a degradation classifier, and obtain the degradation type of the image according to the output result of the degradation classifier;

[0160] The image is used as a module to enhance the image by calling the generative adversarial network corresponding to the degradation type, and the enhanced image is used as the image.

[0161] The mathematical formula recognition device provided by the embodiment of the present invention uses a context content extraction module to monitor the clipboard, obtain the image to be subjected to mathematical formula recognition captured in the clipboard, and extract the mathematical formula content and the context content of the mathematical formula content from the image. Not only can the mathematical formula content used to obtain the mathematical formula be obtained, but also the context content can be obtained. Then, the mathematical formula obtaining module is used to identify the mathematical formula content and obtain the mathematical formula, thereby realizing the recognition of the data formula itself. Then, the semantic information obtaining module is used to extract key content related to the mathematical formula from the context content and identify the key content to obtain semantic information of the mathematical formula that helps improve the readability of the mathematical formula recognition result. Finally, the mathematical formula recognition result is used as a module to use the mathematical formula and the semantic information as the mathematical formula recognition result of the image, thereby realizing the recognition of the mathematical formula. The above-mentioned device can improve the readability of the mathematical formula recognition result by using the mathematical formula and the semantic information that helps improve the readability of the mathematical formula recognition result as the mathematical formula recognition result, thereby solving the problem of poor readability of the mathematical formula recognition result.

[0162] The mathematical formula recognition device provided in the embodiment of the present invention can execute the mathematical formula recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0163] It is worth noting that in the embodiment of the above-mentioned mathematical formula recognition device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0164] Figure 7 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0165] like Figure 7 As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0166] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0167] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the mathematical formula recognition method.

[0168] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0169] In some embodiments, the mathematical formula recognition method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the mathematical formula recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the mathematical formula recognition method in any other suitable manner (e.g., via firmware).

[0170] Various implementations of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0171] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0172] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0173] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0174] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0175] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0176] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0177] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.< / annotation>

Claims

1. A mathematical formula recognition method, characterized in that: include: By monitoring the clipboard, an image to be subjected to mathematical formula recognition captured in the clipboard is acquired, and mathematical formula content and contextual content of the mathematical formula content are extracted from the image; Identify the content of the mathematical formula to obtain the mathematical formula; Extracting key content related to the mathematical formula from the context content, identifying the key content, and obtaining semantic information of the mathematical formula; The mathematical formula and the semantic information are used as a mathematical formula recognition result of the image.

2. The method according to claim 1, characterized in that Applied to a terminal device, the identifying the content of the mathematical formula to obtain the mathematical formula includes: Obtaining an optical character recognition model deployed on the terminal device; The mathematical formula content is input into the optical character recognition model, and the mathematical formula is obtained according to the output result of the optical character recognition model.

3. The method according to claim 2, characterized in that After obtaining the mathematical formula, the method further includes: Displaying the mathematical formula on a preset interactive interface; In response to a correction instruction for the displayed mathematical formula, correction content is obtained, and the mathematical formula, the correction content, and the correction result after correcting the mathematical formula based on the correction content are used as a set of fine-tuning samples to fine-tune the optical character recognition model based on multiple sets of the fine-tuning samples.

4. The method according to claim 2, characterized in that The optical character recognition model includes an encoder and a Transformer decoder, wherein the Transformer decoder includes multiple attention layers; The step of inputting the mathematical formula content into the optical character recognition model and obtaining the mathematical formula according to the output result of the optical character recognition model includes: Inputting the mathematical formula content into the optical character recognition model, encoding the mathematical formula content using the encoder to obtain mathematical formula features, and using the decoder to screen a target attention layer from the multiple attention layers according to the complexity of the mathematical formula content, and decoding the mathematical formula features based on the target attention layer; A mathematical formula is obtained according to the decoding result output by the optical character recognition model.

5. The method according to claim 1, wherein The mathematical formula obtained after identifying the content of the mathematical formula is expressed in Lathe format, and the method further includes: converting the mathematical formula represented by the Latach format into the mathematical formula represented by a mathematical markup language format; On a preset interactive interface, the mathematical formula expressed in the Latach format and the mathematical formula expressed in the mathematical markup language format are displayed.

6. The method according to claim 5, characterized in that After converting the mathematical formula represented by the Latach format into the mathematical formula represented by a mathematical markup language format, the method further includes: The annotations in the mathematical formula expressed in the Latach format are embedded in the mathematical formula expressed in the mathematical markup language format.

7. The method according to claim 1, characterized in that The step of acquiring the image to be subjected to mathematical formula recognition captured in the clipboard by monitoring the clipboard includes: Monitoring the clipboard and determining whether an image to be subjected to mathematical formula recognition is captured in the clipboard; If captured, acquiring the image; The method further comprises: If the image is not captured, a prompt message is displayed on a preset interactive interface, wherein the prompt message indicates that the image is not captured in the clipboard.

8. The method according to claim 7, characterized in that After acquiring the image, the method further includes: If the size of the image exceeds a preset size, reducing the image based on the preset size to obtain a thumbnail, and displaying the thumbnail on the interactive interface; Otherwise, the image is displayed on the interactive interface.

9. The method according to claim 1, characterized in that After acquiring the image to be subjected to mathematical formula recognition captured in the clipboard, the method further includes: Inputting the image into a degradation classifier, and obtaining a degradation type of the image according to an output result of the degradation classifier; The image is enhanced by calling a generative adversarial network corresponding to the degradation type, and the enhanced image is used as the image.

10. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the mathematical formula recognition method according to any one of claims 1 to 9.

Images