Formula editing generation method and system based on symbol marking and pictographic input
Through a combination method based on symbolic markup language (SML) and pictographic input method, the difficulty of mathematical formula editing and input is solved, efficient formula editing and input is achieved, the process is simplified and the user experience is improved.
Patent Information
- Application Number
- CN202411177326.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, there are difficulties in editing and inputting mathematical formulas, especially the code method is complex, the WYSIWYG method is inefficient, the handwriting method is not accurate in recognition, and the lack of dedicated input methods for mathematical editing, resulting in low user input efficiency. .
Using a combination method based on symbolic markup language (SML) and pictogram input method, the symbolic pictogram parsing process is performed by listening to the characters input by the user, and the input is converted into LaTeX format using the recursive algorithm, and the input is rendered and displayed in real time through the MathJax library to provide a user Friendly formula editor SymbolML.
It realizes efficient mathematical formula editing, reduces learning costs and redundancy, improves input efficiency, simplifies the formula editing process, and enables users to enter and edit mathematical formulas quickly and conveniently.
Smart Images

Figure CN119089868A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer security technology, and in particular to a formula editing and generating method and system based on symbolic marking and pictographic input. Background Art
[0002] In scientific research and teaching, how to input formulas quickly and conveniently has always been a difficult problem that people have been trying to solve. Since mathematical expressions are basically two-dimensional (2D) structures composed of symbols, they are usually nested structures, in which complex formulas are generally composed of multiple smaller formulas. As stated in the W3C standard, "a notable feature of mathematics is the use of a complex and highly evolved two-dimensional symbol annotation system." Therefore, this complexity makes it often difficult to input mathematical formulas.
[0003] Currently, the editing and input methods of mathematical formulas have great limitations, making it difficult for users to input and edit, difficult to master, and low work efficiency.
[0004] There are relatively few studies on formula editing methods. Mathematical formula editing methods can generally be summarized into three types: code method, WYSIWYG method and handwriting method.
[0005] Code methods include mathematical computing tools (such as SymPy, Mathematica, and Matlab), as well as formats that can be used for formal representation (such as MathML, UnicodeMath, AsciiMath, and LaTeX). These formats usually support mutual conversion, enhancing their versatility. Some simple SymPy expressions can be converted to LaTeX format, and can also be further converted to MathML. In addition, for online LaTeX editors, Overleaf and Texpage provide accessible platforms for LaTeX editing, making it easier for users to effectively edit and manage LaTeX code.
[0006] The WYSIWYG approach includes tools such as Microsoft Word and MathType, which provide intuitive interfaces. In addition, Lyx, also described as a WYSIWYM approach, and similar tools such as MathQuill and Mathlive, where users edit displayed code in a WYSIWYG format.
[0007] To put it simply, the handwriting method refers to the editing method of operating with a mouse on a computer. This method has gradually emerged with the integration of machine learning technology, and handwriting recognition of formulas has made great progress. Online tools such as Mathpix and SimpleTex and large language models such as ChatGPT can better recognize handwritten mathematical expressions.
[0008] A large number of studies have shown that people prefer to use LaTeX to edit formulas compared to Microsoft Word, and found that when there are many formulas in the manuscript, many people prefer to use LaTeX. Microsoft Word can convert formula templates into LaTeX code, or directly convert LaTeX code into WYSIWYG form. Latex has always been the most famous for writing mathematical formulas in scientific and technological documents.
[0009] Recently, most of the research on LaTeX has focused on enhancing understanding, expression, and usability. Diaz et al. (2021) introduced a formula editor MathDeck that integrates LaTeX with structure editing and search functions to facilitate interactive input of formulas; Gobert et al. (2022) developed an editor i-LaTeX with conversion representation that simplifies the operation of formulas, tables, images, and grid layouts; Halyna M. Hubal (2023) enhanced the expressiveness of mathematical and scientific texts and expanded the functionality of LaTeX by creating new macro commands to add references and footnotes. In addition, various LaTeX auxiliary tools have been developed to improve the user experience. Tools such as SyncTeX and LaTeXTools allow users to hover the mouse over the code to preview formulas and images. Plug-ins such as LaTeX Suite and Quick LaTeX on the Obsidian platform provide shortcuts to help input, thereby simplifying the LaTeX editing process.
[0010] Although the above three editing methods are all being used by users, each method has great limitations. The code method, exemplified by the well-known LaTeX, has extremely complex syntax, is difficult to learn, and requires many rules to be memorized, resulting in low efficiency in inputting formulas. The WYSIWYG method requires constant selection of templates and clicking, which reduces the user's input efficiency and does not support copying and pasting. Although the handwriting method is more intuitive, it has problems such as low recognition accuracy and many inconveniences when using a mouse to input formulas on a computer.
[0011] There is relatively little research on input methods. Huang et al. (2018) proposed a customizable Chinese pinyin input method assisted by a neural network, aiming to improve input efficiency. Similarly, Moirangthem et al. (2021) introduced an input method editor designed specifically for Manipuri, which is challenging to input using a standard QWERTY keyboard.
[0012] In addition, the mainstream input methods we are familiar with, such as Baidu, Sogou, Google, Microsoft Input Method, and the customizable PIME input method. These methods all achieve effective input of languages from different regions, but they are not designed for mathematical editing. Therefore, the research on dedicated input methods for mathematical formulas has been slow. Summary of the invention
[0013] Based on this, the embodiment of the present application provides a formula editing and generating method and system based on symbolic marking and pictographic input, which can solve the problem of difficult editing and input of formulas and realize efficient mathematical formula editing.
[0014] In a first aspect, a formula editing and generating method based on symbolic marking and pictographic input is provided, the method comprising:
[0015] Monitor the characters entered by the user and send them to the backend;
[0016] The candidate word processor at the back end performs symbolic pictographic analysis and returns the candidate word list to the front end;
[0017] The user selects the recommended symbol and puts it into the input box. The front end sends the string in the input box to the back end.
[0018] The formula processor at the back end receives the string in the input box, processes it using a recursive algorithm based on the SML syntax, converts the result into LaTex format, and returns it to the front end by calling the API interface;
[0019] The front end uses the MathJax library for real-time rendering and display, presenting mathematical formulas completely and intuitively to users.
[0020] Optionally, listen for user input and send it to the backend, including:
[0021] Use the HTML input element to create an input box, and add an event listener through JavaScript to capture the user's input events. Whenever the user enters a character, the listener will capture the input content and trigger the callback function, sending the input character sequence to the backend server through AJAX.
[0022] Optionally, the candidate word processor at the back end performs symbolic pictographic analysis and returns a list of candidate words to the front end, including:
[0023] After the backend receives the character sequence request sent by the frontend, the candidate word processor starts working; the backend uses the Flask framework;
[0024] The processor uses the pictographic symbol combination analysis library to perform pictographic analysis of symbols according to the character sequence input by the user, and generates a list of candidate words. The list of candidate words is sent back to the front end through the API interface for the user to choose.
[0025] Optionally, the user selects a recommended symbol and puts it into the input box, and the front end sends the string in the input box to the back end, which also includes:
[0026] Use JavaScript to dynamically update the page to display candidate words. The candidate word list is displayed through HTML elements, and each candidate word is a clickable HTML element. When the user clicks on a candidate word, JavaScript is triggered to listen for the click event and fill the selected candidate word into the input box to complete the input process.
[0027] Optionally, a recursive algorithm is used based on the SML syntax to process the result, convert it into LaTex format, and return it to the front end by calling the API interface, including:
[0028] The backend receives the user input string sent by the frontend, initializes the parsing process, and prepares to recursively parse the input string;
[0029] SML uses preset symbols to define box structures to represent spatial structural changes in mathematical formulas. The recursive algorithm first identifies these box structures. For each identified box structure, the algorithm recursively enters the interior to identify and process the formula fragments inside.
[0030] Inside the box, the algorithm continues to identify formula fragments, process them according to the grammatical rules of SML, and convert the identified and processed formula fragments into unrelated texts, where the formula fragments include superscripts, subscripts, and functions;
[0031] Repeat the recognition process until all box structures have been processed, decompose the formula into basic mathematical symbols and structures, build the corresponding LaTeX structure according to the SML processing results, and return it to the front end by calling the API interface.
[0032] In a second aspect, a formula editing and generating system based on symbolic marking and pictographic input is provided, the system comprising:
[0033] The front-end module is used to listen to the characters input by the user and send them to the back-end module. After obtaining the candidate word list sent by the back-end module, the user selects the recommended symbol and puts it into the input box, and sends the string in the input box to the back-end module. The MathJax library is referenced for real-time rendering and display of the LaTex format results sent by the back-end module, presenting the mathematical formula to the user in a complete and intuitive manner.
[0034] The back-end module is used to perform symbolic pictographic analysis on the candidate words monitored by the front-end module and return the candidate word list to the front-end module; the back-end formula processor receives the string in the input box, processes it using a recursive algorithm based on the SML syntax, converts the result into LaTex format, and returns it to the front-end module by calling the API interface;
[0035] The data module is used to deploy the pictographic symbol combination parsing library, store data, and can be expanded and updated in real time.
[0036] Optionally, the front-end module monitors the characters input by the user and sends them to the back-end, including:
[0037] Use the HTML input element to create an input box, and add an event listener through JavaScript to capture the user's input events. Whenever the user enters a character, the listener will capture the input content and trigger the callback function, sending the input character sequence to the backend server through AJAX.
[0038] Optionally, the candidate word processor of the back-end module performs symbolic pictographic analysis and returns a candidate word list to the front-end module, including:
[0039] After the backend receives the character sequence request sent by the frontend, the candidate word processor starts working; the backend uses the Flask framework;
[0040] The processor uses the pictographic symbol combination analysis library to perform pictographic analysis of symbols according to the character sequence input by the user, and generates a list of candidate words. The list of candidate words is sent back to the front end through the API interface for the user to choose.
[0041] Optionally, the user selects a recommended symbol and puts it into the input box, and the front end sends the string in the input box to the back end module, further comprising:
[0042] Use JavaScript to dynamically update the page to display candidate words. The candidate word list is displayed through HTML elements, and each candidate word is a clickable HTML element. When the user clicks on a candidate word, JavaScript is triggered to listen for the click event and fill the selected candidate word into the input box to complete the input process.
[0043] Optionally, a recursive algorithm is used based on the SML syntax to process the result, convert it into LaTex format, and return it to the front-end module by calling the API interface, including:
[0044] The backend receives the user input string sent by the frontend, initializes the parsing process, and prepares to recursively parse the input string;
[0045] SML uses preset symbols to define box structures to represent spatial structural changes in mathematical formulas. The recursive algorithm first identifies these box structures. For each identified box structure, the algorithm recursively enters the interior to identify and process the formula fragments inside.
[0046] Inside the box, the algorithm continues to identify formula fragments, process them according to the grammatical rules of SML, and convert the identified and processed formula fragments into unrelated texts, where the formula fragments include superscripts, subscripts, and functions;
[0047] Repeat the recognition process until all box structures have been processed, decompose the formula into basic mathematical symbols and structures, build the corresponding LaTeX structure according to the SML processing results, and return it to the front end by calling the API interface.
[0048] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:
[0049] (1) Creation of the Symbolic Markup Language (SML): A novel symbolic markup language (SML) was introduced, which aims to standardize mathematical formulas into a symbolic one-dimensional linear representation.
[0050] (2) Developed a symbolic pictographic input method: An innovative symbolic pictographic input method was developed, which allows for more efficient input of mathematical symbols without the need for query.
[0051] (3) Created an accessible online formula editor SymbolML: We built a user-friendly website that allows direct access and use of the Web to efficiently write formulas in the formula editor SymbolML. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.
[0053] Figure 1 A flowchart of the steps of a formula editing and generating method based on symbolic marking and pictographic input provided in an embodiment of the present application;
[0054] Figure 2 A schematic diagram of the user interface of the formula editor SymbolML provided in an embodiment of the present application;
[0055] Figure 3 A schematic diagram of recursive processing of a program provided in an embodiment of the present application;
[0056] Figure 4 A schematic diagram of different meanings of brackets after decomposition provided in the embodiments of the present application;
[0057] Figure 5 A frequency distribution diagram of subscript symbols provided in the embodiments of the present application;
[0058] Figure 6 A recommendation mechanism for the pictographic input method provided in the embodiment of the present application;
[0059] Figure 7 A schematic diagram of a comparative experiment of the operating steps provided in the embodiments of the present application. DETAILED DESCRIPTION
[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0061] In the description of the present invention, the terms "comprises", "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units added based on further optimization schemes conceived by the present invention.
[0062] In order to solve the two major problems of difficult editing and inputting of formulas, this application creates a brand new math formula editor, named SymbolML, to achieve efficient math formula editing. Specifically, first, a new symbol markup language (SML) is defined and created; then, a symbol pictographic input method tailored for SML is developed, which can input symbols intuitively and conveniently without understanding their exact meaning. A large number of quantitative and qualitative evaluation experimental results show that SymbolML has lower redundancy and lower learning cost, and can complete editing tasks such as inserting, copying and pasting formulas at will, thereby making work more efficient. The test version of SymbolML has now been completed.
[0063] Please refer to Figure 1 , which shows a flow chart of a formula editing and generating method based on symbolic marking and pictographic input provided by an embodiment of the present application, which may include the following steps:
[0064] S1, monitors the characters input by the user and sends them to the backend.
[0065] In this step, use the HTML input element to create an input box, and add an event listener through JavaScript to capture the user's input events. Whenever the user enters a character, the listener captures the input content and triggers the callback function, sending the input character sequence to the backend server through AJAX.
[0066] S2, the candidate word processor at the back end performs symbolic pictographic analysis and returns the candidate word list to the front end.
[0067] In this step, after the backend receives the character sequence request sent by the frontend, the candidate word processor starts working; the backend uses the Flask framework;
[0068] The processor uses the pictographic symbol combination analysis library to perform pictographic analysis of symbols according to the character sequence input by the user, and generates a list of candidate words. The list of candidate words is sent back to the front end through the API interface for the user to choose.
[0069] S3, the user selects the recommended symbol and puts it into the input box, and the front end sends the string in the input box to the back end.
[0070] In this step, JavaScript is used to dynamically update the page to display candidate words. The candidate word list is displayed through HTML elements, and each candidate word is a clickable HTML element. When the user clicks on a candidate word, JavaScript is triggered to listen for the click event and fill the selected candidate word into the input box to complete the input process.
[0071] S4, the formula processor at the back end receives the string in the input box, processes it using a recursive algorithm based on the SML syntax, converts the result into LaTex format, and returns it to the front end by calling the API interface.
[0072] The backend receives the user input string sent by the frontend, initializes the parsing process, and prepares to recursively parse the input string;
[0073] SML uses preset symbols to define box structures to represent spatial structural changes in mathematical formulas. The recursive algorithm first identifies these box structures. For each identified box structure, the algorithm recursively enters the interior to identify and process the formula fragments inside.
[0074] Inside the box, the algorithm continues to identify formula fragments, process them according to the grammatical rules of SML, and convert the identified and processed formula fragments into unrelated texts, where the formula fragments include superscripts, subscripts, and functions;
[0075] Repeat the recognition process until all box structures have been processed, decompose the formula into basic mathematical symbols and structures, build the corresponding LaTeX structure according to the SML processing results, and return it to the front end by calling the API interface.
[0076] S5, the front end uses the MathJax library for real-time rendering and display, presenting mathematical formulas completely and intuitively to users.
[0077] like Figure 2 As shown in the figure, the web user interface of the formula editor SymbolML is shown. The interface contains 6 areas: ① input box area, ② symbol input method, ③ custom subscript change rules, ④ real-time formula display area, ⑤ button for copying LaTex code, and ⑥ quick start guide.
[0078] The formula editor proposed in this application is a new type of editor created by integrating markup language with input method. First, the input method captures the user's input and recommends candidate symbols. Then, the input content is sent to the server as a string. The server processes this string and returns it to the front end in LaTeX format. Finally, the MathJax library renders and presents the formula in real time. It is worth noting that modern browsers, web pages, and code compilers support Unicode encoding, providing a solid scientific research foundation.
[0079] (a) Defined and created a new symbolic markup language (SML).
[0080] The Symbolic Markup Language (SML) was created to simplify the process of entering mathematical formulas into computers. Traditional markup languages use "tags" as special symbols to identify different contents or structures in text. For example, in HTML, tags define elements such as headings () and paragraphs (), which help to distinguish different parts of the text. Compared with them, our Symbolic Markup Language (SML) has a significant advantage: it does not require a large number of tags like MathML, but the symbols themselves act as tags, thereby reducing the cost of users to remember or search for complex code symbols. Therefore, as long as users master basic mathematical concepts and symbols, they can easily edit computer-interpretable formulas.<h1>
[0081] (1) Linear representation of spatial structure. Mathematical expressions are essentially two-dimensional (2D) structures, which poses a challenge to direct input and editing on a computer. To address this problem, we proposed a linear representation of the 2D spatial structure that is characterized by unification and standardization. It is guided by two core goals. The first is to reduce the number of symbols used. Too many symbols will confuse readers and may even lead to ambiguity in the understanding of the formula. The second is to strive to minimize the diversity of symbols used. Use unified and simple symbols to type formulas instead of frequently changing symbols in different formulas.
[0082] Based on these two goals, this application proposes a linear representation that uses backslash as a separator for vertical structures (upper and lower parts) and uses spaces to distinguish horizontal structures (left and right sides). This representation is concise and intuitive, and greatly reduces the redundancy of symbols. Table 1 shows the spatial structure of the formula.
[0083] Table 1 Formula space structure table
[0084]
[0085] (2) The grammatical rules of the symbolic markup language (SML). Our design principle is to be as concise as possible to minimize the user's learning curve. Input follows from left to right, from the outermost element to the innermost element. The three core components of the symbolic markup language (SML) are as follows:
[0086] Box structure []. Drawing on the tags (double tags or single tags) in HTML, we introduced the concept of "box structure" in the symbolic markup language (SML). If the spatial structure of a part of the formula (such as a function) changes, a "box" is needed to encapsulate this part of the content, that is, it is indicated by using "[" as the start tag and "]" as the end tag, similar to the tag in HTML. This method ensures that changes in the spatial structure are clearly depicted and easy to explain. On the contrary, if the formula is a pure linear structure, there is no need to use a box, for example: sin(α+β)=sinαcosβ+cosαsinβ.
[0087] Function. In symbolic markup language (SML), each box contains at most one function, for example, "∑x", "∏x", and "A=πr 2 "; However, there are many cases where formulas with common spatial structures may not necessarily be completed using functions. This design ensures that the language is flexible and concise, and meets various structural requirements and is clear in expression.
[0088] Keyword table. In the symbolic markup language (SML), the keywords such as character (string) display form, upper and lower structure, box structure, matrix structure, and distinguishing empty brackets are scientifically defined and explained. Table 2 lists the keywords in SML.
[0089] Table 2 Keyword table
[0090]
[0091] (3) Program execution process. Complex mathematical formulas are composed of basic fragments and simple fragments [8]. Taking advantage of this feature, the symbolic markup language (SML) uses a recursive method to process the input string, starting from the smallest, innermost formula. First, the innermost formula is identified and converted into unrelated text after processing. This step can be understood as "peeling off" the innermost formula, isolating it from the original complex formula. The program then repeats this "peeling off" process until there are no more "boxes" in the string (that is, there are structured formulas that need to be processed). For example Figure 3 , which illustrates this treatment of the example formula.
[0092] (4) Bracket ([]) processing. In this process, brackets ("[]") can be used not only as common symbols, but also as keywords to define the formula hierarchy, such as Figure 4 As shown in the figure, the left and right brackets [] represent ordinary symbols, while the brackets near the middle represent keywords. Symbolic Markup Language (SML) ensures structural consistency and clarity by having an equal number of left brackets "[" and brackets "]".
[0093] (5) Shortcuts for typing subscripts. Through the analysis of the data set, the usage rate of subscript symbols is very high, with an average of about 3.4 per sentence. Figure 5 As shown in Figure 1, the frequency distribution of subscript symbols is given, among which the frequency of numeric subscripts is far ahead of others. In Symbolic Markup Language (SML), for the convenience of input, we interpret the combination of ordinary characters and numbers as subscript structures by default. In addition, subscript shortcuts are editable, allowing customization according to user preferences.
[0094] (2) Innovated and developed symbolic pictographic input method.
[0095] Through the investigation and statistics of a large number of scientific and technological documents, there are more than 1,500 mathematical symbols, many of which have very similar characters. This application innovates and develops an intelligent symbol input method, namely, a symbol pictographic input method, which is named by using a method similar to pictographic characters to quickly input mathematical formulas.
[0096] Through the pictographic analysis of mathematical symbols, a pictographic symbol combination analysis library is constructed.
[0097] We analyzed the mathematical symbols that appeared in nearly 100,000 lines of formulas in the public dataset img2LaTex, extracted more than 200 commonly used mathematical symbols, and used specific image recognition methods to perform pictographic analysis. There are three ways to analyze:
[0098] ① Pictographic independent parsing: According to the maximum pictographic rule, a mathematical symbol is parsed into a similar and independent keyboard symbol. For example, the symbol ∑ is parsed into E, M, W, 3 on the keyboard.
[0099] ② Pictographic combination analysis: Follow the pictographic principle and analyze mathematical symbols into a combination of multiple symbols. There may be many types of such combinations, such as the symbol The symbol can be parsed into one combination: O, / , the symbol ∑ can be parsed into two different combinations: << or ->-, and the symbol Δ can be parsed into three different combinations: / > or <\ or ^_.
[0100] ③ Pictographic minimalist analysis: Symbol analysis is performed based on the principle of minimum pictogram. For example, the ∑ symbol is analyzed as four symbols: -\ / -.
[0101] Each mathematical symbol is parsed in accordance with these three methods in turn, thereby completing the construction of the pictographic symbol combination parsing library. The result of each parsing is stored in the pictographic symbol combination parsing library in a list format: ["symbol", "part 1", x1, "part 2", x2, ..., "partn", xn]. Among them, "symbol" represents the mathematical symbol to be parsed, and "part 1", "part 2", etc. represent the individual partial symbols in the combined symbol. The numbers x1, x2, etc. after each part represent the proportion of each symbol in its combined symbol (referred to as similarity). The system defaults to equal proportions, but supports users to make customized adjustments based on long-term accumulated experience. The biggest advantage of this pictographic parsing method is that it allows users to easily enter mathematical symbols in the same way as entering text.
[0102] Symbol extensibility of the input method. Considering the continuous emergence of new symbols, the symbolic pictographic input method must support the rapid integration of these new symbols. Since the symbols are constructed in the form of a pictographic symbol combination parsing library, they have rigorous structural characteristics, thus ensuring that new symbols can be flexibly and efficiently integrated into the pictographic symbol combination parsing library. When adding new mathematical symbols, as long as the pictographic parsing is carried out in the above three ways in sequence through a specific image recognition method, the parsing results can be quickly added to the library, realizing the immediate expansion of symbols.
[0103] (3) Use artificial intelligence technology to disassemble symbol composition.
[0104] 3.1 In this example, through the convolutional neural network, a simple symbol that is most similar to the complex symbol image to be detected can be identified, and based on this, the recursive execution is continuously performed to obtain a simple symbol set of the complex symbol.
[0105] 3.2 Before applying neural network processing, data collection, data labeling and preprocessing are required. The specific steps are as follows:
[0106] Preparation of basic symbol dataset: The dataset should contain all basic symbols on the keyboard, such as letters (e.g. A, a, O, etc.) and other common symbols (e.g. / , |, -, <, etc.). These symbols should be generated directly from their Unicode encoding in a fixed format to ensure their clarity and consistency in image processing.
[0107] Preparation of mathematical symbol dataset: The dataset should cover various complex mathematical symbols, such as set symbols. Summation symbol (∑), etc.
[0108] Data annotation: Annotate each symbol image, including the simple symbols that the symbol is composed of and its position in the image.
[0109] Preprocessing: Convert each symbol into a standardized 32×32 pixel image and ensure that they are centered. This step is crucial because it ensures that all symbols have a uniform size and alignment when performing similarity matching, thereby improving the accuracy of the matching. At the same time, data augmentation (rotation, translation, scaling, etc.) is performed appropriately to increase the diversity of training data and improve the robustness of the model. Finally, the dataset is divided into training set, validation set and test set, with a ratio of 70% (training), 15% (validation) and 15% (testing).
[0110] 3.3 Core part of model design:
[0111] Choice of loss function: Select mean squared error loss to measure the difference between model predictions and actual labels and guide model training.
[0112] Selection of optimization algorithm: Select Adam optimizer to update model parameters and improve training efficiency.
[0113] Recursive iteration: Repeat the image matching and subtraction steps until the content of the remaining image cannot be matched to any simple symbol, or very little content remains. This process is iterative, and each iteration further refines the structure of the complex symbol until the basic components are reached. In each iteration, the similarity of the remaining image to the basic symbol needs to be re-evaluated.
[0114] Record and verify the combination results: record the simple symbol matching at each step, and finally combine the complete decomposition results of the complex symbol, and ensure that the proportion of each simple symbol adds up to 1. This step is the key to verifying the accuracy of symbol decomposition to ensure the completeness and accuracy of the decomposition results. During the verification process, the consistency and reliability of the decomposition results can be evaluated through methods such as cross-validation.
[0115] 3.4 During the training process, a large number of basic symbols and mathematical symbol image pairs are needed, and the model parameters are continuously optimized through supervised learning until a high matching accuracy is achieved. Specifically: train the model with training data, evaluate it on validation data, and evaluate the model performance with test data. According to the performance of the model on the validation set and test set, adjust the model structure and hyperparameters to further improve the performance. Save the trained model for subsequent use. Load the trained model, predict the new symbol image, and output a simple set of recognized symbols based on the model.
[0116] Recommendation mechanism of input method. In order to allow users to input mathematical symbols quickly and conveniently, there are two problems that need to be solved: First, when users input symbols through the keyboard, which mathematical symbols are recommended? Second, in what order are the recommended mathematical symbols presented to users? Figure 6 shown.
[0117] ① When the user inputs the character a, the system input method quickly searches for all symbol combinations containing the character a in the pictographic symbol combination parsing library, forming a recommendation set S = {A1, A2, ..., An} containing n symbol combinations. The mathematical symbols corresponding to these n symbol combinations A1, A2, ..., An are all candidate symbols to be recommended. To facilitate rapid processing by the system, the concept of similarity in (1) is introduced. The shorter the length of the parsed symbol combination Ai (i = 1, 2, ..., n) containing the character 'a', the greater the similarity corresponding to the character a. It can be inferred that if a symbol combination Ak contains the character a, the mathematical symbol corresponding to the symbol combination Ak is recommended, otherwise it is not recommended.
[0118] ② The recommendation process is dynamic. As the user continues to input, the number of elements in the recommended set S will inevitably decrease (or remain unchanged). When the user inputs character b, because the symbol combination A2 does not contain character b, that is, the similarity of b in A2 is zero, the symbol combination A2 with zero similarity is deleted from the set S. At this time, the number of elements in the set S is n-1, and the recommended candidate symbols are also reduced by one. Repeat the input operation and repeatedly filter the recommended set S.
[0119] ③Through ① and ② above, the recommended set S is continuously and dynamically screened. For example, after t times of screening (input characters are a1, a2, ..., at in sequence), the recommended set S = {B1, B2, ..., Bm} satisfies the following conditions: it contains m symbol combinations, each of which contains at least t characters a1, a2, ..., at, and the parsing lengths of the symbol combinations are sorted from low to high (some of them have the same parsing length), assuming that the parsing lengths Len(B1) = Len(B2) = Len(B3) <Len(B4)<...=...<Len(Bm)。 Figure 5 That is the case when t = 2. Figure 5 It can be seen that Len(A1)=2, Len(An)=3. At this time, the total similarities of characters a and b are k1=100 and kn=66.6 respectively. Their parsed lengths are equal or similar (in other words, their total similarity is equal or similar). In what order should their corresponding mathematical symbols be recommended? In order to facilitate system processing, we studied the mathematical symbols that appeared in nearly 100,000 lines of formulas in the public dataset img2LaTex, analyzed and found the usage rules of mathematical symbols. On this basis, combined with the total similarity, through the recommendation quantitative evaluation, we gave their recommendation quantitative scores respectively. Finally, according to the order of scores from high to low, the corresponding mathematical symbols are recommended. If the scores are the same, any order is acceptable.
[0120] (iv) Created SymbolML, an online formula editor accessible in a laboratory environment.
[0121] The present invention aims to provide an online formula editor SymbolML to realize the construction of an online symbol pictographic input method system, realize the text input of mathematical symbols by combining pictographic symbols, and provide an efficient and convenient user input experience.
[0122] In the laboratory, a friendly website was built and a server was developed and deployed. Users can access and use the Web through a browser to easily and conveniently experience the formula editor SymbolML and achieve efficient input and editing of formulas.
[0123] System architecture: The online symbol pictographic input method system involved in the present invention includes three parts: front-end module, back-end module and data module.
[0124] ① Front-end module: mainly responsible for processing the user interface and input events. It is used to capture the character sequence input by the user in real time and send it to the back-end candidate word processor. At the same time, it updates the candidate word list returned by the back-end in real time and presents it to the user. On the other hand, it receives the LaTex string processed by the back-end formula processor in real time, and references the MathJax library for rendering, presenting the mathematical formula to the user in a complete and intuitive manner.
[0125] ②Backend module: mainly includes candidate word processor and formula processor, whose function is to call the input method engine and API interface. The candidate word processor is used to process the characters input by the user in real time, and return the candidate words to the front end after processing; the formula processor is responsible for processing the characters in the input box of the front-end interface, converting them into LaTex format through core processing and returning them to the front end.
[0126] ③Data module: A pictographic symbol combination parsing library is deployed to store data, which can be expanded and updated in real time.
[0127] Inventions and creations cannot be separated from a large number of experiments. This is an eternal truth summed up by many scientific researchers through practice. Of course, this invention is also achieved on the basis of a large number of experiments. Our experiments involve the following three main elements:
[0128] Dataset. Formulas were randomly selected from the im2LaTex_formulas dataset (102,863 items), and they were divided into four groups according to the number of backslashes: short, medium, long, and extra long. Ten formulas from each group were randomly selected for the experiment.
[0129] Tools. Use Python to develop the server, use the pinyinjs framework to implement the symbol pictogram online input method, and use the JavaScript MathJax library to render mathematical formulas.
[0130] Evaluation indicators. The evaluation includes quantitative evaluation and qualitative evaluation. In terms of quantitative evaluation, we evaluate redundancy, which is the ratio of input characters to characters in the original formula. The lower the ratio, the higher the efficiency. In terms of qualitative evaluation, we evaluate the difficulty of inputting formulas, taking into account factors such as learning cost and ease of use. The main goals of the specific evaluation are the convenience, efficiency and user-friendliness of the input method.
[0131] Under the guidance of the above three elements, the present invention has completed a large number of experiments and achieved certain experimental results. Compared with the prior art and the code method, its advantages are mainly as follows:
[0132] (1) Lower redundancy:
[0133] SML is experimentally compared with LaTeX and MathM. Considering the subjective nature of input speed, we use a quantitative method to compare the three. Table 3 shows the ratio comparison results of the three, which shows that SML has a lower ratio and less redundancy in formulas.
[0134] Table 3 Ratio comparison test table
[0135]
[0136] (2) More unified and simplified:
[0137] Comparison with UnicodeMath and AsciiMath. UnicodeMath is not widely used and is mainly used in Word. Although AsciiMath is similar to UnicodeMath in many aspects, its usage is also limited. Nevertheless, they have made some contributions to the advancement of mathematical symbol input. On this basis, our SymbolML has been further optimized to provide a more comprehensive way to use symbols, and users no longer need to look up tables. In addition, compared with other markup languages, its syntax is simpler and more unified. As shown in the syntax comparison experiment in Table 4, SML syntax is more concise.
[0138] Table 4 Syntax comparison experiment table
[0139]
[0140] (3) Input character length is shorter:
[0141] Comparison with different mathematical calculation methods. Since most mathematical software does not support font formats, this comparison only focuses on string length, excluding font formats. As shown in Table 5, the original formula is compared with inputs of different methods. The experimental results show that the input string length of SML is shorter and has obvious advantages.
[0142] Table 5 Input character length comparison experiment table
[0143]
[0144] Comparison with WYSIWYG methods. WYSIWYG is a widely used method for intuitively inputting mathematical formulas. However, it usually involves multiple steps, such as "click input position - find insert element - click element", which can be cumbersome, especially for complex or unfamiliar formulas. In contrast, SymbolML provides users with a simpler interface, eliminating the need for continuous searching and mouse operations, and only requires simple input steps.
[0145] For a given formula, such as Experiments were conducted using both SymbolML and Microsoft Word, such as Figure 7The experimental diagram of the operation steps comparison is shown in the figure, which lists the specific operation steps of the two. The experimental results show that the total number of operation steps of the two is the same, but Microsoft Word needs to frequently interact and switch with different areas of the interface, wasting a lot of invalid time, while SymbolML only involves direct input of symbols. Through comparison, we can easily conclude that SymbolML is more efficient in input compared with the traditional WYSIWYG method.
[0146] Comparison with handwritten methods. Although handwritten formulas seem convenient, they pose several significant challenges. First, the accuracy of recognition is problematic. Second, many symbols are difficult to write by hand. Furthermore, in handwriting, it is particularly challenging to replicate typographical styles such as bold or floral fonts.
[0147] In addition, additional time and operation steps are required to recognize handwritten formulas, which affects efficiency. In particular, handwritten formulas are not easy to copy and paste, which should be considered for frequent reuse of formulas. In our digital information age, paperless office is becoming a growing trend, and our symbolic markup language (SML) is very consistent with these developments, providing a more efficient and adaptable solution.
[0148] In summary, multiple comparative experiments show that the formula editor SymbolML created by this application based on symbol markup language (SML) and symbol pictographic input method improves user input speed, simplifies the process, shortens the user learning curve, and greatly improves work efficiency. The online network test version has been completed and is available for user testing and use.
[0149] The embodiment of the present application provides a block diagram of a formula editing and generating system based on symbolic marking and pictographic input, including:
[0150] The front-end module is used to listen to the characters input by the user and send them to the back-end module. After obtaining the candidate word list sent by the back-end module, the user selects the recommended symbol and puts it into the input box, and sends the string in the input box to the back-end module. The MathJax library is referenced for real-time rendering and display of the LaTex format results sent by the back-end module, presenting the mathematical formula to the user in a complete and intuitive manner.
[0151] The back-end module is used to perform symbolic pictographic analysis on the candidate words monitored by the front-end module and return the candidate word list to the front-end module; the back-end formula processor receives the string in the input box, processes it using a recursive algorithm based on the SML syntax, converts the result into LaTex format, and returns it to the front-end module by calling the API interface;
[0152] The data module is used to deploy the pictographic symbol combination parsing library, store data, and can be expanded and updated in real time.
[0153] Specifically, the front-end module is implemented by HTML, CSS and JavaScript. The main functions are as follows:
[0154] Input box: A symbol used to receive user input. The input box is displayed through HTML <input> Elements are implemented and CSS is used for style beautification to give the input box a good user experience.
[0155] Event listener: JavaScript implements event listening, which is used to listen to input events of the input box (such as input events). Whenever the user enters a symbol, the event listener will capture the input content and trigger the callback function, using AJAX to send the characters entered by the user to the backend server.
[0156] Dynamically display the candidate word list: After the front end receives the candidate word list returned by the back end, it uses JavaScript to dynamically update the page to display the candidate words. The candidate word list is displayed through HTML elements, and each candidate word is a clickable HTML element; when the user clicks on a candidate word, JavaScript is triggered to listen for the click event and fill the selected candidate word into the input box to complete the input process.
[0157] Rendering and display: Use the MathJax library for real-time rendering and display, presenting mathematical formulas to users in a complete and intuitive manner.
[0158] The backend module uses the web framework Flask and is implemented using the Python programming language. The main functions are as follows:
[0159] Receive input requests sent by the front end: The back-end server receives user input requests sent by the front end through the API interface, and parses the request content to obtain the symbols entered by the user.
[0160] Candidate word processor: The back-end server receives the symbols input by the user, and sends them to the candidate word processor for pictographic analysis to obtain a list of candidate words and return it to the front-end.
[0161] Formula processor: The backend server captures the string in the input box, passes it to the formula processor for layer-by-layer processing, and converts it into LaTex format, which is then returned to the frontend by calling the API interface.
[0162] For the specific limitations of the formula editing and generating system, please refer to the limitations on the formula editing and generating based on symbolic marking and pictographic input above, which will not be repeated here. Each module in the above-mentioned formula editing and generating system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0163] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0164] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.
Claims
1. A formula editing and generation method based on symbolic marking and pictographic input, It is characterized in that The method comprises: Monitor the characters entered by the user and send them to the backend; The candidate word processor at the back end performs symbolic pictographic analysis and returns the candidate word list to the front end; The user selects the recommended symbol and puts it into the input box. The front end sends the string in the input box to the back end. The formula processor at the back end receives the string in the input box and processes it using a recursive algorithm based on the SML syntax. It converts the result into LaTex format and returns it to the front end by calling the API interface. The front end uses the MathJax library for real-time rendering and display, presenting mathematical formulas completely and intuitively to users.
2. The formula editing and generating method according to claim 1, It is characterized in that Listen for characters entered by the user and send them to the backend, including: Use the HTML input element to create an input box, and add an event listener through JavaScript to capture the user's input events. Whenever the user enters a character, the listener will capture the input content and trigger the callback function, sending the input character sequence to the backend server through AJAX.
3. The formula editing and generating method according to claim 1, It is characterized in that The candidate word processor at the back end performs symbolic pictographic analysis and returns a list of candidate words to the front end, including: After the backend receives the character sequence request sent by the frontend, the candidate word processor starts working; the backend uses the Flask framework; The processor uses the pictographic symbol combination analysis library to perform pictographic analysis of symbols according to the character sequence input by the user, and generates a list of candidate words. The list of candidate words is sent back to the front end through the API interface for the user to choose.
4. The formula editing and generating method according to claim 1, It is characterized in that The user selects the recommended symbol and puts it into the input box. The front end sends the string in the input box to the back end, which also includes: Use JavaScript to dynamically update the page to display candidate words. The candidate word list is displayed through HTML elements, and each candidate word is a clickable HTML element. When the user clicks on a candidate word, JavaScript is triggered to listen for the click event and fill the selected candidate word into the input box to complete the input process.
5. The formula editing and generating method according to claim 1, It is characterized in that Based on the SML syntax, a recursive algorithm is used to process the result, convert it into LaTex format, and return it to the front end by calling the API interface, including: The backend receives the user input string sent by the frontend, initializes the parsing process, and prepares to recursively parse the input string; SML uses preset symbols to define box structures to represent spatial structural changes in mathematical formulas. The recursive algorithm first identifies these box structures. For each identified box structure, the algorithm recursively enters the interior to identify and process the formula fragments inside. Inside the box, the algorithm continues to identify formula fragments, process them according to the grammatical rules of SML, and convert the identified and processed formula fragments into unrelated texts, where the formula fragments include superscripts, subscripts, and functions; Repeat the recognition process until all box structures have been processed, decompose the formula into basic mathematical symbols and structures, build the corresponding LaTeX structure according to the SML processing results, and return it to the front end by calling the API interface.
6. A formula editing and generation system based on symbolic marking and pictographic input, It is characterized in that The system comprises: The front-end module is used to listen to the characters input by the user and send them to the back-end module. After obtaining the candidate word list sent by the back-end module, the user selects the recommended symbol and puts it into the input box, and sends the string in the input box to the back-end module. The MathJax library is referenced for real-time rendering and display of the LaTex format results sent by the back-end module, presenting the mathematical formula to the user in a complete and intuitive manner. The back-end module is used to perform symbolic pictographic analysis on the candidate words monitored by the front-end module and return the candidate word list to the front-end module; the back-end formula processor receives the string in the input box, processes it using a recursive algorithm based on the SML syntax, converts the result into LaTex format, and returns it to the front-end module by calling the API interface; The data module is used to deploy the pictographic symbol combination parsing library, store data, and can be expanded and updated in real time.
7. The formula editing and generating system according to claim 6, It is characterized in that The front-end module monitors the characters entered by the user and sends them to the back-end, including: Use the HTML input element to create an input box, and add an event listener through JavaScript to capture the user's input events. Whenever the user enters a character, the listener will capture the input content and trigger the callback function, sending the input character sequence to the backend server through AJAX.
8. The formula editing and generating system according to claim 6, It is characterized in that The candidate word processor of the back-end module performs symbolic pictographic analysis and returns the candidate word list to the front-end module, including: After the backend receives the character sequence request sent by the frontend, the candidate word processor starts working; the backend uses the Flask framework; The processor uses the pictographic symbol combination analysis library to perform pictographic analysis of symbols according to the character sequence input by the user, and generates a list of candidate words. The list of candidate words is sent back to the front end through the API interface for the user to choose.
9. The formula editing and generating system according to claim 6, It is characterized in that The user selects the recommended symbol and puts it into the input box. The front end sends the string in the input box to the back end module, which also includes: Use JavaScript to dynamically update the page to display candidate words. The candidate word list is displayed through HTML elements, and each candidate word is a clickable HTML element. When the user clicks on a candidate word, JavaScript is triggered to listen for the click event and fill the selected candidate word into the input box to complete the input process.
10. The formula editing and generating system according to claim 6, It is characterized in that Based on the SML syntax, a recursive algorithm is used to process the result, convert it into LaTex format, and return it to the front-end module by calling the API interface, including: The backend receives the user input string sent by the frontend, initializes the parsing process, and prepares to recursively parse the input string; SML uses preset symbols to define box structures to represent spatial structural changes in mathematical formulas. The recursive algorithm first identifies these box structures. For each identified box structure, the algorithm recursively enters the interior to identify and process the formula fragments inside. Inside the box, the algorithm continues to identify formula fragments, process them according to the grammatical rules of SML, and convert the identified and processed formula fragments into unrelated texts, where the formula fragments include superscripts, subscripts, and functions; Repeat the recognition process until all box structures have been processed, decompose the formula into basic mathematical symbols and structures, build the corresponding LaTeX structure according to the SML processing results, and return it to the front end by calling the API interface.
