Pen type calculator system based on handwritten formula recognition

Abstract

The invention discloses a handwriting formula recognition-based handwriting calculator system. The handwriting formula recognition-based handwriting calculator system comprises an image acquisition and input module, a handwriting formula recognition module and a handwriting formula recognition module, the recognition module is used for recognizing the handwritten image and outputting the handwritten image as a latex expression; the interaction module is used for detecting whether an equal sign is contained in the latex expression or not, if yes, a calculation function is triggered, and if not, calculation is not carried out, only an identification function is triggered, and an identification result is sent to a front end to be displayed; the latex processing module is used for converting a latex expression into a Sympy expression after the calculation function is triggered; the Sympy calculation module is used for performing result calculation on the Sympy expression by utilizing an evalf function in the Sympy library and converting a result into a numerical value form; and the display module is used for displaying the calculation result in the numerical form on a local interface. Efficient handwriting input interaction is carried out through the mobile terminal, and the efficiency and experience of the user are remarkably improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image recognition and relates to a written test calculator system based on handwritten formula recognition. Background Art

[0002] With the popularity of educational tablets and mobile input devices, interactive input using digital stylus is more portable. Digital handwriting technology input is becoming more and more important and popular in K12 educational products such as tablet learning machines. Traditional calculators are all key input, and the function is determined by the key. It is inconvenient for complex calculation input. Pen calculators realize portable input, especially for integral calculation and trigonometric function calculation input. Summary of the invention

[0003] In order to solve the above technical problems existing in the prior art, the present invention proposes a written test calculator system based on handwritten formula recognition, which can simplify the input form of users using calculation functions, and realize calculation functions through input interaction through a mobile terminal. The specific technical solution is as follows: A written test calculator system based on handwritten formula recognition includes an image acquisition and input module, a recognition module, an interaction module, a latex processing module, a Sympy calculation module and a display module; The image acquisition and input module is used to acquire and process the handwritten formula image of the user's mobile terminal; Recognition module, used to recognize handwritten formula images and process and output them as latex expressions; The interactive module is used to detect whether there is an equal sign in the latex expression. If there is an equal sign, the calculation function is triggered. Otherwise, no calculation is performed, only the recognition function is triggered and the recognition result is sent back to the front-end for display; The latex processing module is used to convert latex expressions into Sympy expressions after triggering the calculation function; Sympy calculation module, used to use the evalf function in the Sympy library to calculate the results of Sympy expressions and convert the results into numerical form; The display module is used to display the calculation results in numerical form on the local interface.

[0004] Furthermore, the image acquisition and input module captures the user's handwriting trajectory in real time and generates a handwritten image, and then identifies the boundary of the handwritten formula through image processing technology, captures the image area containing the complete formula, and then compresses the captured image and transmits it to the cloud server or local software.

[0005] Furthermore, the recognition module includes an image preprocessing submodule and an artificial intelligence algorithm reasoning submodule. The image preprocessing submodule performs grayscale, binarization, and stroke corrosion processing on the intercepted handwritten formula image in sequence; the artificial intelligence algorithm reasoning submodule uses a Densenet network to extract features of the preprocessed image, and uses a pre-trained end-to-end algorithm to output the recognized latex sequence.

[0006] Furthermore, the interactive module detects whether the latex expression contains an equal sign. If an equal sign is recognized, the last equal sign is removed, and the formula part before the equal sign is recognized. Thereafter, the latex processing module and the Sympy calculation module perform calculations in sequence, and the display module displays the calculation results. If no equal sign is recognized, no calculation is performed, and the formula recognition continues and the recognition results are transmitted back to the front end for display.

[0007] Furthermore, the latex processing module first converts the latex expression into a Sympy expression using the latex2Sympy function in the latex2Sympy2 library, then uses the dictionary variances to replace the variables in the Sympy expression, and then uses the simplify function in the Sympy library to simplify the Sympy expression.

[0008] Furthermore, the Sympy calculation module formats the calculation results: integer results are rounded and decimal results are rounded to two decimal places.

[0009] Furthermore, the Sympy calculation module converts the angles expressed in degrees in the user's handwritten formula into radians, specifically including: Angle detection: Use the regular expression (\d+(\.\d+)?)\^\{?\\circ}? to detect the angle representation in the handwritten formula; Angle value extraction: Use the regular expression match.group to extract the angle value, and convert the extracted angle value into a floating point number; Conversion from degrees to radians: Use the formula: degrees = degrees / 180*π to convert the angle value from degrees to radians; Format as latex expression: Format the converted radian value as latex expression; Replace the original expression: Use the regular expression replacement method regex.sub to replace the angle system representation in the original expression with the converted radian system representation.

[0010] Furthermore, when the system is deployed on a cloud server, the display module transmits the calculation results back to the local interface via the http protocol for display; when the system is deployed locally, the calculation results are directly displayed on the local interface.

[0011] The present invention provides a complete set of system processes from image acquisition, formula recognition, LaTeX conversion, formula calculation to result display. The system can efficiently process mathematical formulas written by users and automatically perform symbolic and numerical calculations, thereby helping users to quickly and easily perform calculation input, simplify the input form, solve the cumbersome problem of complex form input, realize portable interaction, and provide a friendly user interaction experience. Its independent latex input calculation method is convenient for localized deployment, realizes offline use, and is convenient for large-scale deployment and use in multiple fields such as education and scientific research. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the handwritten formula recognition process when the system of the present invention is deployed on the server cloud; Figure 2 It is a handwritten formula recognition flow chart of the system of the present invention when it is deployed locally; Figure 3 It is a schematic diagram of the handwritten formula recognition effect of the system of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical scheme and technical effect of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0014] like Figures 1 to 3 As shown, this embodiment discloses a written test calculator system based on handwritten formula recognition, providing two solutions of server cloud deployment and local deployment, and the system includes an image acquisition and input module, a recognition module, an interaction module, a latex processing module, a Sympy calculation module and a display module. The image acquisition and input module is used to collect the handwritten image of the user's mobile terminal and send it to the cloud server or local software according to the deployment plan; the recognition module is used to recognize the handwritten image and output it as a latex expression; the interaction module is used to detect whether the latex expression contains an equal sign, if there is an equal sign, the calculation function is triggered, otherwise no calculation is performed, only the recognition function is triggered and the recognition result is sent to the front end for display; the latex processing module is used to convert the latex expression into a Sympy expression after the calculation function is triggered; the Sympy calculation module is used to use the evalf function in the Sympy library to calculate the result of the Sympy expression and convert the result into a numerical form; the display module is used to display the calculation result in numerical form on the local interface, which is convenient for users to view and operate later.

[0015] Specifically, the image acquisition and input module: the mobile device used by the user captures the image area of ​​the user's handwriting input and intercepts it as an image segment containing the complete formula, which is then uploaded to the cloud server for further processing. More specifically, user input: the user writes a mathematical formula on the handwriting input board of the mobile device, the device captures the handwriting trajectory in real time, and generates a handwriting image. Image interception: the device uses image processing technology to identify the boundaries of the handwritten formula and intercepts the image area containing the complete formula. Image upload: the intercepted image is compressed to reduce transmission time. The image is uploaded to the cloud server via the network for subsequent processing.

[0016] Recognition module: includes two sub-modules: image preprocessing and artificial intelligence algorithm reasoning.

[0017] The preprocessing submodule preprocesses the image of the handwritten formula, including: Grayscale: Convert a color image into a grayscale image to reduce the interference of color information. Use a grayscale algorithm to convert the color value of each pixel into a grayscale value. Binarization: Convert the grayscale image into a binary image to make the handwriting clearer. Use the threshold method to set the pixel values ​​greater than the threshold to white and the pixels less than the threshold to black. Stroke erosion: Perform stroke erosion operation on binary images to remove small noise points and improve the cleanliness of the image. Use morphological erosion algorithm to process the image to improve recognition accuracy.

[0018] The artificial intelligence algorithm reasoning submodule uses a Densenet network to extract image features and uses a pre-trained end-to-end algorithm to output the recognized latex sequence.

[0019] Interaction module: triggers the calculation function after detecting the user inputting an equal sign, and enters the recognition function without performing calculation if the equal sign is not written. Specifically, detect the user's input on the handwriting input board, remove the last equal sign if an equal sign is recognized, recognize the formula part before the equal sign, and then implement the calculation function through the latex processing module and the Sympy calculation module in turn, and the calculation result is displayed by the display module; if the equal sign is not recognized, continue to recognize the formula without performing calculation, and only output the recognition result to be displayed on the front end.

[0020] Latex processing module: Convert latex expressions into a format supported by the Sympy library. Specifically, use the latex2Sympy2 library to convert latex formulas into a format that can be used by Sympy, use the dictionary variances to replace variables in the expression, and use the parse_latex function to parse and load them into a format that can be processed by the Sympy library. More specifically, it includes: Latex parsing: Use the latex2Sympy function in the latex2Sympy2 library to convert latex expressions to Sympy expressions. Ensure the correctness and consistency of syntax during the conversion process.

[0021] Variable replacement: Use the dictionary variances to replace variables in expressions. Ensure the correctness of variable values ​​and avoid calculation errors.

[0022] Expression simplification: Use the simplify function in the Sympy library to simplify expressions. Ensure that the expression is in its simplest form before calculation.

[0023] Exception handling: Capture syntax errors and other exceptions that may occur during the conversion process, and provide user-friendly error prompts to help users correct their input.

[0024] Sympy calculation module: Use the evalf function in the Sympy library to calculate the result, execute all unexecuted operations until the simplest expression is obtained, use variances to replace variables and calculate the numerical result of the expression, including: Expression calculation: Use the evalf function in the Sympy library to calculate the numerical result of the expression. Ensure that all unexecuted operations such as integration and summation are executed. Result formatting: Format the calculation results to ensure the readability of the output results. Round integer results and retain two decimal places for decimal results. Result output: Convert the calculation results into numerical form for easy viewing and use by users, and provide a user interface to display the calculation results. Among them, including the calculation method for trigonometric function angle system. In mathematical calculations, many trigonometric function parameters are required to be in radians, while degrees may be used to represent angles in user-written formulas. In order to ensure the accuracy and consistency of the calculation, the present invention automatically converts the angle system into the radian system during the recognition and calculation process. The process is as follows: Angle detection: Use the regular expression (\d+(\.\d+)?)\^\{?\\circ}? to detect angle expressions in handwritten formulas. This regular expression matches angle expressions such as 30^\circ or 30°.

[0025] Angle value extraction: Use the regular expression match.group to extract the angle value, and convert the extracted angle value into a floating point number. For example, the angle value after 30^\circ extraction is 30.0.

[0026] Conversion from degrees to radians: Use the string formatting method f"{{{angle_in_degrees / 180.0}}}{{\\pi}}" to generate radian representation, that is, use the formula: degrees = degrees / 180*π to convert the angle value from degrees to radians. For example, the angle value 30.0 is converted to the radian value 30.0 / 180*π, which is 0.5236.

[0027] Format as latex expression: Format the converted radian value as a latex expression.

[0028] Replace the original expression: Use the regular expression replacement method regex.sub to replace the angle system in the original expression with the converted radian system. For example, replace 30^\circ with 0.5236\pi and then execute the evalf function calculation in the Sympy library.

[0029] Display module: If the inference is performed on the server, the calculation results are transmitted back to the local interface through the HTTP protocol for display, which is convenient for users to view and operate later. If deployed locally, you can continue to refer to Figure 2 , without the need for network transmission, the calculation results can be displayed directly on the local interface, providing clear visualization effects.

[0030] In summary, this system is designed to realize the user's simple interactive trigger function, without the need for the user to operate the button separately, and only needs to draw an equal sign to trigger this function; and realize the trigonometric function angle calculation function, support the angle input in the handwritten formula trigonometric function calculation; support the separate input of latex formula for calculation, in addition to cloud server reasoning, support local deployment. The system of the present invention obtains the result of the user's answer, takes a screenshot, uses the handwritten formula algorithm for recognition reasoning, outputs different interfaces for the judgment of the equal sign in the recognition result, returns the recognition result and the calculation result, realizes the user calculation function trigger, realizes the formula input from the mobile terminal handwriting pen to the calculation result, supports addition (+), subtraction (-), dot multiplication (·), multiplication (x), division ( / ), exponentiation (^), square root, common functions: logarithm (log), natural logarithm (ln), exponent (exp), angle system and radian system can be trigonometric functions and other functional operations, innovates the calculator input form, and realizes the calculation method based on the user's hand-drawn mobile terminal and digital pen input.

[0031] The above is only a preferred implementation case of the present invention and does not limit the present invention in any form. Although the implementation process of the present invention is described in detail above, for those familiar with the art, they can still modify the technical solutions recorded in the above examples, or replace some of the technical features therein with equivalents. All modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A written test calculator system based on handwritten formula recognition, characterized in that: It includes image acquisition and input module, recognition module, interaction module, latex processing module, Sympy calculation module and display module; The image acquisition and input module is used to acquire and process the handwritten formula image of the user's mobile terminal; Recognition module, used to recognize handwritten formula images and process and output them as latex expressions; The interactive module is used to detect whether there is an equal sign in the latex expression. If there is an equal sign, the calculation function is triggered. Otherwise, no calculation is performed, only the recognition function is triggered and the recognition result is sent back to the front-end for display; The latex processing module is used to convert latex expressions into Sympy expressions after triggering the calculation function; Sympy calculation module, used to use the evalf function in the Sympy library to calculate the results of Sympy expressions and convert the results into numerical form; The display module is used to display the calculation results in numerical form on the local interface.

2. The written test calculator system according to claim 1, characterized in that: The image acquisition and input module captures the user's handwriting trajectory in real time and generates a handwritten image, then identifies the boundary of the handwritten formula through image processing technology, captures the image area containing the complete formula, and then compresses the captured image and transmits it to the cloud server or local software.

3. The written test calculator system according to claim 2, characterized in that: The recognition module includes an image preprocessing submodule and an artificial intelligence algorithm reasoning submodule, and the image preprocessing submodule sequentially performs grayscale, binarization, and stroke corrosion processing on the intercepted handwritten formula image; The artificial intelligence algorithm reasoning submodule uses the Densenet network to extract the features of the preprocessed image and uses the pre-trained end-to-end algorithm to output the recognized latex sequence.

4. The written test calculator system according to claim 1, characterized in that: The interactive module detects whether the latex expression contains an equal sign, and if an equal sign is recognized, removes the last equal sign, recognizes the formula part before the equal sign, and then calculates it in turn through the latex processing module and the Sympy calculation module, and the display module displays the calculation result; If the equal sign is not recognized, no calculation is performed, the formula recognition continues and the recognition result is returned to the front end for display.

5. The written test calculator system according to claim 1, characterized in that: The latex processing module first converts the latex expression into a Sympy expression using the latex2Sympy function in the latex2Sympy2 library, then uses the dictionary variances to replace the variables in the Sympy expression, and then uses the simplify function in the Sympy library to simplify the Sympy expression.

6. The written test calculator system according to claim 1, characterized in that: The Sympy calculation module formats the calculation results: integer results are rounded, and decimal results are rounded to two decimal places.

7. The written test calculator system according to claim 1, characterized in that: The Sympy calculation module converts the angles expressed in degrees in the user's handwritten formula into radians, specifically including: Angle detection: Use the regular expression (\d+(\.\d+)?)\^\{?\\circ}? to detect the angle representation in the handwritten formula; Angle value extraction: Use the regular expression match.group to extract the angle value, and convert the extracted angle value into a floating point number; Conversion from degrees to radians: Use the formula: degrees = degrees / 180*π to convert the angle value from degrees to radians; Format as latex expression: Format the converted radian value as latex expression; Replace the original expression: Use the regular expression replacement method regex.sub to replace the angle system representation in the original expression with the converted radian system representation.

8. The written test calculator system according to claim 2, characterized in that: When the system is deployed on a cloud server, the display module transmits the calculation results back to the local interface through the http protocol for display. When the system is deployed locally, the calculation results are directly displayed on the local interface.

Images