Enhanced assistant for math that is handwritten on a computer device

Abstract

This disclosure describes systems, methods, and devices for analyzing and correcting handwritten math characters entered on a device. A method may include: receiving handwritten strokes entered on the device by a user; identifying mathematical characters represented by the handwritten strokes; inputting the mathematical characters into a first machine learning model for distinguishing between mathematical units and other characters; identifying, by the first machine learning model, first mathematical units of the mathematical characters; removing the mathematical units from the mathematical characters; inputting the mathematical characters without the mathematical units into a second machine learning model for identifying mathematical errors represented by characters; performing, by the second machine learning model, a verification process to identify mathematical errors represented by the mathematical characters; and performing, by the at least one processor, an action based on the verification process.

Description

[0001] ENHANCED ASSISTANT FOR MATH THAT IS HANDWRITTEN ON A COMPUTER DEVICE

[0002] CROSS-REFERENCE TO RELATED APPLICATION

[0003] This application is related to an claims priority under 35 U.S.C. § 119(e) from U.S. Patent Application No. 63 / 531,379, filed August 8, 2023, titled “ENHANCED ASSISTANT FOR MATH THAT IS HANDWRITTEN ON A COMPUTER DEVICE,” the entire content of which is incorporated herein by reference for all purposes.

[0004] TECHNICAL FIELD

[0005] Embodiments of the present invention generally relate to systems and methods for analyzing and indicating errors in math handwritten on a computer device.

[0006] BACKGROUND

[0007] Devices may allow users to handwrite text rather than enter text using keystrokes. Handwritten text on a computer device presents challenges in identifying the characters of the handwritten text that are not present when converting keystrokes to characters.

[0008] BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1 illustrates an example user interface for an enhanced assistant for math that is handwritten on a computer device, in accordance with one embodiment.

[0010] FIG. 2 shows an example process for real-time assessment and correction of text that is handwritten on a computer device, in accordance with one embodiment.

[0011] FIG. 3 shows an example process for identifying errors in handwritten math as in block 210 of FIG. 2, in accordance with one embodiment.

[0012] FIG. 4 illustrates an example user interface for an enhanced assistant for math that is handwritten on a computer device, in accordance with one embodiment.

[0013] FIG. 5 is an example schematic diagram of one or more artificial intelligence models that may be used for an enhanced assistant for math that is handwritten on a computer device, in accordance with one embodiment.

[0014] FIG. 6 is an example system for an enhanced assistant for math that is handwritten on a computer device, in accordance with one embodiment. FIG. 7 is a diagram illustrating an example of a computing system that may be used in implementing embodiments of the present disclosure.

[0015] Certain implementations will now be described more fully below with reference to the accompanying drawings, in which various implementations and / or aspects are shown. However, various aspects may be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Like numbers in the figures refer to like elements throughout. Hence, if a feature is used across several drawings, the number used to identify the feature in the drawing where the feature first appeared will be used in later drawings.

[0016] DETAILED DESCRIPTION

[0017] Aspects of the present disclosure involve systems, methods, and the like, for analyzing and indicating errors in math handwritten on a computer device.

[0018] Devices may allow users to input characters in a variety of ways, such as with keystrokes and with stylus strokes. When a user enters a keystroke (e.g., using a keyboard), the keystroke is converted to a corresponding character, such as a letter, number, symbol, or punctuation mark. When a key is pressed on a keyboard, it is converted into a binary number that represents a character, so there is no ambiguity in determining which character a user typed with a keystroke. In contrast, when a user handwrites text into a device, such as with a stylus or their finger, there are many variations in the handwriting that introduce ambiguity when determining what characters the handwriting represents. Analyzing characters handwritten into a device, therefore, depends on the ability of the device to correctly identify the characters represented by the handwriting. Humans may identify and categorize handwritten characters after seeing only a few examples, but a machine’s ability to identify and categorize handwritten characters may require significantly more examples to train. An electronic device encompasses a broad array of electronic gadgets, including tools such as a digital stylus or any comparable apparatus, which permit the user to sketch characters on a computer interface as a form of handdrawn or handwritten input. Beyond the use of an electronic device for inputting strokes onto the computer device, users can also engage the intuitiveness of their own fingers as a dynamic and natural means to accomplish the same task, thus providing a more direct and tactile interaction with the digital interface. Throughout this disclosure, while electronic devices are primarily illustrated as examples, it should be understood that the scope of interaction is not limited to these alone. A user’s finger also serves as a viable tool for interacting with computer devices. Hence, the exemplification of an electronic device should not be misconstrued as a limitation, but rather, it serves as one among many possible methods for interaction in the broader digital landscape. A computer device, such as a laptop, tablet, or smartphone, can be described as a sophisticated system equipped with an interactive interface designed to accept and interpret strokes from an electronic device, recording these inputs as lines, characters, shapes, and more. This interaction transforms abstract human action into digitized elements.

[0019] To allow a computer device to analyze math input by characters handwritten into the computer device, correctly identifying the handwritten text is important to the computer device’s ability to assess the math represented by the handwritten text. If the computer device improperly identifies handwritten math, then the computer device may not correctly assess whether the math is correct.

[0020] A device-based analysis of handwritten math also must be able to process the characters identified from the handwritten inputs to the device, recognize that they represent mathematical steps and operations, and determine whether the mathematical steps and operations are correct both individually and in combination. An analysis simply comparing entered characters to a pre-set mathematical solution may be too rigid, as there may be multiple correct ways to represent steps to solving a mathematical problem.

[0021] Analysis of answers to mathematical questions limited to whether the final answer is correct may not be as helpful to users as the ability to indicate in real-time to the users whether any intermediate steps in their answers may be mistaken. Real-time assessment and correction of math answers inputted to a computer device present unique challenges in identifying errors in intermediate steps without being able to assess whether the final answer that has not yet been inputted is correct. Even with the final answer inputted to the device, the ability of a computer device to assess the intermediate steps leading to the final answer may be limited by rigid solutions that simply compare the entered characters to pre-set approved characters representing a correct solution and method for deriving the correct solution.

[0022] There is therefore a need for enhanced device-based assistance for math that is handwritten on a computer device.

[0023] In one or more embodiments, a computer device may receive handwritten strokes on a screen or touchpad, such as with a stylus or a user’s finger, representing handwritten characters. The device may analyze the handwritten strokes to identify the characters represented by the handwritten strokes based on the X and Y coordinates of the strokes on the computer device. The computer device may recognize math represented by the characters, strip units from the math (e.g., X apples and Y oranges as handwritten inputs may be stripped to X and Y without the units - apples and oranges). The computer device may assess the math represented by the unit-stripped characters to determine whether intermediate steps used to derive a final solution, and the final solution, are correct or include mathematical errors. When the computer device detects a mathematical error in the characters, the computer device may generate and present an indication of the error, such as an underline or other annotation indicating to a user that the characters in any portion of the entered answer are incorrect. The analysis and error indication may occur in real-time so that the computer device may notify the user of errors in their answer prior to completing their final answer to a mathematical question. In this manner, the enhanced techniques herein differ from the way that a human operator, such as a teacher or other human instructor, would analyze and correct mathematical answers.

[0024] In one or more embodiments, the computer device may use machine learning for one or multiple aspects of the mathematical analysis and correction. For example, a machine learning model may be used to assess the handwritten strokes as inputs, and identify the characters represented by the strokes based on features of the strokes, such as the X and Y coordinates of the strokes on the computer device. Another machine learning model may use named entity recognition (NER) to strip units from the characters. NER may be trained to identify and differentiate between entities, such as characters representing mathematical values and variables, and characters representing units. Another machine learning model may receive the unit-stripped characters represented by the handwritten strokes as inputs and may be trained to identify whether the steps of the mathematical inputs represented by the unit-stripped characters are consistent with each other and with expected inputs representing the solution to a mathematical problem.

[0025] In one or more embodiments, the computer device may identify lines / portions of the handwritten strokes corresponding to individual steps in a mathematical answer. For example, a step in the answer may span one or multiple lines. The computer device may identify one or more characters that represent an individual step in the answer. The computer device may apply a maximum covering algorithm, for example, to the identified individual steps of the answer. The maximum covering algorithm may include applying a maximum subset algorithm to the individual steps to determine whether the steps are consistent. For example, in a mathematical answer with five steps, step one may be compared to steps two, three, and four, step two may be compared to steps one and three through five, etc. For example, if one step is “2x +2y,” and previous steps include “x+x+y+y+z,” “z=0,” “x+x+y+y,” and “x+x+2y,” the device may filter the steps to identify comparable steps (e.g., current step “2x+2y” only has variables x and y, so the previous steps that are comparable would be “x+x+y+y” and “x+x+2y”). The device may assess the structural equivalence and numerical equivalence between the comparable steps, whether the current step is relational to the previous steps (e.g., are the steps symmetrical - left / right-hand sides of = signs or comparative signs, etc.), and whether the current step has evolved from the previous comparable steps. When the comparable steps have structural and numerical equivalence (e.g., compared to respective threshold equivalences), and when the current line has evolved with respect to the comparable steps, such may indicate an error (e.g., contradiction) in the current step, which may be indicated via the device by presenting an underline or other annotation with the step. In another example in which the question is “8x+8=108,” one step in the entered answer may be to subtract 8 from both sides: “8x+8 (-8) = 108 (-8),” which results in “8x = 100.” However, when the handwritten answer includes the step “8x = 116,” indicating that 8 was added to 108 instead of subtracted, the device may display the “8x = 116” with an underline to indicate the error.

[0026] In one or more embodiments, the error indication may be presented with hints for how to correct the error. For example, when the text in error, or its annotation (e.g., underline, highlight, different text color than the characters with no errors, etc.) is selected by the user via the computer device, the computer device may present suggestions for how to correct the error. The steps may be general, such as the general approach to solving the problem (e.g., move the variable to one side of the equation and the constants to the other side, then solve for the variable), or more specific to the identified error (e.g., “check addition / subtraction here,” etc.). Artificial intelligence may be used to customize the hints based on the initial question and the particular step in the answer for which an error has been identified.

[0027] In one or more embodiments, the text identification of handwritten characters may use few-shot learning, one-shot learning, or no-shot learning. In few-shot learning, computer vision and / or natural language processing may be used to recognize, parse, and classify handwritten characters. In one-shot learning, images of handwritten text may be used to identify similarities on the example images and the handwritten text inputs. In zero-shot learning, a machine learning model may not need to be trained, but instead learns the ability to predict handwritten characters.

[0028] The above descriptions are for purposes of illustration and are not meant to be limiting. Numerous other examples, configurations, processes, etc., may exist, some of which are described in greater detail below. Example embodiments will now be described with reference to the accompanying figures.

[0029] FIG. 1 illustrates an example user interface for an enhanced assistant for math that is handwritten on a computer device, in accordance with one embodiment.

[0030] Referring to FIG. 1, a device 102 (e.g., a laptop, tablet, smartphone, touchscreen, television, smart home assistant, VR / AR device, or the like) may present one or more user interfaces capable of presenting mathematical equations. For example, the device may present a user interface 104, which may present questions (e.g., question 106) such as mathematical questions / problems (e.g., the problem 8x + = 108 as shown) for a user to solve. The interface 104 (or another interface) may present selectable answers 108 for the question 106, some of which may be correct, and some of which may be incorrect. The interface 104 (or another interface) may present an option to show hints 110 such that, when selected, hints may be presented using the device 102 to a user (e.g., as shown in FIG. 4).

[0031] Still referring to FIG. 1, the device 102 may present an interface 120, which may be part of a same application or different application as the interface 104. The interface 120 may allow a user to handwrite answers (e.g., using a finger or input device such as a stylus 122, etc.) to mathematical questions. As shown in FIG. 1, the mathematical question 106 (e.g., 8x + 8 = 108) may be presented using the interface 120, along with the answers 108 including a correct answer (e.g., 12.5) and one or more incorrect answers. A user may select from among the answers 108, and an application on the device 102 and / or executing remotely (e.g., responsible for presenting the user interface 104 and / or the interface 120) may determine whether the user’s selected answer is correct or not.

[0032] In addition to assessing the user’s answer to the question, the application may assess in real-time the user’s intermediate mathematical steps, proofs, and the like, used to support and / or arrive at the answer. For example, as shown, the user may handwrite “8x+8-8=108-8” as a step 124 to solve the question 106, and then “8x=116” as a step 126. Before the user handwrites subsequent mathematical steps and / or the final answer, the application may determine whether the handwritten steps are correct (e.g., because the answer and some steps to reach the answer are known by the application). In the example shown, 8x should equal 100 (e.g., 108-8), but the handwritten text shows “8x=116,” representing an error in which 8 was added to 108 instead of subtracted from 108. When the application detects the error in 8x=l 16, the application may generate and present, using the interface 120, an indication 128 of the error (e.g., an underline of the 8x=l 16 handwritten text) to communicate the error to the user so that the user may correct the error prior to continuing to handwrite additional steps and / or the final answer. Math assistance (e.g., artificial intelligence math assistance 130) may be presented along with the indication 128 of the error, and may indicate that an error has been identified and where (e.g., using an indicator 132 showing which step of the user’s solution is identified as being in error).

[0033] In one or more embodiments, the application’s real-time analysis of mathematical consistency between steps and mathematical accuracy may depend on real-time detection and identification of handwritten characters, the ability to distinguish between different steps of an entered solution, and the ability to detect whether a particular step includes mathematical errors (e.g., adding 8 to 100 instead of subtracting 8 from 100 in the example shown in FIG. 1) and / or to detect whether a step is mathematically consistent with a previous or subsequent step entered via user handwriting. Because the real-time handwriting analysis and detection is improved by the techniques herein, the ability to detect mathematical errors also is improved. By detecting and presenting the errors in real-time, a user may be prompted to modify their handwritten solutions before continuing to enter incorrect answers.

[0034] FIG. 2 shows an example process 200 for real-time assessment and correction of text that is handwritten on a computer device, in accordance with one embodiment.

[0035] At block 202, a device (or system, e.g., the handwriting and math devices 719 of FIG. 7) representing an application running on a user device and / or a remote server (e.g., a cloudbased server) may detect handwritten strokes entered on a display of the user device. For example, a user may input handwritten strokes with a finger, stylus (e.g., the stylus 122 of FIG. 1), or another instrument / input device. The strokes may be input into one or more user interfaces of the application (e.g., the interface 104 and / or the interface 120) so that the application may detect them.

[0036] At block 204, the device may convert the handwritten strokes to characters. The handwritten strokes may have pixel coordinates (e.g., on the device 102, a touchpad or portion of a device, etc.) where the user’s finger, stylus, or other handwriting input device touched the display. The pixel coordinates (e.g., X and Y coordinates of the display) may correspond to characters. In this manner, detecting the handwritten characters differs from mapping a keyboard input to a character. The conversion of handwritten strokes to characters may use machine learning (e.g., FIG. 5), such a model trained to detect characters based on similarities and / or differences with known handwritten characters (e.g., previously learned and / or trained with training data), including the pixel coordinates, and other features such as shape, size, and the like. The characters may include numbers, letters, symbols, math constructs, functions, matrices, and the like.

[0037] At block 206, the device may identify mathematical steps represented by the characters. The conversion also may include a detection of the characters representing math (e.g., distinguishing between math characters and other types of handwriting, such as sentences, etc.). The device may separate the characters into lines / steps (e.g., the step 124, the step 126). For example, a math step may include one or more lines of characters entered in the user interface of the application. Detecting where a step begins and ends may include parsing the characters to identify whether there are characters to the left and right of an “=” sign, and / or whether there are math operators with subsequent characters, etc. The device may generate code indicating where math characters begin and end, where an equation begins and ends, and where each step / line begins and ends.

[0038] At block 208, the device may remove units from the mathematical steps (e.g., unit stripping). For example, when the characters represent X of one object and Y of another object, the objects may be removed from X and Y. Unit stripping may use named entity recognition and or deep learning (e.g., BERT, etc.) to identify units represented by the characters. Units identified may be removed from the mathematical steps by the device.

[0039] At block 210, the device may identify errors in the mathematical steps (e.g., an example process for this step is shown in FIG. 3). The process 200 may be performed in real-time as new handwritten strokes are input to the device, so the ability of the device to detect errors may depend on which strokes have been entered at a given time. For example, there may be no mathematical errors at one moment, but at a later moment after one or more additional characters have been handwritten, the additional characters may represent a mathematical error that the device may identify. Identifying mathematical errors may include using a maximum covering algorithm in which maximum subset algorithms may be used to compare the different lines / steps to one another to identify consistent sets. When a step is not relational to another step, for example, such may trigger an error. When a step is relational to another step, but indicates that the solution has evolved (e.g., the error of incorrectly adding 8 to 108 as shown in FIG. 1), such may trigger an error.

[0040] At block 212, the device may generate error indications for concurrent presentation with the handwritten strokes (e.g., presentation using the user interface). The indications may include underlining, highlighting, and / or other annotations of the characters where the error is identified. At block 214, the device may present the error indications concurrently with the handwritten strokes in the user interface in real-time (e.g., prior to / during subsequent handwritten strokes being input to the device), such as while the user is still writing or prior to the user entering a next step / line.

[0041] At block 216, the device optionally may generate and present hints for correcting the detected errors. The hints may be presented concurrently (e.g., FIG. 4) with the handwriting using the user interface. For example, when the text in error, or its annotation (e.g., underline, highlight, different text color than the characters with no errors, etc.) is selected by the user via the device, the device may present suggestions for how to correct the error. The steps may be general, such as the general approach to solving the problem (e.g., move the variable to one side of the equation and the constants to the other side, then solve for the variable), or more specific to the identified error (e.g., “check addition / subtraction here,” etc.). Artificial intelligence (e.g., FIG. 5) may be used to customize the hints based on the initial question and the particular step in the answer for which an error has been identified.

[0042] The examples herein are not meant to be limiting.

[0043] FIG. 3 shows an example process 300 for identifying errors in handwritten math as in block 210 of FIG. 2, in accordance with one embodiment.

[0044] A computer device (or system, e.g., the handwriting and math devices 719 of FIG. 7) representing an application running on a user device and / or a remote server (e.g., a cloud-based server) may apply a maximum covering algorithm, for example, to the identified individual steps of the answer. At step 302, the computer device may initiate sets (e.g., steps of the math solution) and use a set enumerator 304 to identify the number of steps that a user has handwritten at any point in time. When there are not at least two math sets identified at step 306, the process 300 may end 308. When there are at least two math sets identified at step 306, they may be analyzed in relation to each other. For example, using set A, set B, and set C, at step 310, the intent of step C may be identified based on the intent of set A and set B.

[0045] Based on the intent of set C, the process 300 may include checking a numerical equivalence at step 312 (e.g., numerical equivalence between sets A, B, and C). When the numerical equivalence is below a numerical equivalence threshold at step 314, the process 300 may return to the set enumerator 304 to identify the number of sets entered at that time. When the sets exhibit numerical equivalence at step 314, the process 300 may continue to step 316, where set A and set B may be fused together. In addition to the numerical equivalence, the process 300 may assess numerical solvability of sets A, B, and C at step 318. When the sets are solvable at step 320, the process 300 may continue to step 316. When the sets are not solvable at step 320, the process 300 may evaluate the sets for a transformation at step 322. When a valid transformation is identified at step 324, the process 300 may continue to step 316. When a valid transformation is not identified at step 324, the process 300 may return to the set enumerator 304.

[0046] In addition to the numerical equivalence and the numerical solvability, the process 300 may assess the sets for contradictions at step 326. For example, when set C contradicts the math of set A or set B, the sets may be considered inconsistent at step 328 and the process 300 may return to the set enumerator 304. When the sets are consistent at step 328, the process 300 may proceed to step 316.

[0047] The maximum covering algorithm represented by the process 300 may include applying a maximum subset algorithm to the individual mathematical steps (sets) to determine whether the steps are consistent. For example, in a mathematical answer with five steps, step one may be compared to steps two, three, and four, step two may be compared to steps one and three through five, etc. For example, if one step is “2x +2y,” and previous steps include “x+x+y+y+z,” “z=0,” “x+x+y+y,” and “x+x+2y,” the device may filter the steps to identify comparable steps (e.g., current step “2x+2y” only has variables x and y, so the previous steps that are comparable would be “x+x+y+y” and “x+x+2y”). The device may assess the structural equivalence and numerical equivalence between the comparable steps, whether the current step is relational to the previous steps (e.g., are the steps symmetrical - left / right-hand sides of = signs or comparative signs, etc.), and whether the current step has evolved from the previous comparable steps.

[0048] When the comparable steps have structural and numerical equivalence (e.g., compared to respective threshold equivalences), and when the current line has evolved with respect to the comparable steps, such may indicate an error (e.g., contradiction) in the current step, which may be indicated via the device by presenting an underline or other annotation with the step. In another example in which the question is “8x+8=108,” one step in the entered answer may be to subtract 8 from both sides: “8x+8 (-8) = 108 (-8),” which results in “8x = 100.” However, when the handwritten answer includes the step “8x = 116,” indicating that 8 was added to 108 instead of subtracted, the device may display the “8x = 116” with an underline to indicate the error. Referring to FIGs. 2 and 3, the processes may input the user’s handwriting and the official solution for any question (e.g., the question 106 of FIG. 1). Handwriting recognition may allow for recognizing math represented by the handwritten characters (e.g., distinguishing the handwritten characters as math rather than sentences, etc.). Unit stripping and parsing allow for identifying the steps (sets) of the mathematical solution written by the user. Once the steps have been identified, the maximum covering and maximum subset algorithms may be used to output a list of consistent sets or identify errors that may be presented along with hints for solving the question / problem.

[0049] FIG. 4 illustrates an example user interface for an enhanced assistant for math that is handwritten on a computer device, in accordance with one embodiment.

[0050] Referring to FIG. 4, the device 102 of FIG. 1 may present the interface 120, which may present a question along with hints 402 (e.g., when the user selects the hints 110 option of FIG. 1). The hints 402 may include hint text 402 along with recommended strategy 406 for solving the question / problem. The interface 104 may present options to present additional hints 408 when there are multiple available for presentation. As shown in FIG. 4, the hints 402 may be presented concurrently with steps entered by the user (e.g., step 422, step 424, entered using the stylus 122). In this manner, the hints 402 may be presented as the user enters steps of the solution, and the hints 402 may update based on recognized errors in the user’s solution as they write it in real time.

[0051] For example, when the text of the steps in error, or its annotation (e.g., underline, highlight, different text color than the characters with no errors, etc.) is selected by the user via the user interface 120, the device 102 may present suggestions for how to correct the error. The steps may be general, such as the general approach to solving the problem (e.g., move the variable to one side of the equation and the constants to the other side, then solve for the variable), or more specific to the identified error (e.g., “check addition / subtraction here,” etc.). Artificial intelligence (e.g., FIG. 5) may be used to customize the hints based on the initial question and the particular step in the answer for which an error has been identified.

[0052] FIG. 5 is an example schematic diagram of one or more artificial intelligence models that may be used for an enhanced assistant for math that is handwritten on a computer device, in accordance with one embodiment.

[0053] Referring to FIG. 5, one or more artificial intelligence (Al) models 502 (or machine learning models) may be used for any of detecting the handwritten characters, determining that the handwritten characters represent math, identifying steps / lines in the math, unit stripping, and / or identifying errors in the math. The one or more Al models 502 may receive inputs, optionally may receive data 504 (e.g., training data, one- or few-shot examples, user feedback, etc.), and may generate outputs 508. Optionally, feedback 510 from the outputs 508 may be input into the one or more Al models 502, such as human-in-the-loop feedback, user feedback, comparisons of the outputs 508 to known outputs and their differences (e.g., used to adjust the one or more Al models 502, such as by adjusting weights for identifying characters, steps / lines, errors, etc.).

[0054] In one or more embodiments, the text identification of handwritten characters may use few-shot learning, one-shot learning, or no-shot learning. In few-shot learning, computer vision and / or natural language processing may be used to recognize, parse, and classify handwritten characters. In one-shot learning, images of handwritten text may be used to identify similarities on the example images and the handwritten text inputs. In zero-shot learning, a machine learning model may not need to be trained, but instead learns the ability to predict handwritten characters.

[0055] In one or more embodiments, when the one or more Al models 502 are used to detect handwritten characters, the inputs 506 may be the handwritten strokes and / or characteristics of the handwritten strokes, such as their pixel coordinates on the display with which they were input. The data 504 may include features of characters, such as their coordinates, shapes, sizes, and the like, accounting for different fonts, such as cursive, block letters, etc. The outputs 508 may include the characters identified from the handwritten strokes. The outputs 508 may be re-input to the one or more Al models 502 until the one or more Al models 502 determine that the confidence score assigned to the identified characters exceeds a threshold confidence. The closer the similarities between the inputs 506 and the known characters, for example, the higher the confidence score for identifying the characters.

[0056] In one or more embodiments, when the one or more Al models 502 are used for math recognition, the inputs 506 may include the identified characters from the handwritten strokes. The data 504 may include mathematical characters and non-mathematical characters (e.g., for distinguishing between math and other handwritten characters). The outputs 508 may include one or more files indicating that the characters represent math, where steps / lines begin and end, where an answer begins and ends, and the like.

[0057] In one or more embodiments, when the one or more Al models 502 are used for unit stripping, the inputs 506 may include the mathematical characters and / or one or more files indicating that the characters represent math, where steps / lines begin and end, where an answer begins and ends, and the like. The data 504 may include variables, constants, and / or units (e.g., to distinguish between units and non-units in the math). The outputs 508 may include unit- stripped math characters.

[0058] In one or more embodiments, when the one or more Al models 502 are used for checking the math for errors, the inputs may include the unit-stripped math, which may include the one or more files indicating that the characters represent math, where steps / lines begin and end, where an answer begins and ends, and the like, with the units removed from the numbers / variables in the math. The data 504 may include previous lines / steps of the math answer provided based on the handwritten strokes, and / or data showing similarities and differences between mathematical structures and / or numerical equivalence. The outputs 508 may include an indication of any errors identified (e.g., FIG. 3).

[0059] FIG. 6 is an example system 600 for an enhanced assistant for math that is handwritten using a device, in accordance with one embodiment.

[0060] Referring to FIG. 6, the system 600 may include one or more devices 602 (e.g., laptops, desktops, smartphones, smart home assistants, wearable devices, televisions, or the like) capable of displaying text and receiving handwritten strokes (e.g., from a stylus 604, a finger of a user 606, or another input device). The system 600 may include one or more remote devices 608 (e.g., servers, cloud-based devices, etc.). The one or more devices 602 and / or the one or more remote devices 608 may execute applications that receive, analyze, and correct handwritten strokes input via the one or more devices 602. For example, the one or more devices 602 may transmit indications of the handwritten strokes and / or any analysis of the handwritten strokes to the one or more remote devices 608 (e.g., a front-end / back-end integration of the application). Alternatively, the one or more devices 602 may analyze, detect errors, and correct the handwritten text locally.

[0061] Still referring to FIG. 6, the one or more devices 602 and / or the one or more remote devices 608 may include handwriting modules 610 (e.g., for receiving and detecting handwritten strokes, identifying the characters of the handwritten strokes), math modules 612 (e.g., for detecting math in the identified characters, detecting errors in the math), one or more user interface modules 612 (e.g., for generating the presentable data of the user interfaces shown in the figures, including the handwritten strokes, error indications, and / or hints), and Al models 616 (e.g., the one or more Al models 502 of FIG. 5).

[0062] In one or more embodiments, the one or more devices 602 may receive handwritten strokes on a screen or touchpad, such as with the stylus 604 or a user’s finger, representing handwritten characters. The handwriting modules 610 may analyze the handwritten strokes to identify the characters represented by the handwritten strokes based on the X and Y coordinates of the strokes on the one or more devices 602. The handwriting modules 610 and / or the math modules 612 may recognize math represented by the characters, strip units from the math (e.g., X apples and Y oranges as handwritten inputs may be stripped to X and Y without the units - apples and oranges). The math modules 612 may assess the math represented by the unitstripped characters to determine whether intermediate steps used to derive a final solution, and the final solution, are correct or include mathematical errors. When the math modules 612 detect a mathematical error in the characters, the user interface modules 614 may generate and present an indication of the error, such as an underline or other annotation indicating to a user that the characters in any portion of the entered answer are incorrect. The analysis and error indication may occur in real-time so that the one or more devices 602 may notify the user of errors in their answer prior to completing their final answer to a mathematical question. In this manner, the enhanced techniques herein differ from the way that a human operator, such as a teacher or other human instructor, would analyze and correct mathematical answers.

[0063] In one or more embodiments, the one or more devices 602 and / or the one or more remote devices 608 may use machine learning (e.g., the Al models 616) for one or multiple aspects of the mathematical analysis and correction. For example, a machine learning model may be used to assess the handwritten strokes as inputs, and identify the characters represented by the strokes based on features of the strokes, such as the X and Y coordinates of the strokes on the device. Another machine learning model may use named entity recognition (NER) to strip units from the characters. NER may be trained to identify and differentiate between entities, such as characters representing mathematical values and variables, and characters representing units. Another machine learning model may receive the unit-stripped characters represented by the handwritten strokes as inputs, and may be trained to identify whether the steps of the mathematical inputs represented by the unit-stripped characters are consistent with each other and with expected inputs representing the solution to a mathematical problem.

[0064] In one or more embodiments, the math modules 612 may identify lines / portions of the handwritten strokes corresponding to individual steps in a mathematical answer. For example, a step in the answer may span one or multiple lines. The math modules 612 may identify one or more characters that represent an individual step in the answer. The math modules 612 may apply a maximum covering algorithm, for example, to the identified individual steps of the answer. The maximum covering algorithm may include applying a maximum subset algorithm to the individual steps to determine whether the steps are consistent. For example, in a mathematical answer with five steps, step one may be compared to steps two, three, and four, step two may be compared to steps one and three through five, etc. For example, if one step is “2x +2y,” and previous steps include “x+x+y+y+z,” “z=0,” “x+x+y+y,” and “x+x+2y,” the device may filter the steps to identify comparable steps (e.g., current step “2x+2y” only has variables x and y, so the previous steps that are comparable would be “x+x+y+y” and “x+x+2y”). The math modules 612 may assess the structural equivalence and numerical equivalence between the comparable steps, whether the current step is relational to the previous steps (e.g., are the steps symmetrical - left / right-hand sides of = signs or comparative signs, etc.), and whether the current step has evolved from the previous comparable steps. When the comparable steps have structural and numerical equivalence (e.g., compared to respective threshold equivalences), and when the current line has evolved with respect to the comparable steps, such may indicate an error (e.g., contradiction) in the current step, which may be indicated via the device by presenting an underline or other annotation with the step. In another example in which the question is “8x+8=108,” one step in the entered answer may be to subtract 8 from both sides: “8x+8 (-8) = 108 (-8),” which results in “8x = 100.” However, when the handwritten answer includes the step “8x = 116,” indicating that 8 was added to 108 instead of subtracted, the one or more devices 602 may display the “8x = 116” with an underline to indicate the error.

[0065] In one or more embodiments, the error indication may be presented with hints for how to correct the error. For example, when the text in error, or its annotation (e.g., underline, highlight, different text color than the characters with no errors, etc.) is selected by the user via the device, the device may present suggestions for how to correct the error. The steps may be general, such as the general approach to solving the problem (e.g., move the variable to one side of the equation and the constants to the other side, then solve for the variable), or more specific to the identified error (e.g., “check addition / subtraction here,” etc.). The Al models 616 may be used to customize the hints based on the initial question and the particular step in the answer for which an error has been identified.

[0066] It is understood that the above descriptions are for purposes of illustration and are not meant to be limiting.

[0067] FIG. 7 is a diagram illustrating an example of a computing system 700 that may be used in implementing embodiments of the present disclosure. FIG. 7 is a block diagram illustrating an example of a computing device or computer system 700 which may be used in implementing the embodiments of the components disclosed above. For example, the computing system 700 of FIG. 7 may represent at least a portion of the system one or more devices 602 and / or the one or more remote devices 608 of FIG. 6, as discussed above. The computer system (system) includes one or more processors 702-706. Processors 702-706 may include one or more internal levels of cache (not shown) and a bus controller 722 or bus interface unit to direct interaction with the processor bus 712. Processor bus 712, also known as the host bus or the front side bus, may be used to couple the processors 702-706 with the system interface 724. System interface 724 may be connected to the processor bus 712 to interface other components of the system 700 with the processor bus 712. For example, system interface 724 may include a memory controller 718 for interfacing a main memory 716 with the processor bus 712. The main memory 716 typically includes one or more memory cards and a control circuit (not shown). System interface 724 may also include an input / output (I / O) interface 720 to interface one or more I / O bridges 725 or I / O devices with the processor bus 712. One or more I / O controllers and / or I / O devices may be connected with the I / O bus 726, such as I / O controller 728 and I / O device 730, as illustrated. The system 700 may include one or more handwriting and math devices 719 (e.g., representing at least a portion of the handwriting modules 610, the math modules 612, the user interface modules 614, and / or the Al models 616 of FIG. 6).

[0068] I / O device 730 may also include an input device (not shown), such as an alphanumeric input device, including alphanumeric and other keys for communicating information and / or command selections to the processors 702-706. Another type of user input device includes cursor control, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to the processors 702-706 and for controlling cursor movement on the display device.

[0069] System 700 may include a dynamic storage device, referred to as main memory 716, or a random access memory (RAM) or other computer-readable devices coupled to the processor bus 712 for storing information and instructions to be executed by the processors 702-706. Main memory 716 also may be used for storing temporary variables or other intermediate information during execution of instructions by the processors 702-706. System 700 may include a read only memory (ROM) and / or other static storage device coupled to the processor bus 712 for storing static information and instructions for the processors 702-706. The system outlined in FIG. 7 is but one possible example of a computer system that may employ or be configured in accordance with aspects of the present disclosure.

[0070] According to one embodiment, the above techniques may be performed by computer system 700 in response to processor 704 executing one or more sequences of one or more instructions contained in main memory 716. These instructions may be read into main memory 716 from another machine-readable medium, such as a storage device. Execution of the sequences of instructions contained in main memory 716 may cause processors 702-706 to perform the process steps described herein. In alternative embodiments, circuitry may be used in place of or in combination with the software instructions. Thus, embodiments of the present disclosure may include both hardware and software components.

[0071] A machine readable medium includes any mechanism for storing or transmitting information in a form (e.g., software, processing application) readable by a machine (e.g., a computer). Such media may take the form of, but is not limited to, non-volatile media and volatile media and may include removable data storage media, non-removable data storage media, and / or external storage devices made available via a wired or wireless network architecture with such computer program products, including one or more database management products, web server products, application server products, and / or other additional software components. Examples of removable data storage media include Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc Read-Only Memory (DVD- ROM), magneto-optical disks, flash drives, and the like. Examples of non-removable data storage media include internal magnetic hard disks, SSDs, and the like. The one or more memory devices 706 may include volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and / or non-volatile memory (e.g., readonly memory (ROM), flash memory, etc.).

[0072] Computer program products containing mechanisms to effectuate the systems and methods in accordance with the presently described technology may reside in main memory 716, which may be referred to as machine-readable media. It will be appreciated that machine- readable media may include any tangible non-transitory medium that is capable of storing or encoding instructions to perform any one or more of the operations of the present disclosure for execution by a machine or that is capable of storing or encoding data structures and / or modules utilized by or associated with such instructions. Machine-readable media may include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store the one or more executable instructions or data structures. The following examples are not meant to be exclusive.

[0073] Example 1 may include a method for analyzing and correcting handwritten math characters entered on a device, the method comprising: receiving, by at least one processor of a device, handwritten strokes entered on the device by a user; identifying, by the at least one processor, mathematical characters represented by the handwritten strokes; inputting, by the at least one processor, the mathematical characters into a first machine learning model configured to distinguish between mathematical units and other characters; identifying, by the first machine learning model, first mathematical units of the mathematical characters; removing, by the at least one processor, the mathematical units from the mathematical characters; inputting, by the at least one processor, the mathematical characters without the mathematical units into a second machine learning model; performing, by the second machine learning model, a verification process to identify mathematical errors represented by the mathematical characters; and performing, by the at least one processor, an action based on the verification process.

[0074] Example 2 may include the method of example 1 and / or any other example herein, wherein performing the action comprises: presenting the mathematical characters on a display, or performing a correction process based on identifying mathematical errors represented by the mathematical characters.

[0075] Example 3 may include the method of example 2 and / or any other example herein, wherein performing the correction process comprises: identifying, by the second machine learning model, a first mathematical error in a portion of the mathematical characters without the mathematical units; generating, by the at least one processor, an indication of the first mathematical error for presentation by the device; and presenting, by the at least one processor, on the device, the indication with the portion.

[0076] Example 4 may include the method of example 3 and / or any other example herein, wherein the indication is presented with the portion during entry of second handwritten strokes by the user on the device.

[0077] Example 5 may include the method of example 4 and / or any other example herein, wherein the portion represents a step in an answer to a mathematical question, and wherein the second handwritten strokes represent a second step of the mathematical question.

[0078] Example 6 may include the method of example 5 and / or any other example herein, further comprising: distinguishing, by the second machine learning model, between the step and a third step in the answer to the mathematical question. Example 7 may include the method of example 3 and / or any other example herein, wherein the indication further comprises a hint associated with correcting the first mathematical error.

[0079] Example 8 may include the method of example 1 and / or any other example herein, further comprising: receiving second handwritten strokes entered on the device by the user, the second handwritten strokes representing a correction of the handwritten strokes; identifying second mathematical characters represented by the second handwritten strokes; determining, by the second machine learning model, that the second mathematical characters do not include a mathematical error; and presenting the second mathematical characters on the device without any indication of a mathematical error.

[0080] Example 9 may include the method of example 1 and / or any other example herein, wherein the second machine learning model is configured to apply a maximum covering algorithm to the mathematical characters without the mathematical units.

[0081] Example 10 may include the method of example 9 and / or any other example herein, wherein the maximum covering algorithm comprises applying a maximum subset algorithm to individual steps of a mathematical answer represented by the mathematical characters to determine whether the individual steps are consistent with one another.

[0082] Example 11 may include a system for analyzing and correcting handwritten math characters entered on a device, the system comprising memory coupled to at least one processor, the at least one processor configured to: receive handwritten strokes entered on a device by a user; identify mathematical characters represented by the handwritten strokes; input the mathematical characters into a first machine learning model configured to distinguish between mathematical units and other characters; identify, by the first machine learning model, first mathematical units of the mathematical characters; remove, by the at least one processor, the mathematical units from the mathematical characters; input the mathematical characters without the mathematical units into a second machine learning model; perform, by the second machine learning model, a verification process to identify mathematical errors represented by the mathematical characters; and perform an action based on the verification process.

[0083] Example 12 may include the system of example 11 and / or any other example herein, wherein to perform the action comprises to: present the mathematical characters on a display, or perform a correction process based on identifying mathematical errors represented by the mathematical characters. Example 13 may include the system of example 12 and / or any other example herein, wherein to perform the correction process comprises to: identify, by the second machine learning model, a first mathematical error in a portion of the mathematical characters without the mathematical units; generate an indication of the first mathematical error for presentation by the device; and present, on the device, the indication with the portion.

[0084] Example 14 may include the system of example 13 and / or any other example herein, wherein the indication is presented with the portion during entry of second handwritten strokes by the user on the device.

[0085] Example 15 may include the system of example 14 and / or any other example herein, wherein the portion represents a step in an answer to a mathematical question, and wherein the second handwritten strokes represent a second step of the mathematical question.

[0086] Example 16 may include the system of example 15 and / or any other example herein, wherein the at least one processor is further configured to: distinguish, using the second machine learning model, between the step and a third step in the answer to the mathematical question.

[0087] Example 17 may include the system of example 13 and / or any other example herein, wherein the indication further comprises a hint associated with correcting the first mathematical error.

[0088] Example 18 may include the system of example 11 and / or any other example herein, wherein the at least one processor is further configured to: receive second handwritten strokes entered on the device by the user, the second handwritten strokes representing a correction of the handwritten strokes; identify second mathematical characters represented by the second handwritten strokes; determine, using the second machine learning model, that the second mathematical characters do not include a mathematical error; and presenting the second mathematical characters on the device without any indication of a mathematical error.

[0089] Example 19 may include a computer-readable storage medium comprising instructions to cause at least one processor for analyzing and correcting handwritten math characters entered on a device, upon execution of the instructions by the at least one processor, to: receive handwritten strokes entered on a device by a user; identify mathematical characters represented by the handwritten strokes; input the mathematical characters into a first machine learning model configured to distinguish between mathematical units and other characters; identify, using the first machine learning model, first mathematical units of the mathematical characters; remove, by the at least one processor, the mathematical units from the mathematical characters; input the mathematical characters without the mathematical units into a second machine learning model configured to identify mathematical errors represented by characters; perform, by the second machine learning model, a verification process to identify mathematical errors represented by the mathematical characters; and perform an action based on the verification process.

[0090] Example 20 may include the computer-readable storage medium of example 19 and / or any other example herein, wherein to perform the action comprises to: present the mathematical characters on a display, or perform a correction process based on identifying mathematical errors represented by the mathematical characters.

[0091] Example 21 may include an apparatus comprising means for: receiving handwritten strokes entered on a device by a user; identifying mathematical characters represented by the handwritten strokes; inputting the mathematical characters into a first machine learning model configured to distinguish between mathematical units and other characters; identifying, using the first machine learning model, first mathematical units of the mathematical characters; removing the mathematical units from the mathematical characters; inputting the mathematical characters without the mathematical units into a second machine learning model; performing, using the second machine learning model, a verification process to identify mathematical errors represented by the mathematical characters; and performing an action based on the verification process.

[0092] Embodiments according to the disclosure are in particular disclosed in the attached claims directed to a method, a storage medium, a device and a computer program product, wherein any feature mentioned in one claim category, e.g., method, can be claimed in another claim category, e.g., system, as well. The dependencies or references back in the attached claims are chosen for formal reasons only. However, any subject matter resulting from a deliberate reference back to any previous claims (in particular multiple dependencies) can be claimed as well, so that any combination of claims and the features thereof are disclosed and can be claimed regardless of the dependencies chosen in the attached claims. The subjectmatter which can be claimed comprises not only the combinations of features as set out in the attached claims but also any other combination of features in the claims, wherein each feature mentioned in the claims can be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein can be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or with any of the features of the attached claims. Embodiments of the present disclosure include various steps, which are described in this specification. The steps may be performed by hardware components or may be embodied in machine-executable instructions, which may be used to cause a general-purpose or specialpurpose processor programmed with the instructions to perform the steps. Alternatively, the steps may be performed by a combination of hardware, software and / or firmware.

[0093] Various modifications and additions can be made to the exemplary embodiments discussed without departing from the scope of the present invention. For example, while the embodiments described above refer to particular features, the scope of this invention also includes embodiments having different combinations of features and embodiments that do not include all of the described features. Accordingly, the scope of the present invention is intended to embrace all such alternatives, modifications, and variations together with all equivalents thereof.

Claims

CLAIMSWHAT IS CLAIMED:

1. A method for analyzing and correcting handwritten math characters entered on a device, the method comprising: receiving, by at least one processor of a device, handwritten strokes entered on the device by a user; identifying, by the at least one processor, mathematical characters represented by the handwritten strokes; inputting, by the at least one processor, the mathematical characters into a first machine learning model configured to distinguish between mathematical units and other characters; identifying, by the first machine learning model, first mathematical units of the mathematical characters; removing, by the at least one processor, the mathematical units from the mathematical characters; inputting, by the at least one processor, the mathematical characters without the mathematical units into a second machine learning model; performing, by the second machine learning model, a verification process to identify mathematical errors represented by the mathematical characters; and performing, by the at least one processor, an action based on the verification process.

2. The method of claim 1, wherein performing the action comprises: presenting the mathematical characters on a display, or performing a correction process based on identifying mathematical errors represented by the mathematical characters.

3. The method of claim 2, wherein performing the correction process comprises: identifying, by the second machine learning model, a first mathematical error in a portion of the mathematical characters without the mathematical units; generating, by the at least one processor, an indication of the first mathematical error for presentation by the device; and presenting, by the at least one processor, on the device, the indication with the portion.

4. The method of claim 3, wherein the indication is presented with the portion during entry of second handwritten strokes by the user on the device.

5. The method of claim 4, wherein the portion represents a step in an answer to a mathematical question, and wherein the second handwritten strokes represent a second step of the mathematical question.

6. The method of claim 5, further comprising: distinguishing, by the second machine learning model, between the step and a third step in the answer to the mathematical question.

7. The method of claim 3, wherein the indication further comprises a hint associated with correcting the first mathematical error.

8. The method of any of claim 1 or claim 2, further comprising: receiving second handwritten strokes entered on the device by the user, the second handwritten strokes representing a correction of the handwritten strokes; identifying second mathematical characters represented by the second handwritten strokes; determining, by the second machine learning model, that the second mathematical characters do not include a mathematical error; and presenting the second mathematical characters on the device without any indication of a mathematical error.

9. The method of claim 1, wherein the second machine learning model is configured to apply a maximum covering algorithm to the mathematical characters without the mathematical units.

10. The method of claim 9, wherein the maximum covering algorithm comprises applying a maximum subset algorithm to individual steps of a mathematical answer represented by the mathematical characters to determine whether the individual steps are consistent with one another.

11. A system for analyzing and correcting handwritten math characters entered on a device, the system comprising memory coupled to at least one processor, the at least one processor configured to: receive handwritten strokes entered on a device by a user; identify mathematical characters represented by the handwritten strokes; input the mathematical characters into a first machine learning model configured to distinguish between mathematical units and other characters; identify, by the first machine learning model, first mathematical units of the mathematical characters; remove, by the at least one processor, the mathematical units from the mathematical characters; input the mathematical characters without the mathematical units into a second machine learning model; perform, by the second machine learning model, a verification process to identify mathematical errors represented by the mathematical characters; and perform an action based on the verification process.

12. The system of claim 11, wherein to perform the action comprises to: present the mathematical characters on a display, or perform a correction process based on identifying mathematical errors represented by the mathematical characters.

13. The system of claim 12, wherein to perform the correction process comprises to: identify, by the second machine learning model, a first mathematical error in a portion of the mathematical characters without the mathematical units; generate an indication of the first mathematical error for presentation by the device; and present, on the device, the indication with the portion.

14. The system of claim 13, wherein the indication is presented with the portion during entry of second handwritten strokes by the user on the device.

15. The system of claim 14, wherein the portion represents a step in an answer to a mathematical question, and wherein the second handwritten strokes represent a second step of the mathematical question.

16. The system of claim 15, wherein the at least one processor is further configured to: distinguish, using the second machine learning model, between the step and a third step in the answer to the mathematical question.

17. The system of claim 13, wherein the indication further comprises a hint associated with correcting the first mathematical error.

18. The system of any of claim 11 or claim 12, wherein the at least one processor is further configured to: receive second handwritten strokes entered on the device by the user, the second handwritten strokes representing a correction of the handwritten strokes; identify second mathematical characters represented by the second handwritten strokes; determine, using the second machine learning model, that the second mathematical characters do not include a mathematical error; and presenting the second mathematical characters on the device without any indication of a mathematical error.

19. A computer-readable storage medium comprising instructions to cause at least one processor for analyzing and correcting handwritten math characters entered on a device, upon execution of the instructions by the at least one processor, to: receive handwritten strokes entered on a device by a user; identify mathematical characters represented by the handwritten strokes; input the mathematical characters into a first machine learning model configured to distinguish between mathematical units and other characters; identify, using the first machine learning model, first mathematical units of the mathematical characters; remove, by the at least one processor, the mathematical units from the mathematical characters;input the mathematical characters without the mathematical units into a second machine learning model configured to identify mathematical errors represented by characters; perform, by the second machine learning model, a verification process to identify mathematical errors represented by the mathematical characters; and perform an action based on the verification process.

20. The computer-readable storage medium of claim 19, wherein to perform the action comprises to: present the mathematical characters on a display, or perform a correction process based on identifying mathematical errors represented by the mathematical characters.

Images