Handwritten mathematical formula recommendation system based on probability distribution and implementation method thereof

Abstract

The invention belongs to the technical field of handwriting recognition, and discloses a handwriting mathematical formula recommendation system based on probability distribution and an implementation method thereof.The handwriting mathematical formula recommendation system comprises a handwriting recognition module, a probability distribution analysis module, a candidate generation module, an interactive modification module, a confidence visualization module and a feedback learning module. And formula-level and character-level candidates are given, so that the accuracy is improved. The operation of changing the characters in the formula into other candidate characters through interaction modes such as long press / right key / gesture and the like is provided for the user, and the formula or symbol with low confidence coefficient is displayed by using different colors to assist the user in more effectively discovering identification errors. Through formula recognition accuracy and user interaction correction, a more suitable handwritten formula recognition result can be provided for the user, the situation that a single result output mode cannot correct the user is made up, and the recommendation result can be closer to the writing intention of the user.

Description

Technical Field

[0001] The present invention belongs to the technical field of handwriting recognition, and in particular relates to a handwritten mathematical formula recommendation system based on probability distribution and an implementation method thereof. Background Art

[0002] With the popularization of electronic devices and digital learning, the importance of handwritten math formula recognition technology has become increasingly prominent. However, existing handwritten math formula recognition systems usually only provide a single recognition result and lack diverse recommendation options. This single result output method is prone to recognition errors when the user's handwriting is blurred or irregular, which in turn affects the user experience. Therefore, how to improve the accuracy of handwritten math formula recognition and provide recommendation results that are more in line with the user's writing intentions has become a problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the present invention is to provide a handwritten mathematical formula recommendation system based on probability distribution and its implementation method to solve the above-mentioned technical problems.

[0004] In order to solve the above technical problems, the specific technical solutions of a handwritten mathematical formula recommendation system based on probability distribution and its implementation method of the present invention are as follows: A handwritten mathematical formula recommendation system based on probability distribution includes a handwriting recognition module, a probability distribution analysis module, a candidate generation module, an interactive modification module, a confidence visualization module and a feedback learning module; the handwriting recognition module is responsible for converting the mathematical formula written by the user into a digital mathematical expression; the probability distribution analysis module generates candidates at the formula and character levels according to the output of the recognition model, and calculates the confidence of each candidate; the candidate generation module provides multiple candidate recognition results, which are displayed at the formula level and the character level; the interactive modification module allows the user to modify the recognition result through an interactive interface; the confidence visualization module identifies the confidence level with different colors to help the user quickly find recognition errors; the feedback learning module records the user's modification behavior and uses it to optimize the recognition model.

[0005] Furthermore, the handwriting recognition module uses a deep learning model to recognize handwritten mathematical formulas input by the user. The recognition model is based on a combination of a convolutional neural network (CNN) and a recurrent neural network (RNN) to perform feature extraction and sequence modeling on handwritten characters and symbols. The following steps are specifically performed: preprocessing: denoising, binarization and normalization of the handwritten image input by the user; feature extraction: extracting spatial features of the handwritten image through CNN; sequence modeling: using RNN to model the character sequence and generate preliminary recognition results of the formula; probability distribution output: the recognition result is output in the form of a probability distribution, including the confidence of each character.

[0006] Furthermore, the probability distribution analysis module includes character-level candidates: for each recognized character, multiple candidate characters are provided and sorted by confidence; formula-level candidates: for the recognition result of the entire formula, multiple possible formula expressions are generated; confidence calculation: the confidence is calculated for each candidate, and the candidate with higher confidence is displayed first.

[0007] Furthermore, the candidate generation module includes formula candidates: displaying multiple possible complete formula expressions; character candidates: providing corresponding candidate characters for each recognized character; sorting and screening: sorting the candidates according to confidence, and placing the most likely candidate in the first place.

[0008] Furthermore, the interactive modification module includes long press modification: the user can view the candidate character list by long pressing the characters in the recognition result; right-click menu: on a desktop device, the user can call up the candidate character menu by right-clicking the character; gesture operation: on a touch screen device, the user can select the character to be modified through gesture operation; real-time update: when the user selects a candidate character, the system immediately updates the recognition result of the formula.

[0009] Furthermore, the confidence visualization module includes a default color: characters and formulas with high confidence are displayed in a default color; a warning color: characters and formulas with low confidence are displayed in a warning color; and a dynamic prompt: when a user clicks a character with low confidence, the system automatically pops up a list of candidate characters.

[0010] Furthermore, the feedback learning module includes modification records: recording the characters modified by the user each time and their corresponding candidate selections; model optimization: using the user's modification data to train the recognition model to better adapt to the user's writing habits; and adaptive learning: improving the system's recognition accuracy by continuously accumulating user feedback.

[0011] The present invention also discloses a method for implementing a handwritten mathematical formula recommendation system based on probability distribution, comprising the following steps: a user inputs a handwritten mathematical formula; a handwriting recognition module recognizes the formula and outputs a preliminary recognition result and a probability distribution; a probability distribution analysis module generates formula-level and character-level candidate items; a candidate generation module sorts the candidate items according to confidence and displays the candidate items; the user makes modifications through an interactive modification module, and the system updates the recognition results in real time; a feedback learning module records the user's modification behavior and optimizes the model.

[0012] Furthermore, the implementation method includes the following specific steps: Step 1: The user inputs a formula on the tablet or handwriting board input terminal, obtains the user's drawing trajectory and saves the data, saves the final drawing image, processes the image features, captures the image area of ​​the handwriting input on the device, and intercepts it as an image segment containing the complete formula, which is then uploaded to the cloud server for further processing; User input: The user writes a mathematical formula on the handwriting input board of the mobile device, and the device captures the handwriting trajectory in real time and generates a handwriting image; Image interception: The device identifies the boundary of the handwritten formula through image processing technology and intercepts the image area containing the complete formula; Image upload: The intercepted image is compressed to reduce transmission time, and the image is uploaded to the cloud server through the network for subsequent processing; Step 2: After the user writes the formula on the mobile terminal, the image is uploaded to the terminal, and the image is preprocessed and inferred; the preprocessing module includes preprocessing the image of the handwritten formula, and the artificial intelligence algorithm inference module uses the Densenet network to extract image features; Step 3: Use the deep learning algorithm and the pre-trained end-to-end algorithm to perform image reasoning and output the latex sequence. After reasoning the last fully connected layer, the tensor feature tensor is passed to the softmax classification layer. The softmax layer infers the probability of each symbol recognition, calculates the weighted sum, obtains character-level candidates, provides multiple candidate characters for each recognized character and sorts them. For formula-level candidates, multiple possible formula expressions are generated, and the confidence of the candidates is calculated and sorted. Step 4: Recommend the sorting results to the user through the hand-drawn software interface, generate and display candidate items based on the analysis results, display the formula option with the highest score on the front-end interface, hide the character candidates, call them through gestures, and display the most likely results first; Step 5. The user can view the character candidate list by long pressing the character, call out the hidden character candidates, and modify the character menu by sliding. After the user modifies, the system updates the recognition result in real time, and uses the confidence visualization function to help the user find recognition errors. The default black color is used for the characters with the highest confidence, and the red color is used for the characters with low confidence. When the user calls up the character candidate, the result is displayed; Step 6: After the user makes a selection, record the user behavior data, including categories and selections, and return the user's behavior and selection data and the collected data information to the database for information storage; store the images and user-confirmed results for subsequent training optimization and for the next recommendation.

[0013] The handwritten mathematical formula recommendation system based on probability distribution and its implementation method of the present invention have the following advantages: The present invention provides a handwritten mathematical formula recommendation system based on probability distribution and its implementation method, which uses the information provided by the recognition system to provide formula-level and character-level candidates to improve the accuracy. The system provides users with the operation of changing the characters in the formula to other candidate characters through interactive methods such as long press / right key / gesture, and displays formulas or symbols with low confidence in different colors to assist users in more effectively discovering recognition errors. Through the correction of formula recognition accuracy and user interaction, users can be provided with more suitable handwritten formula recognition results, making up for the situation where the single result output method cannot be corrected for the user, so that the recommended results can be closer to the user's writing intention. The system can make more targeted recommendations by rearranging the results obtained by the original image recognition algorithm, further improving recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 2 is a schematic diagram of a process framework of a handwritten mathematical formula recommendation system based on probability distribution according to an embodiment of the present invention; Figure 2 It is a schematic diagram showing the front-end interface of the present invention. DETAILED DESCRIPTION

[0015] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a handwritten mathematical formula recommendation system based on probability distribution and its implementation method of the present invention in conjunction with the accompanying drawings.

[0016] like Figure 1 Figure 2 As shown, a handwritten mathematical formula recommendation system based on probability distribution of the present invention includes a handwriting recognition module, a probability distribution analysis module, a candidate generation module, an interactive modification module, a confidence visualization module and a feedback learning module. Handwriting recognition module: responsible for converting the user's handwritten mathematical formula into a digital mathematical expression. Probability distribution analysis module: based on the output of the recognition model, generates candidates at the formula and character levels, and calculates the confidence of each candidate. Candidate generation module: provides multiple alternative recognition results, displayed at the formula level and character level. The confidence visualization module uses different colors to identify the confidence level, helping users to quickly find recognition errors. Interactive modification module: users can correct the recognition results through the interactive interface. Feedback learning module: records the user's modification behavior and uses it to optimize the recognition model.

[0017] Specifically, The handwriting recognition module uses a deep learning model to recognize handwritten mathematical formulas input by users. The recognition model is based on a combination of convolutional neural networks (CNN) and recurrent neural networks (RNN) to perform feature extraction and sequence modeling on handwritten characters and symbols. The specific steps include: Preprocessing: De-noising, binarization and normalization of the handwritten images input by users. Feature extraction: Extracting spatial features of handwritten images through CNN. Sequence modeling: Modeling character sequences using RNN to generate preliminary recognition results of formulas. Probability distribution output: The recognition results are output in the form of probability distribution, including the confidence of each character.

[0018] The probability distribution analysis module analyzes the probability distribution output by the handwriting recognition module and generates candidates at the formula level and character level. Character-level candidates: For each recognized character, multiple candidate characters are provided and sorted by confidence. Formula-level candidates: For the recognition result of the entire formula, multiple possible formula expressions are generated. Confidence calculation: The confidence is calculated for each candidate, and candidates with higher confidence are displayed first.

[0019] The candidate generation module generates candidates based on the output of the probability distribution analysis module and displays them to the user. Formula candidates: Display multiple possible complete formula expressions. Character candidates: For each recognized character, provide corresponding candidate characters. Sorting and filtering: Sort the candidates by confidence, and put the most likely candidate first.

[0020] The interactive modification module provides the user with the system's interactive functions. Long press modification: Users can view the candidate character list by long pressing the character in the recognition result. Right-click menu: On desktop devices, users can call up the candidate character menu by right-clicking the character. Gesture operation: On touch-screen devices, users can select the character to be modified through gesture operation. Real-time update: When the user selects a candidate character, the system immediately updates the recognition result of the formula.

[0021] The confidence visualization module uses different colors to indicate the confidence level, helping users to quickly find recognition errors. Default color: Characters and formulas with high confidence are displayed in the default color (such as black). Warning color: Characters and formulas with low confidence are displayed in a warning color (such as red). Dynamic prompt: When a user clicks a character with low confidence, the system automatically pops up a list of candidate characters.

[0022] The feedback learning module records the user's modification behavior and uses this data to continuously optimize the recognition model. Modification record: records the characters modified by the user each time and the corresponding candidate selection. Model optimization: The user's modification data is used to train the recognition model to better adapt to the user's writing habits. Adaptive learning: By continuously accumulating user feedback, the system's recognition accuracy is gradually improved.

[0023] The present invention provides a method for implementing a handwritten mathematical formula recommendation system based on probability distribution. The user inputs a handwritten mathematical formula. The handwriting recognition module recognizes the formula and outputs a preliminary recognition result and a probability distribution. The probability distribution analysis module generates candidates at the formula level and the character level. The candidate generation module sorts the candidates according to the confidence level and displays the candidates. The user makes modifications through the interactive modification module, and the system updates the recognition results in real time. The feedback learning module records the user's modification behavior and optimizes the model. Specifically, the method includes the following steps: Step 1. The user inputs the formula on the input terminal such as a tablet or handwriting board, obtains the user's drawing trajectory and saves the data, saves the final drawing image, and processes the image features. The device captures the image area of ​​the handwriting input and cuts it into an image fragment containing the complete formula. The image fragment is then uploaded to the cloud server for further processing. User input: The user writes a mathematical formula on the handwriting input board of the mobile device, and the device captures the handwriting trajectory in real time and generates a handwriting image. Image capture: The device uses image processing technology to identify the boundaries of the handwritten formula and cuts off the image area containing the complete formula. Image upload: The captured image is compressed to reduce transmission time. The image is uploaded to the cloud server via the network for subsequent processing.

[0024] Step 2: After the user writes the formula on the mobile terminal, the image is uploaded to the terminal for preprocessing and reasoning. The preprocessing module includes preprocessing the image of the handwritten formula, including grayscale (converting the color image to a grayscale image to reduce the interference of color information, using the grayscale algorithm to convert the color value of each pixel to a grayscale value), binarization (converting the grayscale image to a binary image to make the handwriting clearer, using the threshold method to set the pixel value greater than the threshold to white, and less than the threshold to black), and stroke erosion (performing stroke erosion operations on the binary image to remove small noise points and improve the cleanliness of the image, and using the morphological erosion algorithm to process the image) operations to improve recognition accuracy. The artificial intelligence algorithm reasoning module uses the Densenet network to extract image features.

[0025] Step 3: Use deep learning algorithms and pre-trained end-to-end algorithms to perform image reasoning and output latex sequences. After reasoning the last fully connected layer, the tensor feature tensor is passed to the softmax classification layer. The softmax layer infers the probability of each symbol recognition, calculates the weighted sum, obtains character-level candidates, and provides multiple candidate characters for each recognized character and sorts them. For formula-level candidates, multiple possible formula expressions are generated, and the confidence of the candidates is calculated and sorted.

[0026] Step 4: Recommend the sorting results to the user through the hand-drawn software interface, generate and display candidate items based on the analysis results, display the formula option with the highest score on the front-end interface, hide the character candidates, call them through gestures, and display the most likely results first.

[0027] Step 5. The user can view the character candidate list by long pressing the character, call out the hidden character candidates, and modify the character menu by sliding. After the user modifies, the system updates the recognition results in real time, and uses the confidence visualization function to help users find recognition errors. The default black color is used for the characters with the highest confidence, and red is used for the characters with low confidence. When the user calls up the character candidate, the result is displayed.

[0028] Step 6: After the user makes a selection, the user behavior data, including the category and selection, is recorded, and the user behavior and selection data and the collected data information are returned to the database for information storage. The image and the user's confirmation results are stored for subsequent training optimization and for the next recommendation.

[0029] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A handwritten mathematical formula recommendation system based on probability distribution, characterized in that: It includes a handwriting recognition module, a probability distribution analysis module, a candidate generation module, an interactive modification module, a confidence visualization module and a feedback learning module; the handwriting recognition module is responsible for converting the user's handwritten mathematical formula into a digital mathematical expression; The probability distribution analysis module generates candidates at the formula and character levels according to the output of the recognition model, and calculates the confidence of each candidate; the candidate generation module provides multiple candidate recognition results, which are displayed at the formula and character levels; the interactive modification module allows users to modify the recognition results through the interactive interface; The confidence visualization module uses different colors to mark the confidence level, helping users to quickly find recognition errors; the feedback learning module records the user's modification behavior and uses it to optimize the recognition model.

2. The handwritten mathematical formula recommendation system based on probability distribution according to claim 1, characterized in that: The handwriting recognition module uses a deep learning model to recognize the handwritten mathematical formulas input by the user. The recognition model is based on a combination of a convolutional neural network (CNN) and a recurrent neural network (RNN) to perform feature extraction and sequence modeling on handwritten characters and symbols. The following steps are specifically performed: Preprocessing: De-noising, binarization and normalization of the handwritten image input by the user; Feature extraction: Extract the spatial features of handwritten images through CNN; Sequence modeling: Use RNN to model the character sequence and generate the preliminary recognition result of the formula; Probability distribution output: The recognition result is output in the form of probability distribution, including the confidence of each character.

3. The handwritten mathematical formula recommendation system based on probability distribution according to claim 1, characterized in that: The probability distribution analysis module includes character-level candidates: for each recognized character, multiple candidate characters are provided and sorted by confidence; formula-level candidates: for the recognition result of the entire formula, multiple possible formula expressions are generated; confidence calculation: the confidence is calculated for each candidate, and the candidate with higher confidence is displayed first.

4. The handwritten mathematical formula recommendation system based on probability distribution according to claim 1, characterized in that: The candidate generation module includes formula candidates: displaying multiple possible complete formula expressions; character candidates: providing corresponding candidate characters for each recognized character; sorting and screening: sorting the candidates according to confidence and placing the most likely candidate in the first place.

5. The handwritten mathematical formula recommendation system based on probability distribution according to claim 1, characterized in that: The interactive modification module includes long press modification: the user can view the candidate character list by long pressing the character in the recognition result; Right-click menu: On desktop devices, users can right-click a character to invoke a menu of candidate characters; Gesture operation: On touch screen devices, users can use gestures to select the characters to be modified; Real-time update: When the user selects a candidate character, the system immediately updates the recognition results of the formula.

6. The handwritten mathematical formula recommendation system based on probability distribution according to claim 1, characterized in that: The confidence visualization module includes a default color: characters and formulas with high confidence are displayed in a default color; a warning color: characters and formulas with low confidence are displayed in a warning color; and a dynamic prompt: when a user clicks a character with low confidence, the system automatically pops up a list of candidate characters.

7. The handwritten mathematical formula recommendation system based on probability distribution according to claim 1, characterized in that: The feedback learning module includes modification records: recording the characters modified by the user each time and their corresponding candidate selections; model optimization: using the user's modification data to train the recognition model to better adapt it to the user's writing habits; Adaptive learning: Improve the recognition accuracy of the system by continuously accumulating user feedback.

8. A method for implementing a handwritten mathematical formula recommendation system based on probability distribution according to any one of claims 1 to 7, characterized in that: The method comprises the following steps: a user inputs a handwritten mathematical formula; a handwriting recognition module recognizes the formula and outputs a preliminary recognition result and probability distribution; a probability distribution analysis module generates formula-level and character-level candidate items; The candidate generation module sorts the candidates by confidence and displays them; Users can make changes through the interactive modification module, and the system will update the recognition results in real time; The feedback learning module records user modification behavior and optimizes the model.

9. The method for implementing the handwritten mathematical formula recommendation system based on probability distribution according to claim 8, characterized in that: The implementation method comprises the following specific steps: Step 1: The user inputs a formula on the tablet or handwriting board input terminal, obtains the user's drawing trajectory and saves the data, saves the final drawing image, processes the image features, captures the image area of ​​the handwriting input on the device, and intercepts it as an image segment containing the complete formula, which is then uploaded to the cloud server for further processing; User input: The user writes a mathematical formula on the handwriting input board of the mobile device, and the device captures the handwriting trajectory in real time and generates a handwriting image; Image interception: The device identifies the boundary of the handwritten formula through image processing technology and intercepts the image area containing the complete formula; Image upload: The intercepted image is compressed to reduce transmission time, and the image is uploaded to the cloud server through the network for subsequent processing; Step 2: After the user writes the formula on the mobile terminal, the image is uploaded to the terminal for preprocessing and reasoning; The preprocessing module includes preprocessing the image of the handwritten formula, and the artificial intelligence algorithm reasoning module uses the Densenet network to extract image features; Step 3: Use the deep learning algorithm and the pre-trained end-to-end algorithm to perform image reasoning and output the latex sequence. After reasoning the last fully connected layer, the tensor feature tensor is passed to the softmax classification layer. The softmax layer infers the probability of each symbol recognition, calculates the weighted sum, obtains character-level candidates, provides multiple candidate characters for each recognized character and sorts them. For formula-level candidates, multiple possible formula expressions are generated, and the confidence of the candidates is calculated and sorted. Step 4: Recommend the sorting results to the user through the hand-drawn software interface, generate and display candidate items based on the analysis results, display the formula option with the highest score on the front-end interface, hide the character candidates, call them through gestures, and display the most likely results first; Step 5. The user can view the character candidate list by long pressing the character, call out the hidden character candidates, and modify the character menu by sliding. After the user modifies, the system updates the recognition result in real time, and uses the confidence visualization function to help the user find recognition errors. The default black color is used for the characters with the highest confidence, and the red color is used for the characters with low confidence. When the user calls up the character candidate, the result is displayed; Step 6: After the user makes a selection, record the user behavior data, including categories and selections, and return the user's behavior and selection data and the collected data information to the database for information storage; store the images and user-confirmed results for subsequent training optimization and for the next recommendation.

Images