Visual interaction method and system based on handwritten function recognition and image generation

Abstract

The invention belongs to the technical field of computers, and discloses a visual interaction method and system based on handwritten function recognition and image generation. The method comprises the following steps: firstly, designing a handwriting restoration algorithm by recording stroke track information in a handwriting function process to obtain a handwriting function original image; then, carrying out inverse gray scale transformation on the image to generate a gray scale inversion image; extracting features of the original image and the inverse grayscale image through a double-branch encoder structure, performing feature fusion, inputting the features into a multi-scale counting module to obtain a counting vector, and inputting the counting vector and the fused features into a decoder to obtain an identification result in a LaTeX format; and then designing a visual interactive interface to display the mathematical function image, and in response to interactive modification of the function variable by the user, updating the function image in real time. According to the method, the problem of attention deviation caused by hand drawing blank background interference can be improved, stroke feature extraction is enhanced through double-branch feature fusion, and an efficient solution is provided for the fields of education, engineering and the like.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and human-computer interaction, and particularly relates to a visual interaction method and system based on handwritten function recognition and image generation. BACKGROUND

[0002] Mathematical function expressions are the core information carriers in human scientific computing, engineering design and education teaching. With the popularity of touch devices and digital handwriting technology, handwritten function recognition technology has shown great potential in teaching assistance, scientific computing and engineering modeling fields.

[0003] However, the handwriting restoration image of the handwritten function expression has a small stroke proportion and a large blank background proportion, which can easily cause interference during feature extraction. At the same time, the existing system interaction process has affected the user experience. On the one hand, the user needs to manually copy the recognition result to the drawing tool, which breaks the continuity of handwriting recognition and immediate visualization. On the other hand, after the function image is generated, the user cannot directly modify the function variable parameters (such as coefficients, exponents) and observe the dynamic changes of the function in real time, which reduces the exploration efficiency.

[0004] The existing handwritten mathematical expression recognition public model CAN improves the recognition accuracy by jointly optimizing the symbol count and HMER, and uses DenseNet as the backbone network. The ExpRate on the CROHME 2019 dataset is 63.22%, which is 7.17% higher than the baseline network DWAP, but there is still room for improvement. In the inverse discriminative network (IDN), the inverse grayscale image is obtained by inverse grayscale transformation, and the extraction of signature strokes is strengthened by designing a neural network stream with shared weights and a multi-path attention module, but it has not been applied to the HMER field. SUMMARY

[0005] The present application aims to provide a visual interaction method and system based on handwritten function recognition and image generation to solve the above technical problems.

[0006] To solve the above technical problems, the specific technical solutions of the visual interaction method and system based on handwritten function recognition and image generation of the present application are as follows:

[0007] A visual interaction method based on handwritten function recognition and image generation, comprising the following steps:

[0008] Step 1: obtaining the stroke trajectory information of the handwritten function;

[0009] Step 2: obtaining the original image of the handwritten function through a handwriting restoration algorithm;

[0010] Step 3: inverse gray scale transformation is performed on the original image to generate a gray scale reversed image;

[0011] Step 4: the original image and the inverse gray scale image are input into the attention-based HMER encoder-decoder model in parallel to obtain a LaTeX format function recognition result;

[0012] Step 5: the LaTeX format function expression and the function image are displayed on the visualization interface;

[0013] Step 6: in response to the user's interactive modification operation on the variable in the function expression, the corresponding function image is generated and updated in real time, and the user can modify the handwritten input according to the visualization result to form a closed loop system.

[0014] Further, the step 1 comprises the following steps:

[0015] The user's handwritten input is captured in real time by an electromagnetic induction drawing board or a capacitive touch screen device, and the device records the space-time coordinate sequence of the pen tip. The sketch stroke trajectory information is obtained by recording the key point coordinates of the sketch drawn by the drawer on the hand-drawing board.

[0016] Further, the step 2 comprises the following steps:

[0017] The (x, y) coordinates of the stroke key points are obtained according to the stroke trajectory information, the coordinate normalization operation is performed, the white background image is created, the black handwriting is restored, and the original image is generated.

[0018] Further, the step 3 comprises the following steps:

[0019] Pixel-level gray scale inversion operation I is performed on the original image inverse =255-I original to obtain an inverse gray scale image with a black background and white handwriting.

[0020] Further, the step 4 comprises the following steps:

[0021] The attention-based HMER encoder-decoder model is modified according to the count-aware network CAN. The improved CAN model comprises a double-branch input layer, a shared weight encoder, a feature fusion module, a multi-scale count module MSCM, and a count-aware decoder CCAD, and specifically comprises the following steps:

[0022] Step 401: the original image and the inverse gray scale image are input into the double-branch input layer in parallel;

[0023] The double-branch input layer is constructed: branch one is an original image stream, and the original image is input into the encoder; branch two is an inverse gray scale image stream, and the inverse gray scale image is input into the same backbone network with shared weights;

[0024] Step 402: The encoder extracts features respectively and fuses the double-branch feature maps;

[0025] The encoder adopts DenseNet backbone network, branch one outputs feature map F f1 , branch two outputs feature map F f2 , and the double-path information is integrated through a feature fusion module: first, the SE module is used to perform channel attention weighting on F f1 and F f2 , respectively, to enhance key features; then the weighted features are spliced along the channel dimension; finally, 1x1 convolution is used to reduce the dimension to generate fusion feature F fused :

[0026] F fused =Conv1×1([SE(F f1 )⊕SE(F f2 )])

[0027] Step 403: input the fusion feature into the MSCM module;

[0028] The MSCM module is composed of multi-scale feature extraction, channel attention mechanism and sum-pooling operation. First, two parallel convolution branches are used to extract multi-scale features by using different kernel sizes. After the convolution layer, channel attention is used to further enhance the feature information. In one of the branches, let H∈R H×W×C be the feature map extracted from the convolution layer 3x3 or 5x5, and the enhanced feature S is represented as:

[0029] Q=σ(W1(G(H))+b1)

[0030]

[0031] where G represents global average pooling, σ and g(·) correspond to ReLU and sigmoid function respectively, representing the channel-level product, W1, W2, b1, b2 are trainable weights; after obtaining the enhanced feature S, 1x1 convolution is used to reduce the number of channels to C, where C represents the number of symbol classes; after the 1x1 convolution, a sigmoid function is used to generate a value in the range of 0 to 1 to generate a count map M∈R H×W×C , for each M i ∈R H×W , it should effectively reflect the position of the i-th symbol class, and then the sum-pooling operation is used to obtain the count vector V∈R 1×C ;

[0032] Step 404: based on the decoder, the fusion feature F fusedand the count vector V to get the LaTeX format of the function expression, and the fusion feature vector F is generated in the double-branch feature fusion module fused After that, F fused and V are decoded into the LaTeX sequence of the mathematical function expression.

[0033] Further, the step 5 includes the following steps:

[0034] The LaTeX format of the function expression obtained in step 4 is parsed and converted into the equivalent SymPy form based on the LaTeX2SymPy2 library; then different values of the function parameters are taken respectively, a series of string expressions are executed using the eval() function, and the corresponding expression values are returned; then the function graph is plotted based on matplotlib, and the GUI interface is designed based on PyQt5; finally, the drawn graph is embedded into the GUI interface through the matplotlib.backends.backend_qt5agg class.

[0035] Further, the step 6 includes the following steps:

[0036] The interactive interface is divided into four functional areas: the top handwritten input area, the central function image display area, and the bottom variable control area, forming a layout design that meets the ergonomics;

[0037] The system is integrated in a drawing board or a touch screen device, after the user writes the function expression in the handwritten input area, the recognition model displays the LaTeX format of the recognition result in the same box, and a default value is given to the function parameter, so that the corresponding initial image is displayed in the function image area, then the user modifies the value of any variable by dragging the slider or inputting in the box keyboard, when the user modifies the variable value, the program automatically clears the previous image and redraws the function image according to the new parameter value, achieving real-time updating effect, the user directly modifies the expression in the handwritten area according to the image effect, triggering the full process to be executed again, forming a closed-loop design iteration of “input → recognition → visualization → optimization”. The application also discloses a visualization interactive system based on handwritten function recognition and image generation, comprising: a data acquisition module, a data preprocessing module, a double-branch CAN recognition module, a visualization module, an interactive control module, and an image real-time updating module, wherein the data acquisition module acquires stroke trajectory data; the data preprocessing module obtains a handwritten function original image and an inverse grayscale image; the double-branch CAN recognition module processes the original image and the inverse grayscale image and outputs a LaTeX format of the function expression; the visualization module displays the function expression and the image; the interactive control module parses the variables and generates a slider and an input box; and the image real-time updating module updates the function image in real time based on the modified variable value.

[0038] The visual interaction method and system based on handwriting function recognition and image generation has the following advantages: the present application introduces the inverse gray processing technology into the HMER field, so that the network focuses on stroke structure instead of blank background interference, and the stroke features of the handwritten mathematical function are fully utilized; through the technical chain of "inverse gray preprocessing-double branch feature fusion-real-time interaction engine", the present application realizes the interactive system design of the whole process connection of "handwriting input-recognition-visualization-real-time interaction optimization", and provides an efficient solution for the fields of education and engineering. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The system flow block diagram suitable for the embodiment of the present application;

[0040] Figure 2 The double-branch CAN architecture based on the inverse gray transformation of the present application;

[0041] Figure 3 The visual interaction system interface schematic diagram based on handwriting function recognition and image generation of the embodiment of the present application;

[0042] Figure 4 The visual interaction system interface program flow block diagram based on handwriting function recognition and image generation of the embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to better understand the purpose, structure and function of the present application, the visual interaction method and system based on handwriting function recognition and image generation of the present application will be further described in detail below in combination with the drawings.

[0044] As shown in Figure 1 The visual interaction method based on handwriting function recognition and image generation of the present application comprises the following steps:

[0045] Step 1: obtaining stroke trajectory information of a handwritten function.

[0046] The user's handwriting input is captured in real time through an electromagnetic induction drawing board or a capacitive touch screen device. The device records the time-space coordinate sequence of the pen tip, and the key point coordinate of the sketch drawn by the drawer on the hand-drawing board is recorded to obtain the sketch stroke trajectory information.

[0047] Step 2: obtaining a handwritten function original image through a handwriting restoration algorithm.

[0048] According to the stroke trajectory information, the (x, y) coordinates of the stroke key points are obtained, the coordinate normalization operation is performed, the white background image is created, the black handwriting is restored, and the original image is generated.

[0049] Step 3: performing inverse gray transformation on the original image to generate a gray inversion image.

[0050] Considering a robust model encoder that can capture the features of hand-drawn strokes rather than the image itself, the same features should still be extracted when the grayscale values of the image are reversed. Therefore, a pixel-level grayscale inversion operation I inverse = 255 - I original is performed on the original image to obtain an inverse grayscale image with a black background and white strokes.

[0051] Step 4: Use an attention-based handwritten mathematical expression recognition (HMER) encoder-decoder model to input the original image and the inverse grayscale image into the recognition model in parallel to obtain the LaTeX format function recognition result.

[0052] The attention-based HMER encoder-decoder model is modified based on a counting-aware network (CAN). The improved CAN model includes a dual-branch input layer, a shared weight encoder, a feature fusion module, a multi-scale counting module (MSCM), and a counting-combined attentional decoder (CCAD), as shown in Figure 2 . Specifically, the following steps are included:

[0053] Step 401: Input the original image and the inverse grayscale image into the dual-branch input layer in parallel.

[0054] Construct the dual-branch input layer: Branch one is the original image stream, which inputs the original image into the encoder; Branch two is the inverse grayscale image stream, which inputs the inverse grayscale image into the same backbone network with shared weights.

[0055] Step 402: Construct the encoder to extract features and fuse the dual-branch feature maps.

[0056] The encoder uses a DenseNet backbone network, branch one outputs a feature map F f1 , and branch two outputs a feature map F f2 of the same dimension. The feature fusion module integrates the information of the two paths: First, the SE module is used to perform channel attention weighting on F f1 and F f2 to enhance key features; then the weighted features are concatenated along the channel dimension; finally, a 1x1 convolution is used to reduce the dimension to generate the fused feature F fused .

[0057]

[0058] Step 403: Input the fusion feature into the MSCM module.

[0059] The MSCM module is composed of multi-scale feature extraction, channel attention mechanism, and sum-pooling operation. Due to different writing habits, formula images usually contain symbols of various sizes. A single kernel size cannot effectively handle these scale changes. To this end, first, two parallel convolution branches are used to extract multi-scale features by using different kernel sizes (set to 3x3 and 5x5). After the convolution layer, channel attention is adopted to further enhance feature information. Here, one of the branches is selected for a simple description. Let H ∈ R H×W×C be the feature map extracted from the convolution layer (3x3 or 5x5). The enhanced feature S can be represented as:

[0060] Q = σ(W1(G(H)) + b1)

[0061]

[0062] where G represents global average pooling. σ and g(·) correspond to ReLU and sigmoid functions, respectively. represents the channel-level product, and W1, W2, b1, b2 are trainable weights. After obtaining the enhanced feature S, a 1x1 convolution is used to reduce the number of channels to C, where C represents the number of symbol classes. After the 1x1 convolution, a sigmoid function is used to generate a value in the range of 0 to 1 to generate a count map M ∈ R H×W×C . For each M i ∈ R H×W , it should effectively reflect the position of the i-th symbol class. Then, through sum-pooling operation, the count vector V ∈ R 1×C is obtained.

[0063] Step 404: Based on the decoder, the LaTeX format function expression is obtained from the fusion feature F fused and the count vector V.

[0064] After the dual-branch feature fusion module generates the fusion feature vector F fused , based on CCAD, F fused and V are decoded into LaTeX sequences of mathematical function expressions.

[0065] Step 5: Display the LaTeX format function expression and function image on the visualization interface.

[0066] The LaTeX format function expression obtained in step 4 is parsed based on the LaTeX2SymPy2 library and converted into an equivalent SymPy form; then, different values of function parameters are taken respectively, a series of string expressions are executed by using the eval() function, and the values of the corresponding expressions are returned; then, function graphs are drawn based on matplotlib, and a GUI interface is designed based on PyQt5; finally, the drawn graphs are embedded into the GUI interface through the matplotlib.backends.backend_qt5agg class.

[0067] Step 6: In response to the user's interactive modification operation of the variable in the function expression, the corresponding function image is generated and updated in real time, and the user can modify the handwritten input according to the visual result, forming a closed-loop system.

[0068] Figure 3 The interface of the visual interactive system based on handwritten function recognition and image generation of the application is shown in the figure. The interactive interface of the embodiment is divided into four functional areas: the top handwritten input area (also the recognition result display area), the central function image display area, the bottom variable control area (including a sliding bar and a digital input box), forming a layout design conforming to the ergonomics.

[0069] The system can be integrated in a drawing board or a touch screen device. After the user writes a function expression in the handwritten input area, the recognition model displays the LaTeX format recognition result in the same box, and gives a default value for the function parameters (such as k, a, b), so as to display the corresponding initial image in the function image area. Then, the user can modify the value of any variable by dragging the sliding bar or inputting in the form of a box keyboard. When the user modifies the variable value, the program automatically clears the previous image and redraws the function image according to the new parameter value, achieving the effect of real-time updating. The user can directly modify the expression in the handwritten area according to the image effect, triggering the re-execution of the whole process, forming a closed-loop design iteration of “input → recognition → visualization → optimization”, and greatly improving the efficiency of scientific exploration and teaching design.

[0070] The visual interactive system based on handwritten function recognition and image generation of the application comprises a data acquisition module, a data preprocessing module, a double-branch CAN recognition module, an interactive control module, and an image real-time updating module. The data acquisition module acquires stroke trajectory data; the data preprocessing module obtains a handwritten function original image and an inverse grayscale image; the double-branch CAN recognition module processes the original image and the inverse grayscale image and outputs a LaTeX format function expression; the visualization module displays the function expression and the image; the interactive control module parses the variable and generates a sliding bar and an input box; and the image real-time updating module updates the function image in real time based on the modified variable value. The above modules constitute a complete whole-process interactive system from handwritten input to dynamic visualization.

[0071] It is to be understood that the present application is described by way of example only, and that modifications or alterations can be made to the features and embodiments described without departing from the spirit and scope of the application. In addition, modifications can be made to the features and embodiments described to accommodate specific situations and materials without departing from the spirit and scope of the application. Accordingly, the application is not limited to the specific embodiments described herein, but rather only by the claims which follow, and their equivalents.

Claims

1. A visual interactive method based on handwritten function recognition and image generation, characterized in that, Includes the following steps: Step 1: Obtain the stroke trajectory information of the handwritten function; Step 2: Obtain the original image of the handwritten function using a handwriting reconstruction algorithm; Step 3: Perform an inverse grayscale transformation on the original image to generate a grayscale inverted image; Step 4: Using an attention-based HMER encoder-decoder model, the original image and the inverse grayscale image are input into the recognition model in parallel to obtain the function recognition result in LaTeX format; Step 5: Display the LaTeX format function expression and function graph on the visualization interface; Step 6: Respond to the user's interactive modification of variables in the function expression, generate and update the corresponding function graph in real time, and the user can modify the handwritten input based on the visualization results, forming a closed-loop system.

2. The visual interactive method based on handwritten function recognition and image generation according to claim 1, characterized in that, Step 1 includes the following steps: The device captures the user's handwriting input in real time using an electromagnetic induction drawing tablet or a capacitive touch screen device. The device records the spatiotemporal coordinate sequence of the pen tip and obtains the stroke trajectory information of the sketch by recording the coordinates of key points of the sketch on the drawing tablet.

3. The visual interactive method based on handwritten function recognition and image generation according to claim 1, characterized in that, Step 2 includes the following steps: Obtain the (x, y) coordinates of key points of the strokes based on the stroke trajectory information, perform coordinate normalization, create a white background image, restore the black strokes, and generate the original image.

4. The visual interactive method based on handwritten function recognition and image generation according to claim 1, characterized in that, Step 3 includes the following steps: Perform pixel-level grayscale inversion operation on the original image. inverse =255-I original This yields an inverse grayscale image of white handwriting on a black background.

5. The visual interactive method based on handwritten function recognition and image generation according to claim 1, characterized in that, Step 4 includes the following steps: The attention-based HMER encoder-decoder model is modified from the counting-aware network (CAN). The improved CAN model includes a dual-branch input layer, a shared-weight encoder, a feature fusion module, a multi-scale counting module (MSCM), and a counting-aware decoder (CCAD). The specific steps are as follows: Step 401: Input the original image and the inverse grayscale image into the dual-branch input layer in parallel; Construct a dual-branch input layer: Branch 1 is the original image stream, which inputs the original image into the encoder; Branch 2 is the inverse grayscale image stream, which inputs the inverse grayscale image into the same backbone network with shared weights. Step 402: Construct an encoder to extract features separately and fuse the dual-branch feature maps; The encoder uses a DenseNet backbone network, with branch 1 outputting feature map F. f1 Branch two output same-dimensional feature map F f2 The dual-path information is integrated through a feature fusion module: firstly, the SE module is used to integrate the F... f1 F f2 Perform channel attention weighting separately to enhance key features; Then, weighted features are concatenated along the channel dimension; finally, 1×1 convolution is used for dimensionality reduction to generate the fused feature F. fused : Step 403: Input the fused features into the MSCM module; The MSCM module consists of multi-scale feature extraction, channel attention mechanism, and summation pooling operation. First, it utilizes two parallel convolutional branches to extract multi-scale features by using different kernel sizes. After the convolutional layers, channel attention is used to further enhance the feature information. In one of the branches, let H∈R... H×W×C The enhanced feature S, derived from the feature map extracted from the 3×3 or 5×5 convolutional layer, is represented as: Q = σ(W1(G(H)) + b1) Where G represents global average pooling, and σ and g(·) correspond to the ReLU and sigmoid functions, respectively. Represents channel-level product, where W1, W2, b1, and b2 are trainable weights; After obtaining the enhanced feature S, a 1×1 convolution is used to reduce the number of channels to C, where C represents the number of symbol classes. After the 1×1 convolution, a sigmoid function is used to generate a value in the range of 0 to 1 to generate a counting map M∈R. H×W×C For each M i ∈R H×W It should effectively reflect the position of the i-th symbol class, and then the counting vector V∈R is obtained through summation and pooling operations. 1×C ; Step 404: Based on the decoder, fuse the features F fused The LaTeX function expression is obtained from the counting vector V, and the fused feature vector F is generated in the dual-branch feature fusion module. fused Then, based on CCAD, F fused V is decoded into a LaTeX sequence of mathematical function expressions.

6. The visual interactive method based on handwritten function recognition and image generation according to claim 1, characterized in that, Step 5 includes the following steps: The LaTeX format function expression obtained in step 4 is parsed using the LaTeX2SymPy2 library and converted into an equivalent SymPy form; then, different values ​​of the function parameters are taken, and a series of string expressions are executed using the eval() function, and the corresponding expression values ​​are returned. Then, function graphs were plotted using matplotlib, and a GUI interface was designed using PyQt5. Finally, the plotted graph is embedded into the GUI interface using the matplotlib.backends.backend_qt5agg class.

7. The visual interactive method based on handwritten function recognition and image generation according to claim 1, characterized in that, Step 6 includes the following steps: The interactive interface is divided into four functional areas: a top handwriting input area, a central function graph display area, and a bottom variable control area, forming a layout design that conforms to human-computer interaction. The system is integrated into a drawing tablet or touch screen device. After the user writes a function expression in the handwriting input area, the recognition model displays the recognition result in LaTeX format in the same box, and gives the function parameters a default value, so that the corresponding initial image is displayed in the function image area. Then, the user can modify the value of any variable by dragging the slider or by typing in the box. When the user modifies the variable value, the program automatically clears the previous image and redraws the function image according to the new parameter value, achieving a real-time update effect. The user can directly modify the expression in the handwriting area according to the image effect, triggering the entire process to be re-executed, forming a closed-loop design iteration of "input → recognition → visualization → optimization".

8. A visual interactive system based on handwritten function recognition and image generation, characterized in that, include: The system includes a data acquisition module, a data preprocessing module, a dual-branch CAN recognition module, a visualization module, an interactive control module, and a real-time image update module. The data acquisition module acquires stroke trajectory data; the data preprocessing module obtains the original image and inverse grayscale image of the handwritten function; the dual-branch CAN recognition module processes the original image and inverse grayscale image and outputs a function expression in LaTeX format; and the visualization module displays the function expression and image. The interactive control module parses the variables and generates sliders and input boxes; the real-time image update module updates the function image in real time based on the modified variable values.

Images