Technical method for extracting mathematical formula on image as latex formula

By using CNN and RNN networks to extract mathematical formulas in images in image processing, the problems of slow convergence and high computational cost in the prior art are solved, and fast and accurate formula recognition on low-computing equipment is achieved.

CN119068498APending Publication Date: 2024-12-03伏星成
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411042967.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, the image formula extraction model converges slowly, has high computing resource cost, poor robustness, and is not suitable for deployment of low computing power equipment.

Method used

Using the pytorch framework, the horizontal and vertical characteristics are obtained by imaging the image, and two sets of CNN networks and RNN neural networks are used, combined with the CTC loss function, and the mathematical formulas in the image are directly extracted, avoiding traditional position coding. and attention mechanisms reduce computational complexity.

Benefits of technology

The model is quickly converged and low-cost training is realized, which reduces the inference computing power demand, is suitable for deployment on low-computing equipment, and improves recognition efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068498A_ABST
    Figure CN119068498A_ABST
Patent Text Reader

Abstract

The invention relates to the field of formula extraction, in particular to a technical method for extracting a mathematical formula on an image into a latex formula, and aims to solve the problems that an existing model is slow in convergence, high in reasoning and training cost and not suitable for deployment of low-computing-power equipment. The method comprises the following steps of: loading picture data loaded with a mathematical formula, converting the picture data into an image matrix, and carrying out image processing on a picture. According to the method, after the attention mechanism is abandoned, training and reasoning are carried out at lower cost, the model convergence is faster, the robustness and accuracy of the model are higher even if the number of training samples is small, the computational power depended by the model reasoning is lower, rapid reasoning can be completed under the pure cpu condition, and the accuracy of the model reasoning is improved. According to the method, a formula recognition engine can be deployed to customers with offline and safety requirements, the calculation resource cost of overall training and reasoning of a formula recognition model can be reduced by using the technology, and unnecessary calculation resource waste is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of formula extraction, and in particular to a technical method for extracting mathematical formulas on an image into latex formulas. Background Art

[0002] This product is mainly implemented using the pytorch framework. First, the image is processed by image science, and then two groups of CNN neural networks are used, one to obtain horizontal features and the other to obtain vertical features. Finally, the horizontal and vertical feature matrices are merged and sent to the RNN neural network. The RNN recurrent neural network obtains the association of the sequence from the comprehensive features of the horizontal and vertical directions of the image, and updates the entire neural network through back propagation and other technologies. The CTC loss function (CTCLoss) is used to calculate the loss, and the cross entropy loss function can also be used. Generally speaking, the ctc loss function is better.

[0003] Most of the existing technologies use a CNN network to extract features from images, then position encode the images, calculate attention, and then send them to the RNN recurrent neural network, or directly put the CNN network into the Transformer network after extracting features to obtain the correct output sequence. The biggest problem with these methods is that the calculation process is very cumbersome, especially these solutions require attention calculation. This approach converges the model very slowly, especially when the number of samples is small, the robustness and accuracy are relatively low, the overall training and inference costs are high, and it is not suitable for deployment on low-computing power devices.

[0004] Aiming at the above problems, a technical method is proposed to extract mathematical formulas on images into latex formulas. Summary of the invention

[0005] The purpose of the present invention is to provide a technical method for extracting mathematical formulas on an image into latex formulas, and to use this device to work, thereby solving the problems in the above-mentioned background that the existing model converges very slowly, the computing resource cost of training and inference is high, the small sample training model has poor robustness, and is not suitable for deployment on low computing power devices.

[0006] To achieve the above object, the present invention provides the following technical solution: a technical method for extracting mathematical formulas on an image into latex formulas, comprising the following steps:

[0007] S1: Image loading: loading image data with mathematical formulas;

[0008] S2: Image conversion: convert image data into image matrix;

[0009] S3: Image processing: Perform image processing on the image, mainly enhancing the vertical and horizontal features of the image, so as to use neural networks to extract the sequence detail features of the image.

[0010] Furthermore, the specific steps of image conversion in S2 are as follows:

[0011] S201: Acquire image features in the horizontal direction of the image;

[0012] S202: Acquire image features in the vertical direction of the image.

[0013] Furthermore, the specific steps of obtaining the image features in the horizontal direction of the image in 201 are as follows:

[0014] S2011: Get the original image;

[0015] S2012: Rotate the original image 180 degrees;

[0016] S2013: Then horizontally merge the original image and the image rotated 180 degrees to obtain a picture in the horizontal direction of the image.

[0017] Furthermore, the specific steps of obtaining the image features in the vertical direction of the image in S202 are as follows:

[0018] S2021: obtaining an original image;

[0019] S2022: Rotate the original image 90 degrees;

[0020] S2023: Rotate another 180 degrees;

[0021] S2024: Merge the image rotated 90 degrees and the feature rotated 180 degrees.

[0022] Furthermore, the specific steps of image processing in S3 are as follows:

[0023] S301: After image processing, we will get two images, which are horizontal and vertical feature images. Next, we need to design two CNN (convolutional neural network) networks to process the horizontal and vertical features respectively.

[0024] S302: By setting these two convolutions, the width of the image is processed into an image matrix of 1.

[0025] Furthermore, the convolution formula in S301 is as follows:

[0026] s(i,j)=(X*W)(i,j)=∑ m ∑ n x(i+m,j+n)w(m,n);

[0027] -s(i,j) is the value of the output feature map (also called activation map) at position (i,j);

[0028] -X represents the input image or the feature map of the previous layer, and x(i+m,j+n) is the pixel value of the input image at position (i+m,j+n);

[0029] -W or w(m,n) represents the convolution kernel (filter), which is a small matrix used to scan the entire input image;

[0030] -Double summation symbol ∑ m ∑ n It represents the weighted summation of all elements of the convolution kernel, where m and n are the indices of the convolution kernel in the horizontal and vertical directions, respectively.

[0031] Furthermore, in S302, the specific steps of processing the width of the image into an image matrix of 1 through two convolutions are as follows:

[0032] S3021: Convolution operation: Let the input image be X∈R 3×28×28 , we first apply the first convolutional layer and set the convolution kernel of the first convolutional layer to be K1∈R 3×k×k , where k is the size of the convolution kernel, usually an odd number (such as 3 or 5). Assuming the number of output channels is C1, the output of the first convolution layer is Y1;

[0033]

[0034] Where l is the output channel index from 0 to C1-1, i and j are the row and column indices of the output feature map;

[0035] S3022: Width compression: Next, the second convolution layer K2 is used to compress the width. Assume that K2 is a convolution kernel of C1×1×28 and its output channel number is C2=512. At this time, the output Y2 shape is (512,1,28).

[0036]

[0037] Where l ranges from 0 to 511, representing the output channel index, and i and j are still the row and column indices of the output feature map;

[0038] S3023: Matrix transformation: transform the matrix of (512, 1, 28) into (28, 512); mathematically, this is equivalent to the transposition operation of the matrix;

[0039] Let Z∈R 512×1×28 is a reshaped version of Y2, then the deformed matrix Z' is: Z'(i,j) = Z(j,0,i), where i ranges from 0 to 27, and j ranges from 0 to 511;

[0040] S3024: Prepare the Latex code corresponding to the image, such as:

[0041] \frac{1}{2^{2}}+\frac{1}{3^{2}}+\cdots+\frac{1}{n^{2}}<\frac{1}{2}\left(1-\frac{1}{3}+\frac{1}{2}-\frac{1}{4}+\fra c{1}{3}-\frac{1}{5}+\cdots+\frac{1}{n-1}-\frac{1}{n+1}\righ t);

[0042] S3025: Then perform one-hot encoding on this latex code.

[0043] Furthermore, the specific steps of one-hot encoding in S3025 are as follows:

[0044] S30251: Convert the latex code to an index dictionary by encoding characters in order from 0 to the length of latex;

[0045] S30252: By using this dictionary to loop through the latex code, and matching the dictionary code with the characters in latex, you can get the serial digital code;

[0046] S0253: After getting the sequence digital code, the feature matrix processed by the CNN neural network can be put into the RNN recurrent neural network, and the RNN recurrent neural network will find the association between the feature and the target sequence;

[0047] S30254: The losses of the actual sequence and the target sequence are obtained by using forward propagation. The CTCLoss loss function is used here, and then the weight matrices of the CNN neural network and the RNN neural network are updated by back propagation and chain derivation techniques.

[0048] S30255: After several rounds of training, our neural network can extract formulas from images into latex code.

[0049] Furthermore, the forward propagation in S30254 is specifically explained as follows: Assume that we have a multilayer perceptron (MLP) consisting of an input layer, a hidden layer, and an output layer; each layer consists of multiple neurons, each neuron has its own weights and biases, and the back propagation algorithm in S30254 is specifically explained as follows: The back propagation algorithm is an algorithm used to calculate the gradients of weights and biases in a neural network; it is based on the chain rule and propagates errors backwards from the output layer to update the weights and biases in the network.

[0050] Furthermore, the basic formula of the recurrent neural network (RNN) in S30253 mainly involves state update and output calculation.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention abandons the traditional solution based on position encoding and attention mechanism. It rotates and merges the horizontal and vertical features of the image and hands them over to two groups of CNN networks to obtain the horizontal and vertical features of the image respectively, and then merges the features. Here, the CNN network is used to obtain the features of the image and split the image into multiple small blocks with a width of 1. Each small block corresponds to 512 features, and each small block is in order. In this way, it can be directly handed over to the RNN network for processing, and the sequence association can be correctly obtained from the wide features of the image, and the correct sequence can be output finally. After abandoning the attention mechanism, training and reasoning can be carried out at a lower cost, the model training converges faster, and the computing power relied on by reasoning is lower. Fast reasoning can be completed in the case of pure CPU. We can deploy the formula recognition engine to some customers with offline and security requirements, and we can also use this technology to reduce reasoning costs and reduce unnecessary energy waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0054] Figure 2 It is a schematic diagram of the specific process of picture conversion in the present invention;

[0055] Figure 3 It is a schematic diagram of the process of obtaining the image features in the horizontal direction of the image in the present invention;

[0056] Figure 4 A schematic diagram of the process of obtaining image features in the vertical direction of an image in the present invention;

[0057] Figure 5 It is a schematic diagram of the specific process of image processing in the present invention;

[0058] Figure 6 It is a specific flow chart of processing the image matrix with the width of the image being 1 in the present invention;

[0059] Figure 7 It is a schematic diagram of a specific process of unique hot encoding in the present invention;

[0060] Figure 8 is a schematic diagram of the original picture in the present invention;

[0061] Fig. 9 A schematic diagram of rotating an original image by 180 degrees in the present invention;

[0062] Fig.10 A schematic diagram of horizontally merging an original image and an image rotated 180 degrees in the present invention;

[0063] Fig.11 A schematic diagram of rotating an original image by 90 degrees in the present invention;

[0064] Fig.12 It is a schematic diagram of the present invention rotated 180 degrees;

[0065] Fig.13 It is a schematic diagram of merging an image rotated 90 degrees and a feature rotated 180 degrees in the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] In order to solve the problems that the existing models of existing technologies converge very slowly, have high inference and training costs, and are not suitable for deployment on low-computing power devices, such as Figure 1 - Figure 5 and Figure 8 - Fig.13 As shown, the following preferred technical solutions are provided:

[0068] A technical method for extracting mathematical formulas on an image into latex formulas includes the following steps:

[0069] S1: Image loading: loading image data with mathematical formulas;

[0070] S2: Image conversion: convert image data into image matrix;

[0071] S3: Image processing: Perform image processing on the image, mainly enhancing the vertical and horizontal features of the image, so as to use neural networks to extract the sequence detail features of the image.

[0072] First, we need to load the image data with mathematical formulas and convert it into an image matrix with a matrix shape of (number of color channels, width, height). Then we need to perform image processing on the image. Mainly, we need to enhance the vertical and horizontal features of the image to facilitate us to use neural networks to extract the sequence detail features of the image.

[0073] The specific steps of image conversion in S2 are as follows:

[0074] S201: Acquire image features in the horizontal direction of the image;

[0075] S202: Acquire image features in the vertical direction of the image.

[0076] By performing image processing on the image, one is to obtain horizontal features, and the other is to obtain vertical features.

[0077] The specific steps of obtaining the image features in the horizontal direction of the image in S201 are as follows:

[0078] S2011: Get the original image;

[0079] S2012: Rotate the original image 180 degrees;

[0080] S2013: Then horizontally merge the original image and the image rotated 180 degrees to obtain a picture in the horizontal direction of the image.

[0081] After merging the horizontal feature matrices, they are sent to the RNN neural network.

[0082] The specific steps of obtaining the image features in the vertical direction of the image in S202 are as follows:

[0083] S2021: obtaining an original image;

[0084] S2022: Rotate the original image 90 degrees;

[0085] S2023: Rotate another 180 degrees;

[0086] S2024: Merge the image rotated 90 degrees and the feature rotated 180 degrees.

[0087] After merging the vertical feature matrices, they are sent to the RNN neural network.

[0088] The specific steps of image processing in S3 are as follows:

[0089] S301: After image processing, we will get two images, which are horizontal and vertical feature images. Next, we need to design two CNN (convolutional neural network) networks to process the horizontal and vertical features respectively.

[0090] S302: By setting these two convolutions, the width of the image is processed into an image matrix of 1.

[0091] For example: the shape of the image matrix is ​​(3, 28, 28). We need to use these two convolutions to transform the image into (512, 1, 28), where 512 is the feature pixel map obtained by the convolution operation, 1 is the width, and 28 is the height. At this time, we need to perform a deformation operation on the image matrix and transform the matrix into (28, 512), that is, there are 28 image sequence features, and each sequence corresponds to 512 feature descriptions.

[0092] In this solution: the image is processed by image processing, and then two groups of CNN neural networks are used, one to obtain horizontal features and the other to obtain vertical features. Finally, the horizontal and vertical feature matrices are merged and sent to the RNN neural network. The RNN recurrent neural network obtains the sequence association from the comprehensive features of the horizontal and vertical directions of the image, and updates the entire neural network through back propagation and other technologies.

[0093] like Figure 6 As shown, the following preferred technical solutions are provided:

[0094] The convolution formula in S301 is as follows:

[0095] s(i,j)=(X*W)(i,j)=∑ m ∑ n x(i+m,j+n)w(m,n);

[0096] -s(i,j) is the value of the output feature map (also called activation map) at position (i,j);

[0097] -X represents the input image or the feature map of the previous layer, and x(i+m,j+n) is the pixel value of the input image at position (i+m,j+n);

[0098] -W or w(m,n) represents the convolution kernel (filter), which is a small matrix used to scan the entire input image;

[0099] -Double summation symbol ∑ m ∑ n It represents the weighted summation of all elements of the convolution kernel, where m and n are the indices of the convolution kernel in the horizontal and vertical directions, respectively.

[0100] The specific steps of processing the width of the image into an image matrix of 1 through two convolutions in S302 are as follows:

[0101] S3021: Convolution operation: Let the input image be X∈R 3×28×28 , we first apply the first convolutional layer and set the convolution kernel of the first convolutional layer to be K1∈R 3×k×k, where k is the size of the convolution kernel, usually an odd number (such as 3 or 5). Assuming the number of output channels is C1, the output of the first convolution layer is Y1;

[0102]

[0103] Where l is the output channel index from 0 to C1-1, i and j are the row and column indices of the output feature map;

[0104] S3022: Width compression: Next, the second convolution layer K2 is used to compress the width. Assume that K2 is a convolution kernel of C1×1×28 and its output channel number is C2=512. At this time, the output Y2 shape is (512,1,28).

[0105]

[0106] Where l ranges from 0 to 511, representing the output channel index, and i and j are still the row and column indices of the output feature map;

[0107] S3023: Matrix transformation: transform the matrix of (512, 1, 28) into (28, 512); mathematically, this is equivalent to the transposition operation of the matrix;

[0108] Let Z∈R 512×1×28 is a reshaped version of Y2, then the deformed matrix Z' is: Z'(i,j) = Z(j,0,i) where i ranges from 0 to 27 and j ranges from 0 to 511;

[0109] S3024: Prepare the Latex code corresponding to the image, such as:

[0110] \frac{1}{2^{2}}+\frac{1}{3^{2}}+\cdots+\frac{1}{n^{2}}<\frac{1}{2}\left(1-\frac{1}{3}+\frac{1}{2}-\frac{1}{4}+\fra c{1}{3}-\frac{1}{5}+\cdots+\frac{1}{n-1}-\frac{1}{n+1}\righ t);

[0111] S3025: Then perform one-hot encoding on this latex code.

[0112] In this solution: the horizontal and vertical features of the image are rotated and merged, and then handed over to two groups of CNN networks to obtain the horizontal and vertical features of the image respectively, and then merge the features. Here, the CNN network is used to obtain the features of the image and split the image into multiple small blocks with a width of 1. Each small block corresponds to 512 features, and each small block is in order. In this way, it can be directly handed over to the RNN network for processing, and the sequence association can be correctly obtained from the wide features of the image, and finally the correct sequence is output.

[0113] like Figure 7 As shown, the following preferred technical solutions are provided:

[0114] The specific steps of one-hot encoding in S3025 are as follows:

[0115] S30251: Convert the latex code to an index dictionary by encoding characters in order from 0 to the length of latex;

[0116] S30252: By using this dictionary to loop through the latex code, and matching the dictionary code with the characters in latex, you can get the serial digital code;

[0117] S30253: After getting the sequence digital code, the feature matrix processed by the CNN neural network can be put into the RNN recurrent neural network, and the RNN recurrent neural network will find the association between the feature and the target sequence;

[0118] S30254: The losses of the actual sequence and the target sequence are obtained by using forward propagation. The CTCLoss loss function is used here, and then the weight matrices of the CNN neural network and the RNN neural network are updated by back propagation and chain derivation techniques.

[0119] S30255: After several rounds of training, our neural network can extract formulas from images into latex code.

[0120] The forward propagation in S30254 in this solution is specifically explained as follows: Assume that we have a multilayer perceptron (MLP) consisting of an input layer, a hidden layer, and an output layer; each layer consists of multiple neurons, each with its own weight and bias; for the lth layer, its forward propagation process can be expressed as:

[0121] 1. Weighted sum: The input of each neuron is the dot product of the output (activation value) of all neurons in the previous layer and the weight of the neuron, plus the bias. That is, for the jth neuron in the lth layer, its weighted sum is It can be expressed as: in, is the weight from the i-th neuron in the l-1th layer to the j-th neuron in the lth layer, is the bias term of the jth neuron, and is the activation value of the i-th neuron in the l-1 layer;

[0122] 2. Activation function: After the weighted sum, a nonlinear activation function σ is usually applied to introduce nonlinearity so that the model can fit more complex functions. The activation value of the jth neuron in the lth layer is Defined as:

[0123] This process starts from the input layer and proceeds layer by layer until it reaches the output layer. At the output layer, the choice of activation function depends on the type of problem (for example, for classification problems, the softmax function may be used; for regression problems, a linear function may be used directly);

[0124] After the forward propagation process is completed, we get the network's predicted output for a given input. Next, we can calculate the value of the loss function by comparing the predicted output with the actual target, and then use the backpropagation algorithm to calculate the gradient of the loss function with respect to the network parameters, and finally use the gradient descent algorithm to update these parameters to minimize the loss function;

[0125] The back propagation algorithm in S30254 is specifically explained as follows: The back propagation algorithm is an algorithm for calculating the gradients of weights and biases in a neural network; it is based on the chain rule and propagates errors backward from the output layer to update the weights and biases in the network; for the neuron j in the lth layer, its gradient (partial derivative) with respect to the loss function L can be expressed as:

[0126]

[0127] in:

[0128]

[0129] - is the activation value of the previous layer;

[0130] -σ'(z) is the derivative of the activation function σ(z);

[0131] - is the weight from the i-th neuron to the j-th neuron in the l-th layer;

[0132] Gradient descent method: Gradient descent is an optimization algorithm used to minimize the loss function. It iteratively updates the parameters by adjusting the parameters along the negative direction of the gradient to find the minimum value of the loss function. The update rule is as follows:

[0133]

[0134] in:

[0135] -α is the learning rate, which controls the step size of each update;

[0136] It should be noted here that the partial derivatives in the above formula are calculated by the back-propagation algorithm;

[0137] The basic formula of the recurrent neural network (RNN) in S30253 mainly involves state update and output calculation; the following is a basic single-layer RNN formula:

[0138] 1. State update: Assume that we have a sequence data, where the data at each time step is represented as x t At each time step t, RNN uses the state h of the previous moment t-1 and the input x at the current time step t To calculate the state h of the current time step t .this

[0139] The process can be expressed by the following formula: t =σ(W hh h t-1 +W xh x t +b h );

[0140] in:

[0141] -W hh is the weight matrix connecting the previous state to the current state;

[0142] -W xh is the weight matrix connecting the input to the state;

[0143] -b h is the bias vector for state update;

[0144] -σ is a nonlinear activation function, such as Sigmoid, Tanh or ReLU;

[0145] 2. Output calculation: The output of RNN can be based on the current state h t The formula of the output layer is as follows: t =W hy h t +b y ;

[0146] in:

[0147] -W hy is the weight matrix from state to output;

[0148] -b y is the bias vector of the output layer;

[0149] 3. Multi-layer RNN: In a multi-layer RNN, each layer has its own state update formula, but usually the bottom layer receives the original input x t , while the top layer directly produces the output y t The state update formula for the middle layer is basically the same as above, except that the state from the previous layer is used instead of the original input.

[0150] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0151] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A technical method to extract mathematical formulas on images into latex formulas. It is characterized in that The following steps are involved: S1: Image loading: loading image data with mathematical formulas; S2: Image conversion: convert image data into image matrix; S3: Image processing: Perform image processing on the image, mainly enhancing the vertical and horizontal features of the image, so as to use neural networks to extract the sequence detail features of the image.

2. According to claim 1, a technical method for extracting mathematical formulas on an image into latex formulas, Features: The specific steps of image conversion in S2 are as follows: S201: Acquire image features in the horizontal direction of the image; S202: Acquire image features in the vertical direction of the image.

3. According to claim 2, a technical method for extracting mathematical formulas on an image into latex formulas, Features: The specific steps of obtaining the image features in the horizontal direction of the image in S201 are as follows: S2011: Get the original image; S2012: Rotate the original image 180 degrees; S2013: Then horizontally merge the original image and the image rotated 180 degrees to obtain a picture in the horizontal direction of the image.

4. A technical method for extracting mathematical formulas on an image into latex formulas according to claim 3, Features: The specific steps of obtaining the image features in the vertical direction of the image in S202 are as follows: S2021: obtaining an original image; S2022: Rotate the original image 90 degrees; S2023: Rotate another 180 degrees; S2024: Merge the image rotated 90 degrees and the feature rotated 180 degrees.

5. According to claim 4, a technical method for extracting mathematical formulas on an image into latex formulas, Features: The specific steps of image processing in S3 are as follows: S301: After image processing, we will get two images, which are horizontal and vertical feature images. Next, we need to design two CNN (convolutional neural network) networks to process the horizontal and vertical features respectively. S302: By setting these two convolutions, the width of the image is processed into an image matrix of 1.

6. A technical method for extracting mathematical formulas on an image into latex formulas according to claim 5, Features: The convolution formula in S301 is as follows: s(i,j)=(X*W)(i,j)=∑ m ∑ n x(i+m,j+n)w(m,n); -s(i,j) is the value of the output feature map (also called activation map) at position (i,j); -X represents the input image or the feature map of the previous layer, and x(i+m,j+n) is the pixel value of the input image at position (i+m,j+n); -W or w(m,n) represents the convolution kernel (filter), which is a small matrix used to scan the entire input image; -Double summation symbol ∑ m ∑ n It represents the weighted summation of all elements of the convolution kernel, where m and n are the indices of the convolution kernel in the horizontal and vertical directions, respectively.

7. The method of extracting mathematical formulas on an image into latex formulas according to claim 6, Features: The specific steps of processing the width of the image into an image matrix of 1 through two convolutions in S302 are as follows: S3021: Convolution operation: Let the input image be X∈R 3×28×28 , we first apply the first convolutional layer and set the convolution kernel of the first convolutional layer to K 1 ∈R 3×k×k , where k is the size of the convolution kernel, usually an odd number (such as 3 or 5). Assume that the number of output channels is C 1 , then the output of the first convolutional layer is Y 1 ; Where l is from 0 to C 1 -1 is the output channel index, i and j are the row and column indices of the output feature map; S3022: Width Compression: Next, use the second convolutional layer K 2 To compress the width, assuming K 2 It is C 1 ×1×28 convolution kernel, the number of output channels is C 2 =512; the output Y at this time 2 The shape is (512, 1, 28); Where l ranges from 0 to 511, representing the output channel index, and i and j are still the row and column indices of the output feature map; S3023: Matrix transformation: transform the matrix of (512, 1, 28) into (28, 512); Mathematically, this is equivalent to the transpose operation of the matrix; Let Z∈R 512×1×28 Yes 2 The reshaped version of , the deformed matrix Z' is: Z'(i,j)=Z(j,0,i) where i ranges from 0 to 27 and j ranges from 0 to 511; S3024: Prepare the Latex code corresponding to the image, such as: \frac{1}{2^{2}}+\frac{1}{3^{2}}+\cdots+\frac{1}{n^{2}}<\frac{1}{2}\left(1-\frac{1}{3}+\frac{1}{2}-\frac{1}{4}+\fra c{1}{3}-\frac{1}{5}+\cdots+\frac{1}{n-1}-\frac{1}{n+1}\righ t); S3025: Then perform one-hot encoding on this latex code.

8. A technical method for extracting mathematical formulas on an image into latex formulas according to claim 7, Features: The specific steps of one-hot encoding in S3025 are as follows: S30251: Convert the latex code to an index dictionary by encoding characters in order from 0 to the length of latex; S30252: By using this dictionary to loop through the latex code, and matching the dictionary code with the characters in latex, you can get the serial digital code; S30253: After getting the sequence digital code, the feature matrix processed by the CNN neural network can be put into the RNN recurrent neural network, and the RNN recurrent neural network will find the association between the feature and the target sequence; S30254: The losses of the actual sequence and the target sequence are obtained by using forward propagation. The CTCLoss loss function is used here, and then the weight matrices of the CNN neural network and the RNN neural network are updated by back propagation and chain derivation techniques. S30255: After several rounds of training, our neural network can extract formulas from images into latex code.

9. The method of extracting mathematical formulas on an image into latex formulas according to claim 8, Features: Forward Propagation in S30254: Assume that we have a multilayer perceptron (MLP) with an input layer, hidden layer, and output layer; each layer consists of multiple neurons, each with its own weights and biases, Backward Propagation Algorithm in S30254: The backpropagation algorithm is an algorithm used to calculate the gradients of weights and biases in a neural network; it is based on the chain rule and propagates the error backwards starting from the output layer to update the weights and biases in the network.

10. The method of extracting mathematical formulas on an image into latex formulas according to claim 9, Features: The basic formula of the recurrent neural network (RNN) in S30253 mainly involves state update and output calculation.