Formula identification method and device and computer readable storage medium

Abstract

The invention discloses a formula recognition method and device and a computer readable storage medium. The method comprises the steps that hidden variables are extracted from a picture containing a formula; packaging the hidden variables to obtain an initial formula sequence; and unpacking the initial formula sequence to obtain a complete formula sequence. According to the scheme of the invention, the formula sequence is packaged and unpackaged, so that the analysis capability of the model on the formula structure in formula identification can be improved.

Description

Technical Field

[0001] The present application generally relates to the field of artificial intelligence technology. More specifically, the present application relates to a method, device and computer-readable storage medium for recognizing formulas. Background Art

[0002] Formula recognition is an important branch of computer vision and artificial intelligence, which focuses on parsing and understanding mathematical formulas written by humans. The core purpose of this technology is to convert handwritten mathematical expressions into a format that can be processed by computers. This is of great significance in the fields of education, scientific research, and automated document processing, especially in automatic grading systems, where it has great potential commercial value.

[0003] With the development of deep learning technology, the efficiency and accuracy of handwritten formula recognition have been significantly improved. Modern methods usually use a model that combines convolutional neural networks (CNN) and recurrent neural networks (RNN), or more generally, a combination of encoders (usually used to process images) and decoders (usually used to process sequence outputs) with attention mechanisms to handle recognition tasks. These models can effectively handle complex image features and long-term dependencies, thereby improving recognition accuracy. Although the attention corresponding to a single character in the image is relatively reasonable, the attention maps of structural invisible characters in LaTeX, such as '{', '}', '_', and '^', are very scattered and distributed in areas that have no logical relationship with the prediction of this character. This is probably one of the reasons for the error in predicting structural related symbols in formula recognition, that is, the attention map is unable to cope with the situation where a large number of invisible characters appear in complex structures.

[0004] In view of this, there is an urgent need to provide a method for recognizing formulas in order to improve the model's ability to parse the formula structure, enhance the distribution of the model's attention on structural symbols, and improve the generalization ability between different writing styles, so as to handle more complex or irregular handwritten formulas. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present application proposes a solution for efficiently identifying a formula in the following aspects.

[0006] In a first aspect, the present application provides a method for identifying a formula, comprising: extracting latent variables from a picture containing a formula; packing the latent variables to obtain an initial formula sequence; and unpacking the initial formula sequence to obtain a complete formula sequence.

[0007] In some embodiments, the image includes an image of a mathematical formula written by a human.

[0008] In some other embodiments, the initial formula sequence includes one or more of structural symbols, characters and structural information.

[0009] In yet other embodiments, the complete sequence of formulas includes mathematical expressions that can be processed by a computer.

[0010] In some further embodiments, the format of the mathematical expression includes LaTeX or MathML.

[0011] In some other embodiments, packing the hidden variables to obtain an initial formula sequence includes: decoding the hidden variables and the start symbol to obtain the first layer of structural symbols and characters outside the structural symbols of the initial formula sequence; packing the first layer of structural symbols and the information therein into structural information; and merging the characters and the structural information to obtain the initial formula sequence.

[0012] In some other embodiments, unpacking the initial formula sequence to obtain a complete formula sequence includes: decoding the initial formula sequence and hidden variables to extract the first-level structure symbols, the second-level structure symbols and the characters between the two-level structure symbols of the structural information in the initial formula sequence; packing the second-level structure symbols and the information therein as structural information; using the first-level structure symbols, the characters between the two-level structure symbols and the structural information to update the formula sequence; replacing the initial formula sequence with the updated formula sequence for iterative unpacking of the next cycle with the hidden variables; and when the second-level structure symbols are empty, using the first-level structure symbols and characters to update the formula sequence to obtain a complete formula sequence.

[0013] In some further embodiments, the mathematical operation priority of the second-level structure symbol is higher than the mathematical operation priority of the first-level structure symbol.

[0014] In some other embodiments, the updating of the formula sequence includes: retaining the characters and structural symbols passed into the formula sequence before decoding; replacing the structural information with the first-level structural symbols extracted after decoding, the characters and structural information between the two-level structural symbols; and when the second-level structural symbols after decoding are empty, using the first-level structural symbols and characters to replace the structural information before decoding.

[0015] In a second aspect, the present application provides a device for identifying a formula, comprising: a processor; and a memory on which computer instructions for identifying a formula are stored, wherein when the computer instructions are executed by the processor, any method described in the first aspect is implemented.

[0016] In a third aspect, the present application provides a computer-readable storage medium having stored thereon computer program instructions for identifying a formula, wherein when the computer program instructions are executed by one or more processors, the method described in any one of the first aspects is implemented.

[0017] Through a method, device and computer-readable storage medium for recognizing formulas as provided above, the embodiments of the present application can improve the model's ability to parse the formula structure by packing and unpacking the formula sequence. Further, in some embodiments, by simultaneously decoding latent variables and sequence information, the distribution of the model's attention on structural symbols can be enhanced. Furthermore, in some embodiments, by iterative loop updates, the generalization ability between different writing styles can be improved, thereby processing more complex or irregular handwritten formulas. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0019] Figure 1 A schematic diagram of a flow chart of a method for identifying a formula according to an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram of a process for packaging a formula sequence according to an embodiment of the present application is shown;

[0021] Figure 3 A schematic diagram of a process for unpacking a formula sequence according to an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of the overall architecture of a model for identifying a formula according to an embodiment of the present application is shown;

[0023] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0025] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present application indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0026] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this application specification and claims, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms. It should also be further understood that the term "and / or" used in this application specification and claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0027] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0028] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0029] Figure 1 A schematic diagram of a method for identifying a formula in an embodiment of the present application is shown below. Figure 1 The method described in this application is further explained.

[0030] like Figure 1 As shown, at step S101, hidden variables are extracted from a picture containing a formula, wherein the picture includes a mathematical formula picture handwritten by a human.

[0031] It should be understood that the mathematical formula is an expression that uses symbols and symbol combinations to describe mathematical concepts, relations, and operations. It is a form of mathematical language used to accurately express mathematical ideas and reasoning. Mathematical formulas usually include, but are not limited to, one or more elements of mathematical symbols, operators, variables, and constants. These symbols and symbol combinations are arranged and organized according to certain rules and conventions to convey accurate mathematical meanings. Further, pictures containing formulas can exist in a variety of forms, including handwriting and typeset. Handwritten formula pictures refer to mathematical formulas written or drawn by humans, which may have personalized styles and characteristics. Typeset formula pictures are mathematical formula images generated by computer software or typesetting tools, usually with standardized symbols and formats.

[0032] Since typeset formula images usually have a more standardized and clear arrangement, they are more suitable for automatic recognition and analysis by computers. However, handwritten formula images may have certain variations and be difficult to recognize due to factors such as individual writing style and handwriting clarity, and require additional processing and analysis.

[0033] Traditionally, handwritten formula recognition based on digital image processing usually involves several key steps. The first is preprocessing, including denoising, normalization, and line smoothing, with the aim of making the handwritten formula more suitable for subsequent processing. Next is feature extraction, which involves identifying key features in the handwriting, such as stroke direction, shape, and relative position. The most critical is symbol recognition and structural analysis, where the algorithm needs to recognize individual mathematical symbols and understand the relationship between them, such as fraction lines, subscripts and subscripts, and square roots.

[0034] However, image data is usually high-dimensional and contains a large amount of pixel information. By extracting the latent variables of the image through a deep learning model, high-dimensional image data can be converted into a low-dimensional representation, thereby reducing the cost of storage and computing. Capturing the key features and abstract representation of the image enables the model to better generalize when processing similar images. This helps to improve the accuracy and robustness of the model. At the same time, there may be huge changes between different images, such as lighting, angle, scale, etc. By extracting the latent variables of the image, these changes can be abstracted into more consistent feature representations, allowing the model to better distinguish different images. Furthermore, extracting the latent variables of the image can compress the original image data and reduce the storage and transmission overhead. This is of great significance for the storage, transmission and processing of images.

[0035] Therefore, in order to better handle complex image features and long-term dependencies, this application constructs a deep learning model as a feature extraction module to extract features from latent variables of the image (model architecture as shown in Figure 4 The specific process will be shown in Figure 4After obtaining the picture containing the formula, the picture is input into the feature extraction module to obtain the latent variables.

[0036] At step S102, the hidden variables are packaged to obtain an initial formula sequence, wherein the initial formula sequence includes one or more of a structural symbol, a character, and structural information.

[0037] The inventors found in experiments that in the current attention maps based on various attention methods, although the attention corresponding to a single character in the image is relatively reasonable, the attention maps of structural invisible characters in LaTeX, such as '{', '}', '_', and '^', are very scattered and distributed in areas that have nothing to do with the prediction of this character logically. One of the reasons for the error in predicting structure-related symbols in formula recognition may be that the attention model cannot effectively handle the situation of a large number of invisible characters in complex structures. In this case, an intuitive solution is to simplify the complex structure and avoid directly predicting situations such as '\frac{N^{x^y}}{\sum{3}}' by splitting the formula into step-by-step predictions. On the contrary, it can be converted into a simple unit form similar to 'A^B' and '\frac{C}{D}'. In this case, the attention model can appear in a simple distribution and is easier to learn.

[0038] The advantage of this method is that it can help the model effectively process various structural elements in the formula. By simplifying complex structures into simple units, the model's reliance on invisible characters is reduced, while improving the accuracy and stability of the model. In addition, when splitting the formula into multiple simple units, different attention weights can be set for each unit to better process the structural information in the formula. Therefore, converting complex mathematical formulas into simple units is an important solution in formula recognition. With this method, the model can better understand the structure and meaning of mathematical formulas, thereby improving the accuracy of mathematical reasoning and calculations.

[0039] It should be noted that the symbolic form of the structural symbols and characters described here is a programming language that can be processed by a computer. In one embodiment, the mathematical formula In LaTeX language, it is represented as \sqrt{(a+b)^2}+c, where the structural symbols include \sqrt{} and ()^2, and the characters include a, +, b and c.

[0040] Furthermore, the structural information is composed of the structural symbol and all the characters and structural symbols within it. In this application, the symbol representing the structural information is set as <struct>In another embodiment, the mathematical formula In LaTeX language, it is represented as \sqrt{(a+b)^2}+c, where (a+b)^2 can be structural information, and its specific symbolic form in the sequence is \sqrt{ <struct>}+c, \sqrt{(a+b)^2} can also be structural information, and its specific symbolic form in the sequence is <struct>+c.

[0041] It should be noted that due to the structural information symbol <struct>Although the structural symbols and characters in the initial formula sequence are programming languages ​​that can be processed by computers, the initial formula sequence itself does not constitute a complete mathematical expression that can be processed by computers.

[0042] Specifically, after obtaining the latent variable output by the feature extraction module, the latent variable and the start symbol are fused and input into the decoder B (e.g. Figure 4 ), to unpack and obtain the initial formula sequence (such as Figure 2 As shown in the Figure 2 The start symbol is a special symbol used to indicate the starting point of the decoder. By combining the latent variable with the feature vector of the start symbol, the encoded information can be guided to the decoder B to start the decoding process. The fusion method includes but is not limited to the following methods:

[0043] Weighted sum: Perform weighted summation on two feature vectors, that is, multiply each by a weight coefficient and add them together. Different weights can be set according to actual conditions to highlight the importance of a feature or balance the contribution of two features.

[0044] Concatenation: concatenate two feature vectors in columns or rows to form a longer feature vector. This method can retain the information of the original features and make the fused features higher in dimension.

[0045] Point product: multiply two feature vectors element by element to obtain a new feature vector. This method can capture the interaction information between two features and is suitable for emphasizing the correlation between features.

[0046] Superposition: Two feature vectors are directly superimposed together to form a larger feature matrix. This method is usually used to introduce more detailed information or context information into the feature representation.

[0047] Neural network fusion: Using the neural network structure, two feature vectors are input into a fusion network, and training is performed to learn how to best combine the two features. This method can flexibly handle various types of feature fusion problems and find the best fusion method in a data-driven situation.

[0048] After obtaining the fusion vector, decoder B is responsible for gradually generating the formula sequence based on the input hidden variables and start symbols. Decoder B uses a series of algorithms and models to gradually disassemble the structure of the formula and generate each character or symbol through continuous iteration and learning. In this process, the decoder will dynamically adjust and update based on the generated partial formula and input information to generate a more accurate and complete formula sequence.

[0049] According to the previous examples, we can know that in the same formula, the setting of structural information may be in many different ways. Therefore, in an implementation scenario, the following steps can be taken to deal with this problem. First, the two feature vectors of latent variables and start symbols are fused, and then passed as input to decoder B. Next, decoder B processes these inputs and outputs a vector. In order to obtain the initial formula sequence, it is necessary to perform a softmax operation on this output vector. An initial formula sequence that does not contain structural symbols is obtained. This sequence carries the structural information of the entire formula, which contains all the structural symbols in the formula. In other words, this sequence does not display structural elements such as brackets, fraction lines, etc. in the formula. The mathematical formula described in the previous article For example, through the first generation of decoder B, the initial formula sequence of the formula is <struct>+c, instead of \sqrt{ <struct>}+c.

[0050] At step S103, the initial formula sequence is unpacked to obtain a complete formula sequence, wherein the complete formula sequence includes a mathematical expression that can be processed by a computer, and the format of the mathematical expression includes LaTeX or MathML.

[0051] It should be noted that when introducing the initial formula sequence in the previous article, the form of the structure symbols and characters was explained, that is, the mathematical programming language that can be understood by computers. <struct>The existence of the initial formula sequence itself is not a mathematical expression that can be processed by a computer. In one embodiment, the mathematical formula In LaTeX language, it is expressed as \sqrt{(a+b)^2}+c, where \sqrt{(a+b)^2}+c is the complete formula sequence.

[0052] Based on what has been mentioned above, in order to simplify complex structures and turn formulas into step-by-step predictions by splitting them, this application has designed a data structure and training method based on these invisible structures to package and identify the contents of brackets and gradually unpack them. The basic logic of the method is to package the substructures 'S^{X}', 'S_{X}', 'S_{X}^{Y}', '\frac{X}{Y}', '\sqrt{X}', '\sqrt[X]{Y}' (where S is a single character, and X, Y are any non-empty content) in a formula from the outside to the inside into specific structural symbols. <struct>, and gradually unpack <struct>The process of content (S, X, Y).

[0053] It should be understood that in the depacketization of decoder B, there is a cyclic process (such as Figure 3 As shown in the figure, the specific cycle process will be Figure 3 , which is explained in detail here, is used to predict the next sequence. Specifically, the process is based on two inputs: latent variables and the sequence of the previous step (after embedding). The latent variable is the representation data obtained by compression and encoding of the encoder module, which contains important features and information of the input data. The sequence of the previous step is the sequence generated by decoder B in the previous iteration, which is converted into a vector form through the embedding operation. When predicting the next sequence, decoder B will send the latent variable and the sequence vector of the previous step into a recurrent neural network to obtain the output of the current time step. The neural network can adopt structures such as LSTM, GRU or Attention, which can effectively process sequence information and long-term dependencies. Through continuous iteration and updating, decoder B can gradually generate a complete formula sequence.

[0054] It is important to note that in each iteration, decoder B is dynamically adjusted and updated based on the generated partial sequence and input information. This means that with each prediction step, decoder B is able to gradually disassemble the structure of the formula and generate a more accurate and complete sequence. At the same time, due to the existence of a cyclic process, decoder B is also able to consider the output results of the previous step to ensure that the generated sequence is coherent and reasonable.

[0055] Figure 2 The following is a flow chart of a method for packaging a formula sequence according to an embodiment of the present application. Specifically, Figure 2 The process of packing the latent variables to obtain an initial formula sequence is further illustrated.

[0056] It should be understood that, in this embodiment, steps S201 to S203 constitute a process of generating an initial formula sequence. This process is based on the input hidden variables and start symbols, and combined with certain algorithms and models to gradually disassemble and reorganize the initial formula sequence to obtain a complete formula sequence.

[0057] like Figure 2 As shown, at step S201, the latent variable and the start symbol are decoded to obtain the first layer of structural symbols and characters other than the structural symbols of the initial formula sequence. Here, the latent variable is the representation data compressed and encoded by the encoder module, which contains important features and information of the input data. Relatively speaking, the start symbol refers to a special symbol used as the starting point in the sequence generation task, which can generally be composed of predefined marks or blank characters. By sending the latent variable and the start symbol into the decoder module for decoding, the first layer of structural symbols and characters other than the structural symbols of the initial formula sequence can be obtained.

[0058] In step S202, the first layer structure symbol and the information inside it are packaged into structure information. Decode, and when \sqrt{}+c is generated, stop generating the \sqrt{} internal sequence, and pack \sqrt{} as a structural information symbol <struct>.

[0059] At step S203, the characters and the structural information are combined to obtain an initial formula sequence. <struct>After that, combine it with the character +c to get the initial formula sequence <struct>+c. It should be noted that in this process, the role of structural information is very important. It can help the generator module better understand the structure and meaning of the formula, thereby generating a more accurate and complete formula sequence.

[0060] In summary, steps S201 to S203 constitute a process of generating a preliminary formula sequence, in which elements such as hidden variables, start symbols, structure symbols and structure information are fully considered and utilized to improve the quality and efficiency of formula generation.

[0061] Figure 3 A schematic diagram of a method for unpacking a formula sequence according to an embodiment of the present application is shown below. Figure 3 The process of unpacking the initial formula sequence to obtain the complete formula sequence is further described. It should be understood that Figure 3 The method shown is mainly used to extract the internal structure information of the formula sequence, and update and combine it with the character information to obtain the complete formula sequence.

[0062] like Figure 3 As shown, at step S301, the initial formula sequence and the latent variables are decoded to extract the first-level structural symbols, the second-level structural symbols, and the characters between the two-level structural symbols of the structural information in the initial formula sequence. In one embodiment, the mathematical operation priority of the second-level structural symbols is higher than the mathematical operation priority of the first-level structural symbols. For example, in the mathematical formula In the equation, \sqrt{} is the first-level structure symbol, and ()^2 is the second-level structure symbol.

[0063] In step S302, the second-level structure symbol and the information inside it are packaged into structure information. The packaging method can be referred to in Figure 2 The description at step S202 is shown.

[0064] At step S303, the formula sequence is updated using the first-layer structure symbol, the characters between the two-layer structure symbols, and the structure information. In one embodiment, this step may include combining and arranging the previously extracted structure information and character information with the first-layer structure symbol to update the formula sequence and obtain a new formula sequence. Wherein, the updating of the formula sequence may include: retaining the characters and structure symbols passed into the formula sequence before decoding; replacing the structure information with the first-layer structure symbol, the characters between the two-layer structure symbols, and the structure information extracted after decoding; and when the second-layer structure symbol after decoding is empty, using the first-layer structure symbol and characters to replace the structure information before decoding.

[0065] At step S304, the updated formula sequence replaces the initial formula sequence to be used for iterative unpacking with the hidden variables in the next cycle. In this step, the updated formula sequence is sent as input to the decoder module for the next round of decoding operation.

[0066] At step S305, when the second-level structure symbol is empty, the first-level structure symbol and characters are used to update the formula sequence to obtain a complete formula sequence. In this step, if the second-level structure symbol does not exist, it means that the formula sequence has been completely unpacked. At this time, the first-level structure symbol and character information are combined and arranged to obtain the final complete formula sequence.

[0067] Next, the unpacking process will be described in combination with the steps and the formula \int_{1}^{\infty}\frac{\frac{3x+2}{2}}{x^2+1}dx in an embodiment.

[0068] First, the outermost \int_{1}^{\infty}, \frac{\frac{3x+2}{2}}{x^2+1} are packaged into <struct>, get the initial formula sequence <struct> <struct>dx;

[0069] The first step is to convert each structural symbol in the previous step into <struct>Unpack and pack the characters corresponding to the structure symbol. The first <struct>Unpacking gives \int_{1}^{\infty}, then looking for structure, but no further structure can be packed, giving \int_{1}^{\infty}, the second <struct>Unpack it to get \frac{\frac{3x+2}{2}}{x^2+1}, and then pack the fraction and superscript inside it into \frac{ <struct>}{ <struct>+1}, and finally merge to get \int_{1}^{\infty}\frac{ <struct>}{ <struct>+1}dx;

[0070] The second step is to repeat for each structural symbol <struct>Unpack and pack the inside to get \int_{1}^{\infty}\frac{\frac{3x+2}{2}}{x^2+1}dx;

[0071] In the third step, it is found that the output of the previous step no longer contains ' <struct>', cannot unpack, abort the process, and output the complete formula sequence.

[0072] In summary, the above is Figure 3 The process can effectively obtain the complete formula sequence by extracting and updating the internal structure information of the formula sequence, thereby realizing efficient and accurate processing and analysis of mathematical formulas.

[0073] Figure 4 The overall structure diagram of the model of a recognition formula in the embodiment of the present application is shown below. Figure 4 The structure of the model and the process of generating formula sequences are further explained.

[0074] In one embodiment, the feature extraction module includes a Transformer model composed of an encoder and a decoder A. The model is a neural network architecture based on a self-attention mechanism. Through the self-attention mechanism, the model can automatically learn the relative relationship and importance between different parts of the picture. This means that the model can understand the contextual information between each pixel or region and other pixels or regions. At the same time, the feedforward neural network performs nonlinear transformation and modeling on the features at each position. This helps to better capture features such as shape, structure and texture in mathematical formula pictures. By stacking multiple layers of encoders, the model can gradually enrich and extract abstract representations of mathematical formula pictures. Each encoder layer will further deepen the understanding and expression of features. In the end, the model will output a feature vector or feature matrix with rich semantic information, which contains important features of mathematical formula pictures. These features can be further used for different tasks, such as classification, generation or reasoning, providing a stronger foundation for subsequent analysis and application.

[0075] Specifically, first, the image to be processed is input into the Transformer model. This can be an image containing mathematical formulas, such as a handwritten mathematical formula image or a typeset mathematical formula image. The input image is then preprocessed to ensure that it fits the input format of the Transformer model. This may include resizing the image, cropping or padding the image, and performing necessary normalization operations. Using convolutional neural networks (CNNs) or other image processing techniques, features are extracted from the image. These features can capture various shapes, textures, and structural information in the image. The extracted features are serialized into a one-dimensional vector or matrix representation for easy input into the Transformer model. This can be achieved using the Flatten operation or other serialization methods. The serialized features are input into the Transformer encoder. The Transformer encoder is a neural network structure consisting of multiple encoder layers for encoding and modeling the input. In the Transformer encoder, a self-attention mechanism is used to capture the relative relationship and importance between features. This helps the model understand the context and dependencies between different features in the image. The representation ability of the model can be deepened by stacking the Transformer encoder layers multiple times. Each encoder layer further models and extracts features, gradually enriching the information.

[0076] The decoder is mainly used to convert the latent variables output by the encoder into readable output results. In the Transformer model, the decoder is a neural network structure composed of multiple decoder layers. Their input is the matrix vector output by the encoder and the previous prediction results, and a new prediction result is generated at each time step. The decoder layer can use self-attention mechanism and multi-head attention mechanism to utilize the encoder output and the current prediction results to better understand the context and dependencies. In the decoder, each decoder layer includes a self-attention sublayer, an encoder-decoder attention sublayer, and a fully connected feedforward network sublayer. The self-attention sublayer is mainly used to capture the context information in the decoder input sequence; the encoder-decoder attention sublayer aligns the encoder output with the decoder input to better understand the global context information; the fully connected feedforward network sublayer further extracts features and produces the final output. By stacking the decoder layers multiple times, long sequences can be gradually generated and the latent variables can be converted into readable output results. These output results can be text, images, or other forms of data, depending on the application scenario.

[0077] Finally, the latent variables are obtained at the output end of the Transformer model decoder A. The latent variables include but are not limited to one-dimensional, two-dimensional and three-dimensional matrix vectors.

[0078] In another embodiment, after the latent variables are obtained through the feature extraction module, it is necessary to obtain the initial formula sequence and the complete formula sequence through a decoder B in a loop iteration manner.

[0079] It should be understood that decoder B includes but is not limited to a model built based on the Attention algorithm. In one embodiment, decoder B converts the fused start symbol and hidden variable into a target sequence. In a sequence-to-sequence model based on the attention algorithm, the decoder is usually composed of multiple decoder layers, and the first input of the decoder is the target sequence element generated previously. When decoding for the first time, a special start symbol is usually used as the first input. The understanding of the context is maintained by storing the state of the previous time step. Based on the current input and the state of the previous time step, the decoder updates its own state. The state includes a hidden state and a cell state, which are designed to capture different sequence features. Using its current state and context vector, the output element of the current time step is generated. Since the decoder usually needs to generate a longer sequence, the decoder based on the attention mechanism is usually composed of multiple decoder layers, each of which can process the input of one time step. These decoder layers usually share the same parameters so that the model can be learned and updated efficiently during training.

[0080] Accordingly, Figure 4 The sequence prediction process based on the Encoder-Decoder architecture model is demonstrated. First, the image is sent to the encoder to encode the image features, and then it is passed through the decoder to obtain the latent variables, and the latent variables are then obtained through processes such as softmax.

[0081] There is a loop process after the hidden variable. Given the hidden variable and the sequence of the previous step (after embedding), the next sequence is predicted. Decoder B is a decoder with the same architecture as Decoder A and is reused in the loop. This process is used in both training and prediction. The difference is that the training loss is the sum of the cross entropy losses of each intermediate step sequence, while the prediction only needs to use the output of the last sequence as the result.

[0082] In the loop unpacking process, decoder A is used to deeply process the image data, while decoder B takes on part of the multimodal function, using the sequence data and the processed image data (hidden variables) to obtain the next sequence.

[0083] Table 1 Performance of various target recognition algorithms on the mathematical alteration removal dataset

[0084] Handwritten formula recognition algorithm ExpRate Error-1 Error-2 DWAP 51.72 69.47 77.99 DWAP-TD 49.10 64.20 67.80 BTTR 53.96 66.02 70.28 ABM 56.85 73.73 81.24 SAN 56.20 72.60 79.20 CoMER 59.33 71.70 75.66 DWAP+loop depacketization 52.95 70.13 80.01 BTTR+ loop unpacking 54.73 67.92 73.07 CoMER+ loop unpacking 57.31 69.12 78.21

[0085] It should be understood that the method of the present application was trained and tested on the CROHME 2014 dataset, and the recognition effect was improved by introducing the loop unpacking mechanism. Table 1 shows the recognition effect of various existing algorithms on the dataset, as well as the effect after applying the loop unpacking mechanism. Among them, ExpRate represents the percentage of the entire formula that is correctly predicted, and Error-1 and Error-2 represent the percentage of correct predictions that allow 1 and 2 character errors, respectively.

[0086] By observing the table, we can find that the effect of loop unpacking fluctuates on different models. However, the overall trend shows that loop unpacking can bring a certain degree of improvement in recognition effect. Especially when 2 character errors are allowed, the loop unpacking mechanism shows a relatively reliable improvement effect. This means that loop unpacking can effectively improve the recognition accuracy of mathematical formulas and improve the robustness of the algorithm in complex situations.

[0087] In summary, the experiment proves that applying the loop unpacking mechanism to the existing algorithm can significantly improve the recognition effect of mathematical formulas. The experimental results of this method on the CROHME 2014 dataset show the potential and effectiveness of loop unpacking, and provide valuable reference for the research and application in the field of mathematical formula recognition.

[0088] Figure 5 FIG. 1 is a schematic block diagram of an electronic device according to an embodiment of the present application. Figure 5 As shown, the electronic device 500 may include a processor 501 and a memory 502. The memory 502 stores computer instructions for identifying a formula. When the computer instructions are executed by the processor 501, the electronic device 500 performs the above-mentioned steps in combination with the above-mentioned steps. Figure 1 to Figure 4 The method described.

[0089] In some embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program code is executed by a processor, the program code is used to cause the processor to perform the method described above. The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0090] Although multiple embodiments of the present application have been shown and described herein, it is obvious to those skilled in the art that such embodiments are provided only by way of example. Those skilled in the art can think of many changes, modifications and alternatives without departing from the thought and spirit of the present application. It should be understood that in the process of practicing the present application, various alternatives to the embodiments of the present application described herein can be adopted. The attached claims are intended to limit the scope of protection of the present application, and therefore cover equivalents or alternatives within the scope of these claims.< / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct> < / struct>

Claims

1. A method for identifying a formula, characterized in that: include: Extract latent variables from images containing formulas; Packing the hidden variables to obtain an initial formula sequence; as well as The initial formula sequence is unpacked to obtain a complete formula sequence.

2. The method according to claim 1, characterized in that: The images include images of mathematical formulas written by humans.

3. The method according to claim 1, characterized in that The initial formula sequence includes one or more of structural symbols, characters and structural information.

4. The method according to claim 1, characterized in that: The complete formula sequence includes mathematical expressions that can be processed by a computer.

5. The method according to claim 4, characterized in that The mathematical expression may be in a format of LaTeX or MathML.

6. The method according to claim 1, characterized in that The hidden variables are packaged to obtain an initial formula sequence, which includes: Decoding the hidden variable and the start symbol to obtain the first-layer structural symbol and characters other than the structural symbol of the initial formula sequence; Packing the first-layer structure symbol and the information therein into structure information; and The character and structure information are combined to obtain an initial formula sequence.

7. The method according to claim 6, characterized in that The initial formula sequence is unpacked to obtain a complete formula sequence including: Decoding the initial formula sequence and latent variables to extract the first-level structural symbols, the second-level structural symbols and the characters between the two-level structural symbols of the structural information in the initial formula sequence; Pack the second-level structure symbol and the information inside it into structure information; Using the first-layer structure symbol, characters between the two-layer structure symbols and structure information to update the formula sequence; Replacing the initial formula sequence with the updated formula sequence for iterative unpacking with the hidden variables in the next cycle; and When the second-level structure symbol is empty, the first-level structure symbol and characters are used to update the formula sequence to obtain a complete formula sequence.

8. The method according to claim 7, characterized in that The mathematical operation priority of the second-level structure symbol is higher than the mathematical operation priority of the first-level structure symbol.

9. The method according to claim 7, characterized in that: The update formula sequence includes: Keep the characters and structure symbols in the formula sequence before decoding; replacing the structural information with the first-layer structural symbol, the characters between the two-layer structural symbols, and the structural information extracted after decoding; and When the second-layer structure symbol is empty after decoding, the structure information before decoding is replaced by the first-layer structure symbol and characters.

10. A device for identifying a formula, characterized in that include: processor; A memory having computer instructions for identifying a formula stored thereon, which, when executed by a processor, implement the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: Computer program instructions for identifying a formula are stored thereon, and when the computer program instructions are executed by one or more processors, the method according to any one of claims 1-9 is implemented.

Images