An online handwritten mathematical formula generation method based on a structure-aware latent space diffusion model

Abstract

This invention discloses an online handwritten mathematical formula generation method based on a structure-aware latent space diffusion model. First, the input LaTeX expression is parsed to obtain symbol content, spatial topological relationships between symbols, and hierarchical depth information of the symbols, and a structured representation (RelAST) is constructed. Second, a generation framework based on a latent space conditional diffusion model is constructed, and a structure-aware sequence variational autoencoder is designed. By introducing symbol recognition constraints and structural relationship constraints, the latent representation simultaneously represents symbol semantics and spatial topological structure information. In the generation stage, using the structured representation (RelAST) as conditional input and combining prior information on the number of symbols, the target latent variable representation is generated in the latent space through the diffusion model. Finally, the decoder recovers the complete online handwritten trajectory sequence. This invention can effectively model the two-dimensional structural relationships of mathematical formulas, achieving unified generation of symbol content and spatial layout.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, pattern recognition, handwritten content generation and sequence modeling technology, and in particular relates to an online handwritten mathematical formula generation method based on a structure-aware latent space diffusion model. Background Technology

[0002] Online handwritten mathematical formula generation technology aims to automatically generate corresponding online handwritten trajectory sequences based on input LaTeX expressions of mathematical formulas. Compared to ordinary text generation tasks, mathematical formulas not only contain symbolic content but also complex two-dimensional layout relationships such as fractions, subscripts, superscripts, matrices, and nested structures. Therefore, it is necessary to simultaneously model symbolic semantics and spatial topology to generate handwritten mathematical formulas with accurate content and structure. Most existing handwritten mathematical formula generation methods adopt a two-stage generation paradigm: first, generating the symbolic content involved in the expression; second, predicting the spatial layout of each symbol based on the syntactic structure of the LaTeX expression. This type of method usually relies on fine-grained spatial annotation information such as symbol-level bounding boxes, resulting in high annotation costs. Under this constraint, the model often has to rely on data-driven learning. However, the modeling assumption of decoupling symbolic content from spatial layout in this type of method may not hold true, especially in complex formula scenarios where the two are often strongly coupled. Therefore, this separate modeling can easily disrupt structural consistency, making it difficult for the generated results to present a natural and coherent overall structure.

[0003] Furthermore, although the original LaTeX expressions implicitly contain the spatial topological relationships between symbols at the syntactic level, their linear sequence organization and reliance on deep nesting structures to express two-dimensional layouts make it difficult for generative models to directly and efficiently model these relationships. Simultaneously, the highly flexible nature of LaTeX syntax means that the same mathematical structure often corresponds to multiple equivalent notations, further increasing the complexity and uncertainty of model learning. To alleviate these problems, LaTeX can often be parsed into a syntax tree structure to explicitly characterize its hierarchical relationships. However, traditional syntax trees contain a large number of non-terminal container nodes, resulting in low information density and a lack of direct modeling of the specific spatial topological relationships between symbols, making them unsuitable as conditional representations for generative models. Furthermore, while the standard MathML abstract syntax tree can fully describe the structural information of mathematical expressions, it still does not explicitly encode the two-dimensional spatial topological relationships between symbols. The model needs to indirectly infer the layout from the nested structure, increasing the modeling difficulty and reducing generation efficiency.

[0004] Therefore, existing technologies urgently need a mathematical formula generation method that does not rely on fine position annotation, can uniformly characterize symbol content, spatial topological relationships and hierarchical structure, and is applicable to the generation of online handwritten trajectory sequences, so as to effectively improve the structural consistency and content accuracy in the process of generating complex formulas. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes an online handwritten mathematical formula generation method based on a structure-aware latent space diffusion model, thereby resolving the issues present in the existing technologies.

[0006] In a first aspect, to achieve the above objectives, the present invention provides an online handwritten mathematical formula generation method based on a structure-aware latent space diffusion model, comprising the following steps: Obtain the LaTeX expression of a mathematical formula and its corresponding online handwritten trajectory sequence; The LaTeX expression of the mathematical formula is parsed to construct a structured representation RelAST. The structured representation RelAST includes symbol content, spatial topological relationships between symbols, and symbol hierarchical depth information, which is used to uniformly model and represent symbol content and spatial topological relationships. A structure-aware sequence variational autoencoder is constructed to encode the online handwritten trajectory sequence into a latent variable representation. The symbol recognition module and the structural relationship recognition module in the structure-aware sequence variational autoencoder are used to apply structure-aware constraints to the latent variable representation so that the latent variable representation can simultaneously represent symbolic semantic information and spatial topological structure information. A conditional diffusion generation model is constructed, using the structured representation RelAST and the prior number of symbols as conditional inputs. A diffusion and denoising process is performed in the latent space where the latent variable representation is located to generate the target latent variable representation. The target latent variable representation is input into the decoder of the structure-aware sequence variational autoencoder to generate an online handwritten trajectory sequence.

[0007] Optionally, the structured representation RelAST construction process includes: The mathematical formula expression is parsed into a standardized MathML representation; Remove non-terminal container nodes from the MathML representation; The MathML representation is transformed into a structured representation RelAST by depth-first traversal. Each element in the RelAST consists of the symbol content, the spatial topological relationship between the symbols, and the hierarchical depth of the symbols.

[0008] Optionally, the structure-aware sequence variational autoencoder includes: A one-dimensional convolutional encoder is used to compress the online handwritten trajectory sequence into the latent variable representation; A one-dimensional transposed convolutional decoder is used to reconstruct the latent variable representation into the original handwritten trajectory sequence; The symbol recognition module is used to perform character recognition on the latent variable representation and output the symbol recognition loss; The structural relationship identification module is used to predict the spatial topological relationships between symbols and output the structural relationship loss.

[0009] Optionally, the application process of the structure-aware constraint includes: During the training process of the structure-aware sequence variational autoencoder, the symbol recognition loss and the structural relation loss are jointly optimized with the reconstruction loss and the KL divergence loss; The symbol recognition loss constrains the symbolic semantic expression represented by the latent variables; The spatial topological relation expression represented by the latent variables is constrained by the structural relation loss.

[0010] Optionally, the conditional diffusion generation model includes: The content encoder is used to embed and encode the symbolic content, spatial topological relationships, and hierarchical depth information in the structured representation RelAST to obtain a structured conditional sequence. A conditional denoising network, consisting of multiple Transformer modules, is used to perform stepwise denoising on the input noise latent variable representation based on the diffusion time step and the structured conditional sequence.

[0011] Optionally, the diffusion and denoising process includes: The number of symbols is introduced as a priori for the global structure. The temporal embedding is subjected to complexity-sensitive modulation based on the diffusion time step, and the modulated temporal embedding and the symbol number prior are jointly injected into the conditional denoising network. A deterministic diffusion sampling strategy is adopted in the inference phase to generate the target latent variable representation within a finite number of steps.

[0012] Optionally, the process of generating the target latent variable representation includes: The LaTeX expression of the target mathematical formula is parsed to obtain the structured representation RelAST and the prior number of symbols; Starting from random Gaussian noise, and using the structured representation RelAST and the prior number of symbols as conditional inputs, the target latent variable representation is generated by performing stepwise denoising through the conditional diffusion generation model.

[0013] Optionally, the process of generating the online handwritten trajectory sequence includes: The target latent variable representation is input into the decoder of the structure-aware sequence variational autoencoder; The decoder recovers the trajectory length through upsampling and outputs the trajectory point coordinate distribution parameters and pen state prediction results; Based on the trajectory point coordinate distribution parameters and pen state prediction results, a complete online handwriting trajectory sequence is generated.

[0014] In a second aspect, the present invention also provides a computer terminal device, comprising: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the online handwritten mathematical formula generation method based on the latent space diffusion model in the first aspect above.

[0015] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the online handwritten mathematical formula generation method based on the latent space diffusion model in the first aspect described above.

[0016] Compared with the prior art, the present invention has the following advantages and technical effects: This invention provides an online handwritten mathematical formula generation method based on a structure-aware latent space diffusion model. It eliminates the need for symbol-level positional annotation or bounding box annotation, obtaining a structured representation (RelAST) suitable for the generation task simply by parsing LaTeX expressions, thus reducing the cost of training data construction. By constructing the structured representation RelAST, the symbol content, spatial topological relationships between symbols, and the hierarchical depth of symbols in mathematical formulas can be explicitly modeled, thereby improving the generation accuracy of complex two-dimensional structural formulas. By introducing joint constraints of symbol recognition and relation recognition into the latent space of the variational autoencoder, the latent space simultaneously retains geometric handwriting features, symbol semantic information, and structural topological information, providing a more suitable structured representation foundation for subsequent diffusion generation. By introducing prior symbol quantity and jointly modulating it with the diffusion time step, the model's ability to model the overall complexity of formulas is enhanced, reducing omissions, misspellings, and structural mismatches in complex formulas. This invention not only generates high-quality online handwritten mathematical formulas but can also be used as a data augmentation technique to improve the recognition accuracy of downstream mathematical formula recognition models. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the construction process of the structured representation RelAST according to an embodiment of the present invention; Figure 3 The following is a schematic diagram of the framework structure of an embodiment of the present invention, wherein (a) shows the process of converting the LaTeX expression of a mathematical formula into the structured representation RelAST; (b) is a schematic diagram of the training of a structure-aware sequence variational autoencoder; and (c) is a schematic diagram of the training of a conditional diffusion generative model. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0020] This embodiment provides an online handwritten mathematical formula generation method based on a structure-aware latent space diffusion model, including the following steps: like Figure 1 As shown, the LaTeX expression of the mathematical formula and its corresponding online handwritten trajectory sequence are obtained; like Figure 2 As shown, the LaTeX expression is parsed, converted into MathML representation, and a structured representation RelAST is constructed. The RelAST includes symbol content, spatial topological relationships between symbols, and hierarchical depth information. like Figure 3 As shown in (a), the process of converting a LaTeX expression of a mathematical formula into a structured representation RelAST is demonstrated. like Figure 3 As shown in (b), a structure-aware sequence variational autoencoder is constructed to encode the online handwritten trajectory sequence into a latent variable representation. Constraints are applied to the latent variable representation through a symbol recognition module and a structural relationship recognition module so that it retains both symbolic semantic information and spatial topological relationship. like Figure 3 As shown in (c), a conditional diffusion generation model is constructed, using the structured representation RelAST and the prior number of symbols as conditional inputs, and performing diffusion and denoising processes in the latent space to generate the target latent variable representation; The target latent variable representation is input into the decoder of the structure-aware sequence variational autoencoder to generate an online handwritten trajectory sequence.

[0021] Furthermore, the construction process of the structured representation RelAST includes: Parse LaTeX expressions into standard MathML representations; Remove the non-terminal container node from the MathML; The structure is serialized by depth-first traversal, resulting in a structured representation RelAST containing symbolic content, spatial topological relationships, and hierarchical depth information.

[0022] The symbol content represents the character or operator to be generated, the spatial topology represents the two-dimensional layout relationship between symbols, and the hierarchy depth represents the nesting level of symbols in the expression, thus explicitly modeling the implicit two-dimensional relationship in the original tree structure.

[0023] Furthermore, the structure-aware sequence variational autoencoder includes: A one-dimensional convolutional encoder is used to compress online handwritten trajectory sequences into a latent variable representation; A one-dimensional transposed convolutional decoder is used to recover trajectory sequences from latent variable representations; The symbol recognition module is used to predict symbol categories and output symbol recognition loss; The structural relationship identification module is used to predict the spatial topological relationships between symbols and output the structural relationship loss.

[0024] The online handwritten trajectory sequence is represented as a sequence of data containing coordinate information and pen state information. The encoder compresses the input trajectory into a latent variable representation, and the decoder recovers the trajectory length through upsampling, and outputs the trajectory point coordinate distribution parameters and pen state prediction results.

[0025] Furthermore, the structure-aware constraints are applied in the following manner: The symbol recognition loss and structural relationship loss are jointly optimized with the reconstruction loss and KL divergence loss; Symbolic semantic representation of latent variables in loss constraints is identified through symbol recognition; Spatial topological relationships are expressed through latent variables constrained by structural relation loss.

[0026] Among them, reconstruction loss is used to ensure the consistency between the generated trajectory and the true trajectory, and KL divergence loss is used to constrain the continuity and stability of the latent variable representation distribution.

[0027] Furthermore, the conditional diffusion generation model includes: The system consists of a content encoder and a conditional denoising network. The content encoder is used to serialize the structured representation RelAST to obtain a structured conditional sequence. The conditional denoising network is composed of multiple Transformers and is used to perform progressive denoising on the input latent variable representation by combining the diffusion time step, the structured conditional sequence and the prior number of symbols during the diffusion process.

[0028] Specifically, the latent space diffusion generation model is based on the Transformer architecture and performs conditional generation modeling in the latent space constructed by the structure-aware variational autoencoder. The conditional generation diffusion model receives noisy latent variable representations, structured conditional sequences, and prior information on the number of symbols, and generates corresponding target latent variable representations at each diffusion time step.

[0029] Furthermore, the construction process of the structured condition sequence includes: The elements in the structured representation RelAST are embedded to obtain a high-dimensional vector representation. The symbol embedding and deep embedding are element-wise summed, while the relation embedding remains independent. These are then arranged in the order of the occurrence of symbols and relations and input into the content encoder to obtain a structured conditional sequence.

[0030] Furthermore, a priori information on the number of symbols is introduced to modulate the embedding of the diffusion time step, enabling the model to perceive the overall complexity of the formula during the generation process, thereby enhancing the constraints on the number of symbols and structural consistency in the later stages of diffusion.

[0031] Furthermore, the process of generating the target latent variable representation includes: The LaTeX expression of the target mathematical formula is parsed to obtain the structured representation RelAST and the prior number of symbols; based on the conditional input, stepwise denoising is performed through a conditional diffusion model to generate the target latent variable representation.

[0032] Furthermore, the process of generating the online handwritten trajectory sequence includes: The target latent variable representation is input into the decoder of the structure-aware sequence variational autoencoder. The sequence length is recovered by upsampling, and the trajectory point coordinate distribution parameters and pen state prediction results are output to generate a complete online handwritten trajectory sequence.

[0033] When used in downstream handwritten mathematical formula recognition tasks, the generated samples can serve as data augmentation data, effectively improving the recognition accuracy of the recognition model. On commonly used benchmark datasets such as MathWriting and CROHME 2014, 2016, and 2019, varying degrees of improvement in recognition accuracy have been achieved. Specifically, recognition models such as Qwen3-VL 8B-Instruct and UniMumer have shown varying degrees of improvement in recognition accuracy.

[0034] As shown in Table 1, the handwritten mathematical formulas generated by this invention can be used to expand the training dataset for the handwritten mathematical formula recognition task, thereby effectively improving the recognition accuracy of the Qwen3-VL 8B-Instruct and UniMumer models on the MathWriting evaluation set.

[0035] Table 1 As shown in Table 2, using the handwritten mathematical formulas generated by this invention to expand the training data for the handwritten mathematical formula recognition task can effectively improve the recognition accuracy of the Qwen3-VL 8B-Instruct and UniMumer models on the CROHME 2014, CROHME2016 and CROHME 2019 evaluation sets.

[0036] Table 2 Example 2 In this embodiment, a computer terminal device is provided, including: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the online handwritten mathematical formula generation method based on the latent space diffusion model described above.

[0037] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the above-described online handwritten mathematical formula generation method based on the latent space diffusion model.

[0038] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating online handwritten mathematical formulas based on a structure-aware latent space diffusion model, characterized in that, include: Obtain the LaTeX expression of a mathematical formula and its corresponding online handwritten trajectory sequence; The LaTeX expression of the mathematical formula is parsed to construct a structured representation RelAST. The structured representation RelAST includes symbol content, spatial topological relationships between symbols, and symbol hierarchical depth information, which is used to uniformly model and represent symbol content and spatial topological relationships. A structure-aware sequence variational autoencoder is constructed to encode the online handwritten trajectory sequence into a latent variable representation. The symbol recognition module and the structural relationship recognition module in the structure-aware sequence variational autoencoder are used to apply structure-aware constraints to the latent variable representation so that the latent variable representation can simultaneously represent symbolic semantic information and spatial topological structure information. A conditional diffusion generation model is constructed, using the structured representation RelAST and the prior number of symbols as conditional inputs. A diffusion and denoising process is performed in the latent space where the latent variable representation is located to generate the target latent variable representation. The target latent variable representation is input into the decoder of the structure-aware sequence variational autoencoder to generate an online handwritten trajectory sequence.

2. The method according to claim 1, characterized in that, The RelAST structured representation construction process includes: The mathematical formula expression is parsed into a standardized MathML representation; Remove non-terminal container nodes from the MathML representation; The MathML representation is transformed into a structured representation RelAST by depth-first traversal. Each element in the RelAST consists of the symbol content, the spatial topological relationship between the symbols, and the hierarchical depth of the symbols.

3. The method according to claim 1, characterized in that, The structure-aware sequence variational autoencoder includes: A one-dimensional convolutional encoder is used to compress the online handwritten trajectory sequence into the latent variable representation; A one-dimensional transposed convolutional decoder is used to reconstruct the latent variable representation into the original handwritten trajectory sequence; The symbol recognition module is used to perform character recognition on the latent variable representation and output the symbol recognition loss; The structural relationship identification module is used to predict the spatial topological relationships between symbols and output the structural relationship loss.

4. The method according to claim 3, characterized in that, The process of applying the structure-aware constraints includes: During the training process of the structure-aware sequence variational autoencoder, the symbol recognition loss and the structural relation loss are jointly optimized with the reconstruction loss and the KL divergence loss; The symbol recognition loss constrains the symbolic semantic expression represented by the latent variables; The spatial topological relation expression represented by the latent variables is constrained by the structural relation loss.

5. The method according to claim 1, characterized in that, The conditional diffusion generation model includes: The content encoder is used to embed and encode the symbolic content, spatial topological relationships, and hierarchical depth information in the structured representation RelAST to obtain a structured conditional sequence. A conditional denoising network, consisting of multiple Transformer modules, is used to perform stepwise denoising on the latent variable representation of the input noise based on the diffusion time step and the structured conditional sequence.

6. The method according to claim 5, characterized in that, The diffusion and denoising process includes: The number of symbols is introduced as a priori for the global structure. The temporal embedding is subjected to complexity-sensitive modulation based on the diffusion time step, and the modulated temporal embedding and the symbol number prior are jointly injected into the conditional denoising network. A deterministic diffusion sampling strategy is adopted in the inference phase to generate the target latent variable representation within a finite number of steps.

7. The method according to claim 1, characterized in that, The generation process of the target latent variable representation includes: The LaTeX expression of the target mathematical formula is parsed to obtain the structured representation RelAST and the prior number of symbols; Starting from random Gaussian noise, and using the structured representation RelAST and the prior number of symbols as conditional inputs, the target latent variable representation is generated by performing stepwise denoising through the conditional diffusion generation model.

8. The method according to claim 1, characterized in that, The process of generating the online handwritten trajectory sequence includes: The target latent variable representation is input into the decoder of the structure-aware sequence variational autoencoder; The decoder recovers the trajectory length through upsampling and outputs the trajectory point coordinate distribution parameters and pen state prediction results; Based on the trajectory point coordinate distribution parameters and pen state prediction results, a complete online handwriting trajectory sequence is generated.

9. A computer terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-8.

Images