Data processing method, device, equipment and storage medium
By converting mathematical formulas into tree structure data and using tree nodes and tree edge labels to train task models, the problem of difficult semantic understanding of mathematical formulas is solved, and the fine-grained understanding of mathematical formulas and more accurate task processing is achieved.
Patent Information
- Application Number
- CN202210118419.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-08
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-02-08
AI Technical Summary
The prior art is difficult to accurately understand mathematical formulas semantics, resulting in the inability to correctly perform task processing.
By converting mathematical formulas into tree structure data, using tree nodes and tree edge labels to train downstream task models, the fine-grained semantic understanding of mathematical formulas is achieved.
It realizes a fine-grained understanding of mathematical formulas, can better complete tasks such as semantic label classification, formula retrieval and knowledge point tracking, and improves the accuracy of task processing.
Smart Images

Figure CN114611460B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a data processing method and device, an electronic device, and a storage medium. Background Art
[0002] Mathematical formulas are a way of expressing the connections that people discover when studying the relationship between things in nature. Mathematical formulas represent the equal or unequal connections between the quantities of different things in nature. They accurately reflect the internal and external relationships of things and are the basis for people to get from one thing to another. Through mathematical formulas, we can better understand the essence and connotation of things.
[0003] At present, mathematical formulas are widely used in fields such as education, science and technology, and are an important part of understanding related content. Specifically, the feature representation and understanding of mathematical formulas are crucial to research fields such as formula information retrieval and automatic answering. However, mathematical formulas are more difficult and challenging to understand than serialized text due to their complex structure and implicit semantic ambiguity.
[0004] Therefore, it is of great significance to accurately understand the semantics of mathematical formulas and then correctly process tasks based on semantic understanding. Summary of the invention
[0005] The embodiment of the present application provides a data processing method to solve the problem that it is impossible to accurately understand the semantics of mathematical formulas and thus it is impossible to correctly process tasks based on the semantic understanding.
[0006] Correspondingly, an embodiment of the present application also provides a data processing device, an electronic device and a storage medium to ensure the implementation and application of the above method.
[0007] In order to solve the above problems, the present application discloses a data processing method, which includes:
[0008] Get the mathematical formula to be processed;
[0009] Converting the mathematical formula into tree structure data; the tree structure data includes a plurality of tree nodes and directed edges between adjacent tree nodes, the directed edges having corresponding tree edge labels;
[0010] The downstream task model to be trained is trained according to the tree nodes and the tree edge labels, so as to perform task processing according to the trained downstream task model.
[0011] Optionally, converting the mathematical formula into tree structure data includes:
[0012] The mathematical formula is encoded in MathML to obtain tree structure data corresponding to the mathematical formula; wherein the tree structure data at least includes OPT tree structure data.
[0013] Optionally, the training of the downstream task model to be trained according to the tree nodes and the tree edge labels, so as to perform task processing according to the trained downstream task model, includes:
[0014] Generate an input vector corresponding to the tree node according to the tree node and the tree edge label corresponding to the tree node;
[0015] Calculate the local subtree structure feature of the tree node according to the input vector; the local subtree structure feature is used to characterize the local feature of the tree node in the tree structure data;
[0016] Calculating a weight coefficient of the tree node according to an input vector of the tree node; the weight coefficient is used to characterize the criticality of the tree node in the tree structure data;
[0017] According to the weight coefficient corresponding to the tree node, the local subtree structure feature corresponding to the tree node is weighted to obtain the global formula semantic feature;
[0018] The global formula semantic features are used to train the downstream task model to be trained, so as to perform task processing according to the trained downstream task model.
[0019] Optionally, generating an input vector corresponding to the tree node according to the tree node and a tree edge label corresponding to the tree node includes:
[0020] Vectorizing the tree nodes to obtain tree node vectors, and vectorizing the tree edge labels to obtain tree edge label vectors;
[0021] The tree node vector and the tree edge label vector corresponding to the tree node are used as the input vector corresponding to the tree node.
[0022] Optionally, calculating the local subtree structural feature of the tree node according to the input vector includes:
[0023] Calculate the attention weights of the tree node and the direct child nodes of the tree node according to the input vector of the tree node and the input vector of the direct child nodes of the tree structure data;
[0024] Calculate the convergence information of the tree node according to the input vectors and corresponding attention weights of the direct child nodes of the tree node;
[0025] A nonlinear transformation operation is performed on the converged information of the tree node to obtain a local subtree structure feature of the tree node.
[0026] Optionally, calculating the weight coefficient of the tree node according to the input vector of the tree node includes:
[0027] Transposing the input vector of the tree node to obtain the transposed input vector of the tree node;
[0028] The weight coefficient of the tree node is calculated according to the transposed input vector of the tree node and the transposed input vectors of other tree nodes.
[0029] Optionally, the task includes at least one of semantic tag classification, formula retrieval and knowledge point tracking.
[0030] An embodiment of the present application also discloses a data processing device, which includes: a mathematical formula acquisition module, used to acquire the mathematical formula to be processed; a mathematical formula conversion module, used to convert the mathematical formula into tree structure data; the tree structure data includes multiple tree nodes and directed edges between adjacent tree nodes, and the directed edges have corresponding tree edge labels; a task processing module, used to train a downstream task model to be trained according to the tree nodes and the tree edge labels, so as to perform task processing according to the trained downstream task model.
[0031] Optionally, the mathematical formula conversion module is used to perform MathML encoding on the mathematical formula to obtain tree structure data corresponding to the mathematical formula; wherein the tree structure data at least includes OPT tree structure data.
[0032] Optionally, the task processing module is used to generate an input vector corresponding to the tree node based on the tree node and the tree edge label corresponding to the tree node; calculate the local subtree structure feature of the tree node based on the input vector; the local subtree structure feature is used to characterize the local features of the tree node in the tree structure data; calculate the weight coefficient of the tree node based on the input vector of the tree node; the weight coefficient is used to characterize the criticality of the tree node in the tree structure data; weight the local subtree structure features corresponding to the tree node according to the weight coefficient corresponding to the tree node to obtain a global formula semantic feature; use the global formula semantic feature to train the downstream task model to be trained, so as to perform task processing according to the trained downstream task model.
[0033] Optionally, the task processing module is used to vectorize the tree nodes to obtain tree node vectors, and to vectorize the tree edge labels to obtain tree edge label vectors; and to use the tree node vectors and tree edge label vectors corresponding to the tree nodes as the input vectors corresponding to the tree nodes.
[0034] Optionally, the task processing module is used to calculate the attention weights of the tree node and its direct child nodes based on the input vector of the tree node and the input vector of its direct child nodes in the tree structure data; calculate the convergence information of the tree node based on the input vector of the direct child nodes of the tree node and the corresponding attention weights; and perform a nonlinear transformation operation on the convergence information of the tree node to obtain the local subtree structure features of the tree node.
[0035] Optionally, the task processing module is used to transpose the input vector of the tree node to obtain the transposed input vector of the tree node; and calculate the weight coefficient of the tree node based on the transposed input vector of the tree node and the transposed input vectors of other tree nodes.
[0036] Optionally, the task includes at least one of semantic tag classification, formula retrieval and knowledge point tracking.
[0037] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the data processing method as described in one or more of the embodiments of the present application.
[0038] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the data processing method as described in one or more of the embodiments of the present application.
[0039] Compared with the prior art, the embodiments of the present application have the following advantages:
[0040] In an embodiment of the present application, a mathematical formula to be processed is obtained, and the mathematical formula is converted into tree structure data, wherein the tree structure data includes multiple tree nodes and directed edges between adjacent tree nodes, and the directed edges have corresponding tree edge labels. Then, the downstream task model to be trained can be trained according to the tree nodes and the tree edge labels, so that task processing can be performed according to the trained downstream task model. In an embodiment of the present application, in addition to utilizing the spatial information between tree nodes in the tree structure data after the mathematical formula is converted, the tree node labels between tree nodes in the tree structure data are also utilized, so that the formula semantics of the mathematical formula can be understood in a fine-grained manner, thereby enabling the downstream task model trained based on the tree nodes and tree edge labels to better complete the corresponding task. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a schematic diagram of OPT tree structure data of a mathematical formula;
[0042] Figure 2 is a flowchart of steps of an embodiment of a data processing method of the present application;
[0043] Figure 3 is a flowchart of another data processing method embodiment of the present application;
[0044] Figure 4 It is a schematic diagram of an overall framework of the present application based on fine-grained understanding of hierarchical formulas;
[0045] Figure 5 It is a schematic diagram of the search results of two schemes of this application;
[0046] Figure 6 This is a schematic diagram of an elliptic equation and a hyperbola equation of this application
[0047] Figure 7 is a structural block diagram of another data processing device embodiment of the present application;
[0048] Figure 8 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0049] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0050] First, some technical terms are introduced, including:
[0051] OPT: the operator tree, a representation format that converts mathematical formula text into tree-structured data, usually obtained by encoding the mathematical formula in MathML;
[0052] SLT: the symbol layout tree, another representation format that converts mathematical formula text into tree-structured data, usually obtained by Latex encoding of mathematical formulas;
[0053] Attention mask: a way to constrain the calculation of attention weights, commonly seen in transformer network structures;
[0054] MLM: mask language model, a form of pre-training model task, masks the words in the input, and the model predicts the words;
[0055] fastText: fastText is a text classification model based on neural network. The main idea is to use a sliding window of length n to divide the text into a series of segments of length n and input them into the model. After training, the embedding (vector) of each segment is obtained and the average is used as the embedding of the text.
[0056] In specific implementation, mathematical formulas are widely used in education, science and technology, and are an important part of understanding related content. The representation and understanding of mathematical formula features are crucial for research fields such as formula information retrieval and automatic answering. However, mathematical formulas are more difficult and challenging to understand than serialized text due to their complex structure and implicit semantic ambiguity. The specific challenges are summarized as follows:
[0057] 1. The representation of mathematical formulas is flexible and varied. The same mathematical formula can be expressed in multiple forms. For example, the formula of a hyperbola equation can be expressed as:
[0058] x 2 / a 2 -y 2 / b 2 =1 and
[0059] 2. Slight changes in operators can lead to very different meanings of mathematical formulas, such as:
[0060] Quadratic equation of one variable and fractional equations
[0061] Hyperbola equation and the equation of the ellipse
[0062] 3. It is not affected by the order of elements in the mathematical formula and can correctly locate the local key features of the mathematical formula, such as the quadratic equation:
[0063] x 2 +3x+1=9 and 1+3x+x 2 =9
[0064] 4. A mathematical formula may involve multiple tags. For example, a mathematical formula in the field of education may involve multiple knowledge test point tags:
[0065] Radical equation | Logarithmic equation | Fractional equation:
[0066] At present, representative research plans related to mathematical formula understanding are as follows:
[0067] 1. Tangent-CFT treats the tuples of SLT and OPT as a sequence, and uses the fastText n-gram embedding model to obtain tuple embedding. The final formula vector is represented as the average of all tuples. The problem is that it only uses serialized text features, does not fully utilize the formula structured information, and has limited ability to understand the formula in a fine-grained manner.
[0068] 2. MathBert is a pre-trained model in the mathematics field. When modeling mathematical formulas, it uses the OPT tree structure data (opt tree) of mathematical formulas. Each tree node in the OPT tree structure data is only connected to its adjacent tree nodes. Using the attention mask mechanism, the tree node vector representation is only calculated from its adjacent tree nodes. However, this method has some problems: (1) The OPT tree structure data is treated as an undirected graph. After the transformer deep network, each tree node can see the information of other tree nodes, which does not reflect the hierarchical structure information of the mathematical formula; (2) Due to the limitation of the undirected graph form, it does not consider the label information of the edges of the tree nodes. Specifically, refer to Figure 1 , is a schematic diagram of the OPT tree structure data of a mathematical formula, which includes a linear equation: And the fractional equation: It is obvious that the OPT tree structure data of the two mathematical formulas are the same, but the position difference of the variable x in the two mathematical formulas is more important for understanding the semantics of the formulas; (3) Pre-trained models in the mathematics field can only be produced through formula, text matching tasks, and MLM-related tasks, but the formula text matching task cannot accurately understand the semantics of the formula in a fine-grained manner.
[0069] 3. Currently, research programs related to mathematical formula understanding use unsupervised methods, and there is a lack of accurate and effective prior knowledge for the fine-grained semantic understanding of mathematical formulas.
[0070] In response to the above problems, the embodiments of the present application propose a hierarchical fine-grained understanding solution for mathematical formulas, which can provide a fine-grained understanding of mathematical formulas, and thus better complete tasks in the fields of education, science, and technology based on the fine-grained understanding of mathematical formulas.
[0071] Reference Figure 2 , is a flowchart of a data processing method embodiment of the present application, comprising the following steps:
[0072] Step 202: Obtain the mathematical formula to be processed.
[0073] In a specific implementation, the corresponding mathematical formula can be obtained as the mathematical formula to be processed according to the requirements of the task. As an optional example, the task can include at least one of semantic tag classification, formula retrieval and knowledge point tracking, and the corresponding mathematical formula can be obtained as the mathematical formula to be processed for these tasks.
[0074] Step 204: convert the mathematical formula into tree structure data; the tree structure data includes a plurality of tree nodes and directed edges between adjacent tree nodes, and the directed edges have corresponding tree edge labels.
[0075] In an embodiment of the present application, a mathematical formula can be converted into tree structure data with a tree structure through conversion processing. For example, it can be converted into OPT tree structure data. The OPT tree structure data may include multiple tree nodes and directed edges between adjacent tree nodes, and each directed edge has a corresponding tree edge label.
[0076] Step 206: Train the downstream task model to be trained according to the tree nodes and the tree edge labels, so as to perform task processing according to the trained downstream task model.
[0077] In an embodiment of the present application, after obtaining the tree node and tree edge labels, the tree node and tree edge labels can be used to train the downstream task model to be trained, so that corresponding task processing can be performed according to the trained downstream task model, such as realizing semantic label classification, formula retrieval, knowledge point tracking and other tasks.
[0078] Obtain a mathematical formula to be processed, and convert the mathematical formula into tree structure data, wherein the tree structure data includes multiple tree nodes and directed edges between adjacent tree nodes, and the directed edges have corresponding tree edge labels, and then the downstream task model to be trained can be trained according to the tree nodes and the tree edge labels, so that task processing can be performed according to the trained downstream task model. In the embodiment of the present application, in addition to utilizing the spatial information between tree nodes in the tree structure data after the mathematical formula is converted, the tree node labels between tree nodes in the tree structure data are also utilized, so that the formula semantics of the mathematical formula can be understood in a fine-grained manner, thereby enabling the downstream task model that has completed supervised training based on the tree nodes and tree edge labels to better complete the corresponding task.
[0079] Reference Figure 3 , is a flowchart of another data processing method embodiment of the present application, comprising the following steps:
[0080] Step 302: Obtain the mathematical formula to be processed.
[0081] Step 304: convert the mathematical formula into tree structure data; the tree structure data includes a plurality of tree nodes and directed edges between adjacent tree nodes, and the directed edges have corresponding tree edge labels.
[0082] In an exemplary embodiment, the step 304, converting the mathematical formula into tree structure data, may include the following steps:
[0083] The mathematical formula is encoded in MathML to obtain tree structure data corresponding to the mathematical formula; wherein the tree structure data at least includes OPT tree structure data.
[0084] In an embodiment of the present application, MathML encoding can be used to convert a mathematical formula into OPT tree structure data, or Latex encoding can be used to convert a mathematical formula into SLT tree structure data, wherein the tree structure data after the mathematical formula is converted can include multiple tree nodes, adjacent tree nodes include directed edges, and the directed edges have corresponding tree edge labels.
[0085] Exemplarily, given a mathematical formula X, its OPT tree structure data representation is defined as a directed graph G = (N, E, L), where N is the set of tree nodes in the OPT tree structure data, that is, N = {n1, n2, n3 ... n M}, M is the number of tree nodes, and E represents the set of directed edges between adjacent parent and child nodes, that is, <n i , n j >∈E, represents the parent node n i To child node n j There is a directed edge, each directed edge has a corresponding tree edge label (directed edge label), l ij ∈Z (Z represents a set of integers, i.e., l ij The values are 0, 1, 2, 3...) to form a set L, that is, L = {l 1p1 ,l 2p2 ,......l MpM}
[0086] Step 306: Generate an input vector corresponding to the tree node according to the tree node and the tree edge label corresponding to the tree node.
[0087] In an exemplary embodiment, the step 306 of generating an input vector corresponding to the tree node according to the tree node and the tree edge label corresponding to the tree node may include the following steps:
[0088] Vectorizing the tree nodes to obtain tree node vectors, and vectorizing the tree edge labels to obtain tree edge label vectors;
[0089] The tree node vector and the tree edge label vector corresponding to the tree node are used as the input vector corresponding to the tree node.
[0090] Reference Figure 4 , is a schematic diagram of an overall framework based on hierarchical formula fine-grained understanding of the present application, and the overall framework is mainly divided into four modules: ①, input representation layer; ②, formula local structure feature layer (local subtree structure feature layer); ③, formula global semantic feature layer; ④, downstream task layer.
[0091] The input representation layer is mainly composed of two parts: tree nodes and the connection edge information (directed edges) between tree nodes and their parent nodes. The vectorized representation of each tree node is N = {n1, n2, n3...n M}, the tree edge label is vectorized to L = {l 1p1 ,l 2p2 ,......l MpM}, then the input representation layer is finally represented as The input vector is
[0092] Step 308: Calculate the local subtree structural features of the tree node according to the input vector; the local subtree structural features are used to characterize the local features of the tree node in the tree structure data;
[0093] In an exemplary embodiment, the step 308 of calculating the local subtree structural features of the tree node according to the input vector may include the following steps:
[0094] Calculate the attention weights of the tree node and the direct child nodes of the tree node according to the input vector of the tree node and the input vector of the direct child nodes of the tree structure data;
[0095] Calculate the convergence information of the tree node according to the input vectors and corresponding attention weights of the direct child nodes of the tree node;
[0096] A nonlinear transformation operation is performed on the converged information of the tree node to obtain a local subtree structure feature of the tree node.
[0097] In the embodiment of the present application, the local structure feature layer of the formula defines the tree structure data as a directed graph, and customizes a multi-layer attention mechanism to implement constraint information using directed edges. For example, the attention weight from the parent node ni to the child node nj of the tth layer is defined as in Specifically, the transformer network structure includes multiple network layers, which are stacked with multiple identical structures. Therefore, t represents the tth layer of the network. They represent the vector representation of nodes i and j in the tth layer of the network respectively. The inner product of the two represents the attention weight between the two nodes. The denominator is used to normalize the weights among all child nodes. ij The weight is calculated only between directly connected parent and child nodes, and is 0 in other cases. The information after the direct connected child nodes in the subtree where the parent node n1 is located are expressed as The local subtree structure feature of the subtree at the t+1 layer is represented by h t+1 =g(υ t ,h t ,γ t ), where g(.) represents a nonlinear transformation operation, such as Figure 3 In layernorm, feedforward, etc., γ t are the trainable network parameters in g(.).
[0098] Step 310: Calculate the weight coefficient of the tree node according to the input vector of the tree node; the weight coefficient is used to characterize the criticality of the tree node in the tree structure data;
[0099] Step 312: weighting the local subtree structure features corresponding to the tree node according to the weight coefficient corresponding to the tree node to obtain the global formula semantic features;
[0100] In an exemplary embodiment, the step 310 of calculating the weight coefficient of the tree node according to the input vector of the tree node may include the following steps:
[0101] Transposing the input vector of the tree node to obtain the transposed input vector of the tree node;
[0102] The weight coefficient of the tree node is calculated according to the transposed input vector of the tree node and the transposed input vectors of other tree nodes.
[0103] In the embodiment of the present application, the global semantic feature layer weights the local subtree structure features of each tree node, strengthens the key information that is conducive to accurately understanding the semantics of the formula, and weakens the irrelevant marginal information. In the global semantic feature layer, the weight coefficient is defined as The final formula obtained at the global semantic feature layer is expressed as
[0104] Step 314: Use the global formula semantic features to train the downstream task model to be trained, so as to perform task processing according to the trained downstream task model.
[0105] In the embodiments of the present application, after obtaining a fine-grained representation of the formula by combining local structural features and global semantic features, how to use it in the downstream task layer can be determined by the specific task type. The specific task type may include semantic label classification, formula retrieval, knowledge point tracking, etc. The supervised training method of the downstream task is more conducive to the fine-grained semantic understanding of the formula than the unsupervised method.
[0106] In order to better illustrate the technical effects that can be achieved by the embodiments of the present application, formula semantic label classification, formula retrieval, and formula vector visualization are compared and explained respectively:
[0107] 1. Formula semantic label classification: The embodiment of the present application mines and constructs a data set of nearly 1 million semantic labels based on strategies such as rule template matching, hitting contextual keywords, and manual verification to obtain formula semantic labels. These formula semantic labels can be divided into a label system according to actual conditions. For example, the label system can be divided into 4 levels, including 66 leaf labels in total. Of course, it is not limited to this in practice.
[0108] For example, in the first-level label of the 4-level label system and its corresponding leaf labels, the first-level label may include: function class, equation class, trigonometric function class, inequality class, inequality group class, vector class, set class, complex number class, probability class, limit class, integral class. Taking the equation class as an example, further sub-labels may include: equation class `linear equation` linear equation of one variable; equation class `linear equation` linear equation of two variables; equation class `linear equation` linear equation of multiple variables; equation class `geometric equation` linear equation; equation class `geometric equation` equation of circle; equation class `geometric equation` elliptic equation; equation class `geometric equation` parabola equation; equation class `geometric equation` hyperbola equation; equation class `geometric equation` other curve equation; equation class `fractional equation; equation class `quadratic equation of one variable; equation class `exponential equation; equation class `logarithmic equation; equation class `higher order equation; equation class `parametric equation; equation class `polar coordinate equation; equation class `quadratic equation of two variables; equation class `radical equation.
[0109] The evaluation indicators of the classification results of the task model of this application (HTE(ours)) and the task models of other solutions (i.e., FastText(text), FastText(OPT), FastText(SLT), etc.) are compared using this data set as shown in Table 1 below, where the evaluation indicators may include Precision, Recall, and F1-score. Obviously, the numerical values of the evaluation indicators of this application are better, so using this application can achieve better results, and this application can correctly identify the formula semantics of mathematical formulas.
[0110] Table 2:
[0111]
[0112] 2. Formula retrieval: The above data set is used to compare the performance of TangentCFT and this application (HTE(ours)) in the retrieval task. The evaluation indicators may include Bpref (indicating that if the correct answer is ranked before the incorrect answer in the result set, the retrieval is more accurate) and P@N (indicating the accuracy of returning the first N results). The evaluation indicators of the specific classification results are shown in Table 2 below:
[0113] Table 2:
[0114]
[0115] It can be seen that this application is far superior to other similar solutions. Figure 5 , is an example of comparison of the retrieval results of the two schemes. The figure below represents similar formula retrieval. The retrieval formula is a linear function, but it is easily confused with a quadratic function visually. Since TangentCFT focuses on recalling co-occurrence structures, it is easy to miss or recall incorrect knowledge point semantics for complex formulas. Therefore, TangentCFT recalls incorrect results. The HTE(ours) model based on this application can recall the correct linear function results. It can be seen that this application achieves a fine-grained understanding of formula semantics, and can better complete the formula retrieval task.
[0116] 3. Formula vector visualization: refer to Figure 6 , is a schematic diagram of an elliptic equation and a hyperbolic equation of the present application, the figure shows an elliptic equation and a hyperbolic equation, the two are easily confused, the key distinguishing feature is the intermediate operator "+" / "-", the scores in the figure (0.01, 0.06...) are the attention weights of the model for each formula symbol in determining the formula type, the model of the present application is similar to the human processing method, and can correctly focus on the key distinguishing features, thereby correctly distinguishing the formula type. Obviously, the present application has the ability to strengthen key features such as the intermediate operator symbol "plus" / "minus" (ie "+" / "-"), the score of "plus" / "minus" is much larger than the elements in other formulas, and weaken edge information such as numbers ("16" / "25"), thereby correctly distinguishing easily confused equations.
[0117] In summary, the application of the embodiments of the present application has at least the following advantages: 1. A supervised learning framework is designed to achieve hierarchical and fine-grained understanding of formulas from local structural features to global semantic features; 2. In terms of local understanding of formulas, the tree structure is regarded as a directed graph, and information is only allowed to be converged and transmitted from child nodes to their directly connected parent nodes, and side information is introduced to strengthen the fine-grained structural distinction. Ultimately, the root node information of the subtree can represent the local features of the subtree; 3. In terms of global understanding of formulas, the features of each local subtree are weighted based on the attention mechanism, the key information with distinguishing forms is strengthened, and the irrelevant marginal information is weakened, so as to achieve overall semantic understanding of the formula.
[0118] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.
[0119] On the basis of the above embodiments, this embodiment further provides a data processing device, which is applied to electronic devices such as terminal devices and servers.
[0120] Reference Figure 7 , shows a structural block diagram of a data processing device embodiment of the present application, which may specifically include the following modules:
[0121] Mathematical formula acquisition module 702, used to acquire the mathematical formula to be processed;
[0122] A mathematical formula conversion module 704, configured to convert the mathematical formula into tree structure data; the tree structure data includes a plurality of tree nodes and directed edges between adjacent tree nodes, the directed edges having corresponding tree edge labels;
[0123] The task processing module 706 is used to train the downstream task model to be trained according to the tree nodes and the tree edge labels, so as to perform task processing according to the trained downstream task model.
[0124] In an exemplary embodiment, the mathematical formula conversion module 704 is used to perform MathML encoding on the mathematical formula to obtain tree structure data corresponding to the mathematical formula; wherein the tree structure data at least includes OPT tree structure data.
[0125] In an exemplary embodiment, the task processing module 706 is used to generate an input vector corresponding to the tree node based on the tree node and the tree edge label corresponding to the tree node; calculate the local subtree structure feature of the tree node based on the input vector; the local subtree structure feature is used to characterize the local feature of the tree node in the tree structure data; calculate the weight coefficient of the tree node based on the input vector of the tree node; the weight coefficient is used to characterize the criticality of the tree node in the tree structure data; according to the weight coefficient corresponding to the tree node, the local subtree structure feature corresponding to the tree node is weighted to obtain a global formula semantic feature; the global formula semantic feature is used to train the downstream task model to be trained, so as to perform task processing according to the trained downstream task model.
[0126] In an exemplary embodiment, the task processing module 706 is used to vectorize the tree nodes to obtain tree node vectors, and to vectorize the tree edge labels to obtain tree edge label vectors; and to use the tree node vectors and tree edge label vectors corresponding to the tree nodes as the input vectors corresponding to the tree nodes.
[0127] In an exemplary embodiment, the task processing module 706 is used to calculate the attention weights of the tree node and its direct child nodes based on the input vector of the tree node and the input vector of its direct child nodes in the tree structure data; calculate the convergence information of the tree node based on the input vector of the direct child nodes of the tree node and the corresponding attention weights; and perform a nonlinear transformation operation on the convergence information of the tree node to obtain the local subtree structure features of the tree node.
[0128] In an exemplary embodiment, the task processing module 706 is used to transpose the input vector of the tree node to obtain the transposed input vector of the tree node; and calculate the weight coefficient of the tree node based on the transposed input vector of the tree node and the transposed input vectors of other tree nodes.
[0129] In an exemplary embodiment, the task includes at least one of semantic tag classification, formula retrieval, and knowledge point tracking.
[0130] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0131] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.
[0132] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).
[0133] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 8 An exemplary apparatus 800 that can be used to implement various embodiments described in this application is schematically illustrated.
[0134] For one embodiment, Figure 8 An exemplary apparatus 800 is shown having one or more processors 802, a control module (chip set) 804 coupled to at least one of the processor(s) 802, a memory 806 coupled to the control module 804, a non-volatile memory (NVM) / storage device 808 coupled to the control module 804, one or more input / output devices 810 coupled to the control module 804, and a network interface 812 coupled to the control module 804.
[0135] The processor 802 may include one or more single-core or multi-core processors, and the processor 802 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 800 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.
[0136] In some embodiments, the apparatus 800 may include one or more computer-readable media (e.g., memory 806 or NVM / storage device 808) having instructions 814 and one or more processors 802 combined with the one or more computer-readable media and configured to execute the instructions 814 to implement a module to perform the actions described in the present disclosure.
[0137] For one embodiment, the control module 804 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 802 and / or any suitable device or component in communication with the control module 804 .
[0138] The control module 804 may include a memory controller module to provide an interface to the memory 806. The memory controller module may be a hardware module, a software module, and / or a firmware module.
[0139] The memory 806 may be used, for example, to load and store data and / or instructions 814 for the device 800. For one embodiment, the memory 806 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 806 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).
[0140] For one embodiment, control module 804 may include one or more input / output controllers to provide an interface to NVM / storage device 808 and input / output device(s) 810 .
[0141] For example, NVM / storage 808 may be used to store data and / or instructions 814. NVM / storage 808 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).
[0142] NVM / storage device 808 may include storage resources that are physically part of the device on which apparatus 800 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 808 may be accessed via input / output device(s) 810 over a network.
[0143] (One or more) input / output devices 810 may provide an interface for the apparatus 800 to communicate with any other appropriate device, and the input / output device 810 may include a communication component, an audio component, a sensor component, etc. The network interface 812 may provide an interface for the apparatus 800 to communicate through one or more networks, and the apparatus 800 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.
[0144] For one embodiment, at least one of the processor(s) 802 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 804. For one embodiment, at least one of the processor(s) 802 may be packaged together with the logic of one or more controllers of the control module 804 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 802 may be integrated on the same die with the logic of one or more controllers of the control module 804. For one embodiment, at least one of the processor(s) 802 may be integrated on the same die with the logic of one or more controllers of the control module 804 to form a system-on-chip (SoC).
[0145] In various embodiments, the device 800 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 800 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 800 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.
[0146] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.
[0147] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0148] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0149] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0152] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0153] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0154] The data processing method and device, an electronic device and a storage medium provided by the present application are introduced in detail above. The principles and implementation methods of the present application are explained in this article using specific examples. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data processing method, characterized in that: The method comprises: Get the mathematical formula to be processed; Converting the mathematical formula into tree structure data; the tree structure data includes a plurality of tree nodes and directed edges between adjacent tree nodes, the directed edges having corresponding tree edge labels; According to the tree nodes and the tree edge labels, the downstream task model to be trained is trained, so as to perform task processing according to the trained downstream task model, including: Generate an input vector corresponding to the tree node according to the tree node and the tree edge label corresponding to the tree node; Calculate the local subtree structure feature of the tree node according to the input vector; the local subtree structure feature is used to characterize the local feature of the tree node in the tree structure data; Calculating a weight coefficient of the tree node according to an input vector of the tree node; the weight coefficient is used to characterize the criticality of the tree node in the tree structure data; According to the weight coefficient corresponding to the tree node, the local subtree structure feature corresponding to the tree node is weighted to obtain the global formula semantic feature; The global formula semantic features are used to train the downstream task model to be trained, so as to perform task processing according to the trained downstream task model.
2. The method according to claim 1, characterized in that The converting the mathematical formula into tree structure data comprises: The mathematical formula is encoded in MathML to obtain tree structure data corresponding to the mathematical formula; wherein the tree structure data at least includes OPT tree structure data.
3. The method according to claim 1, characterized in that The step of generating an input vector corresponding to the tree node according to the tree node and the tree edge label corresponding to the tree node comprises: Vectorizing the tree nodes to obtain tree node vectors, and vectorizing the tree edge labels to obtain tree edge label vectors; The tree node vector and the tree edge label vector corresponding to the tree node are used as the input vector corresponding to the tree node.
4. The method according to claim 1, characterized in that: The calculating the local subtree structural feature of the tree node according to the input vector comprises: Calculate the attention weights of the tree node and the direct child nodes of the tree node according to the input vector of the tree node and the input vector of the direct child nodes of the tree structure data; Calculate the convergence information of the tree node according to the input vectors and corresponding attention weights of the direct child nodes of the tree node; A nonlinear transformation operation is performed on the converged information of the tree node to obtain a local subtree structure feature of the tree node.
5. The method according to claim 1, characterized in that The step of calculating the weight coefficient of the tree node according to the input vector of the tree node comprises: Transposing the input vector of the tree node to obtain the transposed input vector of the tree node; The weight coefficient of the tree node is calculated according to the transposed input vector of the tree node and the transposed input vectors of other tree nodes.
6. The method according to claim 1, characterized in that The task includes at least one of semantic tag classification, formula retrieval and knowledge point tracking.
7. A data processing device, characterized in that: The device comprises: A mathematical formula acquisition module, used to acquire the mathematical formula to be processed; A mathematical formula conversion module, used for converting the mathematical formula into tree structure data; the tree structure data includes a plurality of tree nodes and directed edges between adjacent tree nodes, the directed edges having corresponding tree edge labels; A task processing module is used to generate an input vector corresponding to the tree node according to the tree node and the tree edge label corresponding to the tree node; calculate the local subtree structure feature of the tree node according to the input vector; the local subtree structure feature is used to characterize the local feature of the tree node in the tree structure data; calculate the weight coefficient of the tree node according to the input vector of the tree node; the weight coefficient is used to characterize the criticality of the tree node in the tree structure data; according to the weight coefficient corresponding to the tree node, the local subtree structure feature corresponding to the tree node is weighted to obtain a global formula semantic feature; the global formula semantic feature is used to train the downstream task model to be trained, so as to perform task processing according to the trained downstream task model.
8. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the data processing method as claimed in one or more of claims 1-6.
9. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to perform the data processing method according to one or more of claims 1-6.
Citation Information
Patent Citations
Semantic parsing method and device of mathematical formula
CN107301164A
Method for extracting semantic information of mathematical formula, apparatus and computer-readable recording medium with program therefor
KR1020120095569A
