Methods, systems, apparatus, and media for training and applying machine learning models

Abstract

A method for training a machine learning model is disclosed, comprising obtaining a training set including multiple sentences and their labeled sets; generating parse trees and label trees for the sentences; determining the total loss of the sentence, which is equal to the weighted sum of the parse tree generation loss and the label tree generation loss; and training the machine learning model using the training set to minimize the total loss. This application also relates to systems, apparatus, and media for training machine learning models and for processing sentences using such machine learning models.

Description

Technical Field

[0001] This application relates to machine learning models, and more particularly to methods, systems, apparatuses, and computer-readable storage media for training and applying machine learning models. Background Technology

[0002] Machine learning models are now widely used in natural language processing for scenarios such as predictive interpretation of text, text mining, and multi-intent understanding. For example, a machine learning model can predict one or more labels for a sentence.

[0003] Deep neural networks (NNs) offer excellent performance for natural language processing. However, a lack of interpretability is an inherent problem for NNs. Although many methods have been proposed to provide post-hoc interpretation for NNs, these methods still have limitations, such as the inability to interpret at the phrase and clause levels.

[0004] Therefore, there is a need for solutions that can perform interpretable natural language processing on sentences. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, one or more embodiments of this specification combine deep neural networks and symbolic probability models through a language structuring model, providing a natural language processing scheme with better interpretability.

[0006] One or more embodiments of this specification achieve their above-mentioned objectives through the following technical solutions.

[0007] In one aspect, a method for training a machine learning model is provided, the method comprising: acquiring a training set including a plurality of training samples, each training sample including a sentence and a set of labeled labels for the sentence; generating a parse tree of the sentence using a first machine learning model, the parse tree having a binary tree structure; generating a latent label tree of the sentence using a second machine learning model, the latent label tree having the same structure as the parse tree; determining a label tree generation loss for generating the latent label tree, wherein the label tree generation loss is negatively correlated with the probability that a set of labels collected from the latent label tree based on a predefined collection rule is the same as the set of labeled labels for the sentence; and training the second machine learning model using the training set to minimize the label tree generation loss.

[0008] Preferably, the probability that the tag set collected based on predefined collection rules is the same as the labeled tag set of the sentence is equal to the sum of the probabilities of all possible subspaces in the latent tag tree that the tag set collected based on predefined rules is the same as the labeled tag set.

[0009] Preferably, the first machine learning model is untrained, and the method further includes: determining the total loss of the sentence, the total loss being equal to the weighted sum of the parse tree generation loss for generating the parse tree and the label tree generation loss for generating the latent label tree; and using the training set, simultaneously training the first machine learning model and the second machine learning model to minimize the total loss.

[0010] Preferably, the Fast-R2D2 model is used to generate the parse tree of the sentence.

[0011] Preferably, the nodes in the potential label tree include non-terminating nodes and terminating label nodes, wherein the label of the terminating label node is a task label.

[0012] Preferably, the potential label tree further includes terminating empty label nodes.

[0013] Preferably, the predefined collection rule is: the task label of a node is collected only when a node in the potential label tree has a task label and all of the node's ancestor nodes are non-terminating nodes.

[0014] Preferably, dynamic programming is used to traverse the parse tree.

[0015] Preferably, using dynamic programming to traverse the parse tree includes stopping traversing the child nodes of the terminal node when a terminal node is encountered.

[0016] Preferably, using dynamic programming to traverse the parse tree includes calculating the probability while assuming that two non-overlapping spans of the parse tree are not associated with the same task label.

[0017] Preferably, the representation of each span of the parse tree is converted from bottom-up encoding to top-down encoding using the Transformer model.

[0018] On the other hand, a method for performing processing on a sentence is provided, the method comprising: acquiring a sentence to be processed; predicting a parse tree of the sentence using a trained first machine learning model, the parse tree having a binary tree structure; and predicting a label tree of the sentence using a trained second machine learning model, the label tree having the same structure as the parse tree, wherein predicting the label tree of the sentence includes predicting the label of each node in the label tree.

[0019] Preferably, the method further includes: using predefined collection rules to collect tags in the tag tree to generate a tag set for the sentence.

[0020] Preferably, the method further includes: wherein the second machine learning model is trained according to the method of any one of claims 1-11.

[0021] On the other hand, a system for training a machine learning model is provided, comprising: a training set acquisition module for acquiring a training set including multiple training samples, each training sample including a sentence and a set of labeled labels for the sentence; a parse tree generation module for generating a parse tree of the sentence using a first machine learning model, the parse tree having a binary tree structure; a label tree generation module for generating a latent label tree of the sentence using a second machine learning model, the latent label tree having the same structure as the parse tree; and a training module for determining a label tree generation loss for generating the label tree, wherein the label tree generation loss is negatively correlated with the probability that the set of labels collected from the latent label tree based on a predefined collection rule is the same as the set of labeled labels for the sentence, and for training the second machine learning model using the training set to minimize the label tree generation loss.

[0022] Preferably, the nodes in the tag tree include non-terminating nodes, terminating tag nodes, and terminating empty tag nodes, wherein the tag of the terminating tag node is a task tag, and the predefined collection rule is: the task tag of a node is collected only when a node in the tag tree has a task tag and all of the node's ancestor nodes are non-terminating nodes.

[0023] In another aspect, an apparatus for processing sentences is provided, comprising: a memory; and a processor configured to perform the above-described method.

[0024] In another aspect, a computer-readable storage medium is provided that stores instructions which, when executed by a computer, cause the computer to perform the methods described above.

[0025] Compared with the prior art, one or more embodiments of this specification can achieve one or more of the following technical effects:

[0026] It provides an interpretable natural language processing model;

[0027] It can obtain the tag tree of sentences;

[0028] It can improve processing efficiency; and

[0029] It has good processing performance. Attached Figure Description

[0030] The above-described invention and the following detailed embodiments will be better understood when read in conjunction with the accompanying drawings. It should be noted that the drawings are merely examples of the claimed invention. In the drawings, the same reference numerals represent the same or similar elements.

[0031] Figure 1 A schematic diagram of an example tag tree generated according to an embodiment of this specification is shown.

[0032] Figure 2 A schematic diagram of an example parse tree according to an embodiment of this specification is shown.

[0033] Figure 3 A schematic diagram of the various types of nodes in an example tag tree according to an embodiment of this specification is shown.

[0034] Figure 4 An example of pseudocode for the yield function according to an embodiment of this specification is shown.

[0035] Figure 5 A schematic diagram illustrating the operation of the yield function according to an embodiment of this specification is shown.

[0036] Figure 6 This illustrates potential combinations of tag trees according to embodiments of this specification.

[0037] Figure 7 A schematic diagram is shown of a Transformer model for converting bottom-up encoding to top-down encoding according to an embodiment of this specification.

[0038] Figure 8 A schematic flowchart illustrating an example method for training a machine learning model according to an embodiment of this specification is shown.

[0039] Figure 9 A schematic flowchart of an example method for performing sentence processing according to an embodiment of this specification is shown.

[0040] Figure 10 A schematic block diagram of an example system for training a machine learning model according to an embodiment of this specification is shown.

[0041] Figure 11 A schematic block diagram of an apparatus for implementing a system according to one or more embodiments of this specification is shown. Detailed Implementation

[0042] The following detailed description is sufficient to enable any person skilled in the art to understand the technical content of one or more embodiments of this specification and to implement them accordingly. Furthermore, based on the specification, claims, and drawings disclosed in this specification, those skilled in the art can easily understand the objectives and advantages associated with one or more embodiments of this specification.

[0043] Deep neural networks (DNNs) exhibit good performance in natural language processing (such as text classification tasks). However, DNNs based on stacked layer architectures have historically suffered from inherently poor interpretability. Meanwhile, symbolic probabilistic models possess clear interpretability, but their integration with DNNs remains to be explored. In this specification's embodiments, an attempt is made to combine deep neural networks and symbolic probabilistic models through a language structuring model, proposing a novel label extraction framework based on binary syntax trees. Experiments show that the scheme in this specification achieves good prediction results in single / multi-label text classification tasks and possesses clear and inherent span-level interpretability.

[0044] In a preferred embodiment of this specification, dynamic programming is also used to reduce computational complexity. In another preferred embodiment of this specification, a top-down coding approach, such as using the Transformer model, is employed to further improve model performance.

[0045] To visually illustrate the effects achievable by the solutions according to the embodiments described herein, please refer to... Figure 1 This illustrates a schematic diagram of an example tag tree 100 generated according to embodiments of this specification. Figure 1 As shown, the embodiments of this specification not only generate a set of tags for sentences, but also indicate which word caused the generation of the tags in the tag set.

[0046] like Figure 1 As shown, for the sentence "yes, please, send it on my screen and pick a route to avoid all heavy traffic", the scheme according to the embodiment of this specification can generate a tag set {cfm, show, req, info}, which respectively represent confirm, show_in_screen, request_route, and inform. Furthermore, these tags also correspond to nodes in a binary tree and indicate the correspondence with sentence components.

[0047] Figure 1 The binary tree in the code also includes other symbols, which will be explained in more detail below.

[0048] To provide interpretability at the level of words, phrases, or clauses, in embodiments of this specification, a parse tree of a sentence may be generated. A parse tree (also called an analysis tree, derivation tree, inference tree, derivation relation tree, concrete syntax tree, etc.) is used to structure sentences. In embodiments of this specification, the parse tree may have a binary tree structure.

[0049] See Figure 2 This illustrates a schematic diagram of an example parse tree 200 according to an embodiment of this specification. Figure 2 As shown, the sentence "John hit the ball" is parsed into a binary tree, where the root node is the entire sentence and the leaf nodes are each word in the sentence. Words, based on grammatical structure, constitute sentence components at different levels. For example, "the ball" constitutes one sentence component (represented in parse tree 200 as a subtree rooted at sentence component node 202), while "hit the ball" constitutes another sentence component (represented in parse tree 200 as a subtree rooted at sentence component node 204), and one sentence component may be included within another. Figure 2 In this context, S represents the entire sentence, NP represents a noun phrase, VP represents a phrasal verb, V represents a verb, N represents a noun, and Det represents a determiner. Understanding this is important. Figure 2 The parse tree shown is just an example; other forms of parse trees can be used.

[0050] Various language structuring models can be used to generate parse trees for sentences.

[0051] In a preferred embodiment of this specification, the Fast-R2D2 model is used to generate the parse tree of the sentence. Fast-R2D2 is particularly suitable for the scenario described in this specification due to the following characteristics: First, the Fast-R2D2 model has the ability to learn reasonable syntactic structures in an unsupervised manner, thus eliminating the need for manual annotation, saving manpower and improving efficiency; second, each sentence component node has an intermediate representation; and third, the Fast-R2D2 model allows for the use of pre-training mechanisms to improve representation performance.

[0052] Fast-R2D2 has two components: a top-down parser and a graph-based encoder. For a given sentence S, the distribution of the parser tree estimated by the parser and encoder can be represented as q. φ (z|S) and p θ (z|S). The graph is pruned by a parser (preferably a top-down parser) to achieve linear coding complexity, and by minimizing D... KL [q φ (z|S)||p θ(z|S)](which is represented as The parser is optimized using a bidirectional language model loss mechanism proposed in R2D2. The encoder is pre-trained. This is achieved by interleaving the two losses. and The parser and encoder can promote each other (similar to the policy and value network in the AlphaZero model).

[0053] For more details on R2D2, please refer to the paper titled "R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling" by Xiang Hu et al. (see https: / / aclanthology.org / 2021.acl-long.379 / ). For more details on Fast-R2D2, please refer to the paper titled "Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representation" by Xiang Xu et al. (see https: / / arxiv.org / abs / 2203.00281).

[0054] It should be understood that although Fast-R2D2 is used as the language structuring model to generate the parse tree of the sentence in the preferred embodiment, the embodiments in this specification are not limited thereto. Other suitable language structuring models may be used, such as R2D2, ON-LSTM, DIORA, C-PCFG, etc.

[0055] It should be understood that the language structuring model can be trained separately or in conjunction with the machine learning model used to generate the label trees, as described below. For example, in one scenario, the trained language structuring model can be used to generate the parse tree, eliminating the need to consider the parse tree generation loss during the training of the final machine learning model for generating the label tree. In another scenario, the language structuring model can be trained simultaneously with the model used to generate the label tree, resulting in a synergistic model. This simultaneous training may further improve the overall performance of the model.

[0056] In the embodiments of this specification, as described above, sentence components can be represented as spans in a binary tree-like parse tree. In these embodiments, since a binary tree is used, the span can be represented by the root of that span. In this specification, "span" and "node" are sometimes used interchangeably; in this case, "node" should be interpreted as the span of the tree rooted at that node.

[0057] Therefore, a softmax layer can be used to associate nodes in the parse tree with labels, ultimately forming a label tree that can have the same structure as the parse tree.

[0058] Nevertheless, the embodiments in this specification are also applicable to sentences in which a sentence component has more than one label.

[0059] Next, we will define the tags for the tag tree.

[0060] In this illustrative embodiment, concepts from context-free grammar (Hopcroft & Ullman, 1979) are used to define two types of tags: terminating tags and non-terminating tags. A terminating tag is a tag in a tag set and can have different terminating tags (corresponding to different task tags). A non-terminating tag is simply a tag, which can be represented as "non-terminating tag". For a given sentence and its parse tree, each node in the parse tree can be associated with a terminating or non-terminating tag, thus transforming the parse tree into a tag tree. It is important to note that task tags are all terminating tags, while non-terminating tags are not associated with any task tags. A task tag refers to a tag defined by a task; for example, in a classification task, the goal is to classify a sentence into a category tag selected from multiple predefined category tags, where these multiple predefined category tags are all task tags.

[0061] In the embodiments of this specification, when a parent span and its child spans are associated with different termination labels, only the label of the parent span is retained. For example, in the span "Not good enough", its child span "Good enough" is affirmative, but the entire span "Not good enough" should be negative; therefore, only the label of the parent span "Not good enough", i.e., the negative label, is retained. Considering that in some cases, a parent span may not be associated with a task label, but its child spans are, for example, the span "Neither happy nor sad" has no emotional polarity, but its child span "Unhappy" has negative emotion, while "Not sad" has positive emotion.

[0062] In a preferred embodiment of this specification, for completeness, in addition to "terminal label" and "non-terminal label," a terminal node with an empty label (i.e., without a label) is also introduced. A node with a "non-terminal label" is referred to as a "non-terminal node," and is denoted herein as φ. NT (like Figure 1(As shown in the image); Nodes with labels such as "Termination Label" are called "Termination Label Nodes," which are represented by their corresponding task labels in this paper. Different termination label nodes may have different labels; Nodes with empty labels (i.e., no labels) are called "Termination Empty Label Nodes," which are represented as φ in this paper. T In other words, there is one non-terminating node label, N terminating label nodes (N is the number of task labels), and one terminating empty label node label (empty label or no label).

[0063] See Figure 3 This diagram illustrates various types of nodes in an example tag tree 300 according to an embodiment of this specification. The tag tree 300 described here is merely for illustrating the structure of a tag tree and is not intended to be related to other tag trees. Figure 2 The parse tree 200 is unrelated. For example... Figure 3 As shown, white nodes 302 and 304, and dark gray nodes 310 and 318 are non-terminating nodes, while light gray node 314 is a terminating empty label node. Light gray nodes 306 and 312, and dark gray nodes 308 and 316 are terminating label nodes, with labels A, C, D, and B respectively.

[0064] The goal of generating a label tree in the embodiments of this specification is to find the most appropriate label for each node, which can be a "non-terminating label" (corresponding to a "non-terminating node"), no label (corresponding to a "terminating empty label node"), or a task label selected from a predefined set of task labels (corresponding to a "terminating label node").

[0065] It can be seen that the embodiments of this specification are best suited for sentences with the following characteristics: one sentence component in the text corresponds to at most one tag, that is, there is no nested tag text. In fact, most single-tag and multi-intent problems have this characteristic. For ease of description, this specification uses text without nested tags as an example. However, it should be understood that by allowing non-terminal tags to be associated with task tags, the embodiments of this specification are still feasible for text with nested tags, and those skilled in the art, after reading the specific implementation of the embodiments of this specification, can also implement the solutions of this specification for text with nested tags.

[0066] After generating the tag tree, tags in the tag tree can be collected based on predefined collection rules.

[0067] Preferably, the predefined collection rule is as follows: a node's task label is collected only if it has a task label and all of its ancestor nodes are non-terminating nodes. In other words, only the task labels of nodes whose ancestor nodes are all non-terminating nodes are collected, and if a node's ancestor node has a task label, that node's label is omitted. (Continue) Figure 3 For example, given four task labels A, B, C, and D, since node 316 with task label D has its parent node 314 as the terminating node, label D will not be collected; since node 308 with task label B has its parent node 306 with task label A, label B will be omitted and not collected. Finally, Figure 3 The tag set formed by the tags collected in the tag tree is {A,C}.

[0068] Preferably, the `yield(generator)` function can be used to collect tags from the tag tree based on this predefined collection rule. See also Figure 4 This illustrates an example of pseudocode 400 for the yield function according to an embodiment of this specification.

[0069] Since the labels of the child nodes of the terminal node are not collected, it is not necessary to visit its child nodes again after reaching the terminal node when traversing the label tree. For example, see Figure 5 The schematic diagram of the operation mode 500 of the yield function according to the embodiments of this specification, combined with Figure 5 Let's interpret Figure 4 As can be seen from the `yield` function, it iterates from the root node. If a non-terminating node is encountered, it continues traversing its child nodes. If a terminating tag node is encountered, its tag type is added to the return set, and no further child nodes are visited. If a terminating empty tag node is encountered, no further child nodes are visited, and it is not added to the return tag set. This method reduces the number of nodes visited, improving system efficiency.

[0070] Other methods that can be conceived by those skilled in the art can be used to collect tags from the tag tree based on the predefined collection rules. For example, all nodes can be traversed, or other suitable algorithms can be used.

[0071] Given a sentence S and its optimal parse tree t, and one of its potential label trees represented as... Please note, It has the same tree structure as t. For ease of description, a binary tree is defined using its root.

[0072] make The yield function is used for the labeled tag set (also known as the golden tag set) of sentence S. Train the model so that the predicted labels in the label tree are Maximize the sum of probabilities (or minimize its negative logarithm, as shown in Formula 3 below).

[0073] Let t i，jDenote a tree with span from i to j (where i and j are the sequence numbers of the start node and end node of the tree in the complete tree of the complete sentence S and i < j), and its left and right child nodes are t i，k and t k+1，j . It can be understood that the value of k is determined by the structure of the tree t i，j , so the value of k is given by a language structuring model (such as Fast - R2D2 or other adopted language structuring models). For each node t i，j , its corresponding representation is e i，j , and its corresponding label probability distribution is P(·|t i，j ) = softmax(MLP(e i，j )). That is to say, the probability distribution of the corresponding label of each node can be obtained by performing a multi - layer perceptron operation on the representation of the node and then performing a softmax operation. In this way, the probability distribution of the label of each node can be obtained.

[0074] For a given tree t i，j and any label set , the sum of probabilities of obtaining the label set i，j from the tree t based on the collection rule (such as through the yield function) can be denoted as , that is , here it is abbreviated as The potential label tree that can obtain the labeled label set based on the collection rule can be called a "reasonable label tree", that is, only reasonable label trees are considered when calculating the loss. Therefore, the most direct way to estimate

[0075] is to exhaust all possible sub - spaces and sum them up through dynamic programming. Let

[0076] denote the set pair that satisfies and , and the set containing all valid and and is denoted as Next, it will be discussed how to calculate from See Figure 6 , which shows the possible sub - spaces in the case of according to the embodiments of this specification, where the left side shows the case of , and the right side shows the case of its (that is ).

[0077] For the case of (that is, M = φ), then in and in the case of t i，j is φ T or φ NT (that is, there will be no task label in this case).

[0078] Combined with Figure 6 it can be seen that for t i，j , in the case of i < j, the transition function for calculating is:

[0079]

[0080] while in the case of i = < j, the transition function for calculating is:

[0081]

[0082] This transition function can work in a bottom-up manner and iterate over all In this way, is the final sum of probabilities, and one of the goals of training this function is to minimize the negative logarithm of this final sum of probabilities. In this way, the label tree structured loss function is constructed:

[0083]

[0084] It should be understood that although the negative logarithm is used above, other negatively correlated functions can be adopted. In other words, the label tree generation loss is negatively correlated with the probability that the label set collected from the label tree based on the predefined collection rules is the same as the labeled label set, for example, it is the negative logarithm of this probability.

[0085] However, it can be understood that iterating over all and will have exponential complexity.

[0086] To solve the problem of exponential complexity, in the preferred embodiment of this specification, is reformulated as where and respectively represent the cases where there exists a and satisfying . ]>represents the entire task label set, that is, it includes all labels from which task labels can be selected. Let represent the case where there exists a satisfying . By assuming that the label states do not affect each other, we can obtain:

[0087]

[0088]

[0089] In this embodiment, estimation is not performed by approximation. Because it can be calculated directly as a whole. When the above assumptions hold, the above approximation becomes an equation.

[0090] For the sake of brevity, Represented as and Figure 6 In Similarly, if a label tree exists satisfy Then t i，j The tag can only belong to {l, φ NT}. When t i，j For φ NT At that time, it will definitely satisfy or Therefore, the state transition equations for the nodes in t can be established as follows:

[0091]

[0092] The above transfer function assumes that multiple non-overlapping spans can be associated with the same label.

[0093] To further reduce complexity, in some cases, we can assume a mutual exclusion constraint, meaning that two non-overlapping spans of the parse tree are not associated with the same task label. In other words, when calculating probabilities, two non-overlapping spans are not allowed to be associated with the same task label. The transition function can then be rewritten as:

[0094]

[0095] Under this assumption, if the label of the current node is not 'l', then only one of its left or right child nodes is allowed to be associated with 'l'.

[0096] Similarly, it can be Represented as Similarly, we can obtain:

[0097]

[0098] therefore And the objective function of the given parse tree is:

[0099]

[0100] Similarly, it should be understood that the above negative logarithm is an example of negative correlation, and other negative correlation functions can be used.

[0101] As can be seen, by adopting dynamic programming as described above, exponential complexity can be eliminated, thereby greatly reducing the demand for computing resources and improving the efficiency of the algorithm.

[0102] The above losses This represents the label tree generation loss for generating the label tree. Therefore, the machine learning model can be trained to minimize this label tree generation loss, resulting in a trained model for generating label trees. In the case of training the model for generating label trees separately, the model for generating parse trees (e.g., a language structuring model) can be pre-trained.

[0103] In a preferred embodiment, the language structuring model and the model used to generate the tag tree (hereinafter referred to as the "tag tree generation model") can be trained simultaneously to obtain a comprehensive model. This simultaneous training may further improve the overall performance of the model.

[0104] In this case, an untrained language structuring model can be used. There is also a loss involved in generating the parse tree using the language structuring model, which can be called the parse tree generation loss and is also considered in the final loss. Previous work (e.g., the Fast-R2D2 model mentioned above) has demonstrated that better downstream performance can be achieved by training the label tree generation loss together with the parse tree generation loss.

[0105] Therefore, the total loss of the model can be expressed as:

[0106]

[0107] in The loss for generating parse trees in a language structured model, where α is The weights are determined in any manner that can be conceived by a person skilled in the art.

[0108] In a simple example, the weights can be set to be the same, so the total loss of the model can be expressed as:

[0109]

[0110] As stated in the paper, the loss function of the Fast-R2D2 model is... and Therefore, when Fast-R2D2 is used as the language structuring model, the total loss can be expressed as:

[0111]

[0112] Where q φ(z|S) represents the probability of the tree estimated by the parser in Fast-R2D2. For details, please refer to the relevant description of the Fast-R2D2 model.

[0113] Although weights are not used in the above equations, it should be understood that the total loss can also be defined as the weighted sum of the three.

[0114] Because recursive language models follow a bottom-up hierarchical encoding process, the context outside the sub-span is not visible to it, which may cause low-level spans to fail to provide correct labels. In a preferred embodiment of this specification, a top-down process is constructed to fuse internal and external features to take the context into account, thereby improving the prediction accuracy of low-level spans.

[0115] For a given span (i,j), its bottom-up representation is e. i，j Let its top-down representation be e′ i，j In a preferred embodiment of this specification, a Transformer model is used as a top-down encoder function, with the form [x′1, x′2, ... x′]. k ] = f([x1, x2, ..., x k The top-down decoding process starts from the root and recursively executes on the child nodes. Once the top-down representation e′ is complete... i，j Once you're ready, you can access [·, e′] i，k ,e′ k+1，j ]=f([e′ i，j e i，k e k+1，j ]) to calculate its sub-representation.

[0116] In particular, for the root node, we have [·, e′] 1，n ] = f([e root e 1，n ]), where e root It is an embedding of the special tag [ROOT].

[0117] See Figure 7 It shows a schematic diagram of a Transformer model 700 for converting bottom-up encoding to top-down encoding according to an embodiment of this specification.

[0118] exist Figure 7 In the code, [PRT], [LEFT], and [RIGHT] are the role embeddings corresponding to the input, where [PRT] represents the parent node, [LEFT] represents the left leaf node, and [RIGHT] represents the right leaf node. The code encodes [e′] from bottom to top. i，j e i，k e k+1，j] represents the top-down encoding [·, e′ i，k ,e′ k+1，j ].

[0119] In this way, the representation of each span of the parse tree is transformed from bottom-up encoding to top-down encoding, thereby taking into account the context and improving the prediction accuracy of low-level sub-spans.

[0120] Having detailed the specifics of the machine learning model according to embodiments of this specification above, the following describes the process for training and applying the machine learning model according to embodiments of this specification. The description of these processes can be found in the detailed description above.

[0121] See Figure 8 It shows a schematic flowchart of an example method 800 for training a machine learning model according to an embodiment of this specification.

[0122] like Figure 8 As shown, method 800 may include: acquiring a training set. The training set may include multiple training samples. Each training sample may include a sentence and a set of labeled tags for that sentence. The training set may be acquired from various data sources conceivable to those skilled in the art, such as various private or public corpora (e.g., ATIS, MITMovie, etc.). The training samples in the training set may be manually labeled or otherwise acquired (e.g., labeled using other algorithms and possibly verified manually). Where necessary, the training set may be preprocessed.

[0123] Preferably, the training samples in the training set may only include the label set of the entire sentence, without indicating the correspondence between the labels in the label set and sentence components. For example, with Figure 1 For example, in the sentence "yes, please, send it on my screen and pick a route to avoid all heavy traffic", the tag set may only contain the tag set {cfm,show,req,info} of the sentence, without indicating the correspondence between the tags in the tag set and the words or phrases in the sentence.

[0124] Although called a "tag set", a tag set can include not only multiple tags, but also just a single tag. In some cases, a "tag set" may not contain any tags, that is, the tag set is an empty set.

[0125] Method 800 may include, in operation 804, using a first machine learning model to generate a parse tree for the sentence. The parse tree may have a binary tree structure.

[0126] For an example of a parse tree, please refer to the section above. Figure 2 The first machine learning model can be a language structuring model as described above. Preferably, the Fast-R2D2 model can be used to generate the parse tree of the sentence. As described above, any other method that can be conceived by those skilled in the art can be used to generate the parse tree. Further details about parse trees and parse trees can be found in the description above.

[0127] Method 800 may include: in operation 806, using a second machine learning model to generate a label tree for the sentence.

[0128] As described above and shown in the accompanying drawings, the label tree may have the same binary tree structure. Preferably, the nodes in the label tree may include non-terminating nodes and terminating label nodes, wherein the label of the terminating label node is the task label of the prediction task of the machine learning model (e.g., any label described in the label set above). The label of the non-terminating node is simply called a "non-terminating label". Preferably, the label tree also includes terminating empty label nodes, whose labels are either "terminating empty label" or no label.

[0129] Preferably, the tag tree and the parse tree have the same structure. The same structure means that there is a one-to-one correspondence between the nodes of the two trees, and that the nodes are in the same position within the tree.

[0130] Predefined collection rules include, for example, collecting a node's task label only if the node in the label tree has a task label and all of the node's ancestor nodes are non-terminating nodes.

[0131] In the embodiments described in this specification, the collection of tags based on predefined collection rules can be implemented using the `yield` function. Alternatively, any other suitable method can be used to implement the predefined collection rules. Details regarding the predefined collection rules and their implementation can be found in the description above. As mentioned above, dynamic programming can be used to traverse the parse tree. For example, as described in Equations 1 to 11 and the related descriptions above.

[0132] Preferably, the representation of each span of the parse tree is encoded in a top-down manner. Preferably, a Transformer model can be used to convert the bottom-up encoding of the representation of each span of the parse tree to a top-down encoding, as referenced above. Figure 7 As mentioned above.

[0133] Method 800 may include, in operation 808, determining a label tree generation loss for generating the latent label tree. As described above, this label tree generation loss is negatively correlated with the probability that the set of labels collected from the latent label tree based on predefined collection rules is the same as the labeled label set of the sentence. For example, the label tree generation loss may be the negative logarithm of this probability. Preferably, the probability that the set of labels collected based on predefined collection rules is the same as the labeled label set of the sentence is equal to the sum of the probabilities of all possible subspaces in the latent label tree where the set of labels collected based on predefined rules is the same as the labeled label set.

[0134] The label tree generation loss refers to the loss associated with the label tree of the generated sentence. Preferably, this label tree generation loss is based on the sum of probabilities of the labels in the label tree. It should be understood that the label tree considered is the latent label tree of the same label set as that collected based on the collection rules (e.g., implemented via the yield function) and the standard label set.

[0135] For specific details regarding the label tree generation loss and total loss, please refer to the description above.

[0136] Method 800 may include, in operation 810, training the second machine learning model using the training set to minimize the label tree generation loss. Any method known to those skilled in the art can be used to minimize this loss to train the model.

[0137] In a preferred embodiment, instead of determining the loss of the second machine learning model separately, the total loss of the sentence is determined, which may be equal to the weighted sum of the parse tree generation loss for generating the parse tree and the label tree generation loss for generating the label tree. For example, in this case, the first machine learning model may be untrained. Accordingly, the first and second machine learning models can be trained simultaneously using a training set to minimize the total loss.

[0138] The parse tree generation loss refers to the loss associated with generating the parse tree of the sentence. For example, when using the Fast-R2D2 model to generate a parse tree, the parse tree generation loss is... and For its specific meaning and calculation method, please refer to the Fast-R2D2 paper. When using other models, the parse tree generation loss is the loss function of the corresponding model.

[0139] In this case, by minimizing the total loss, the machine learning model is trained to minimize both the parse tree generation loss and the label tree generation loss, thereby achieving a model that optimizes both the tree structure and the distribution of labels among the nodes of the tree, thus improving the accuracy of the generated label tree.

[0140] Preferably, after the machine learning model is generated, a test set can be used to test the machine learning model.

[0141] See Figure 9 It shows a schematic flowchart of an example method 900 for performing sentence processing according to an embodiment of this specification.

[0142] The method may include: in operation 902, acquiring the sentence to be processed. The sentence can be acquired from any data source, such as various computer programs, mobile applications, mini-programs, web pages, etc. The sentence can be acquired and processed in real time, or a stored sentence can be acquired. The sentence can be an unannotated sentence, i.e., it may not have a set of annotated tags.

[0143] The method may include, at operation 904, using a trained first machine learning model to predict a parse tree of the sentence, the parse tree having a binary tree structure. The first machine learning model may be, for example, any language structuring model described above (e.g., the Fast-R2D2 model), which may be trained alone or simultaneously with a second machine learning model.

[0144] Method 900 may include, in operation 906, using a trained second machine learning model to predict a label tree for the sentence. This label tree has the same structure as the parse tree, wherein predicting the label tree for the sentence includes predicting the label of each node in the label tree. The label tree can be obtained knowing the label of each node in the label tree and the structure of the label tree (which is the same as the structure of the parse tree).

[0145] Preferably, the second machine learning model can be trained using the method described above. Having been described in detail above, those skilled in the art will understand how to use the trained machine learning model to process inputs.

[0146] Optionally, the method may include: outputting the generated tag tree to the user. Figure 9 (Not shown in the image). For example, a visual representation of the generated tag tree can be displayed to the user. This visual representation could be similar to... Figure 1 The example shown.

[0147] Optionally, the method may include: in operation 908, using predefined collection rules to collect tags from the tag tree to generate a tag set for the sentence. The predefined collection rules are those described above. For example, the `yield` function can be used to collect tags from the tag tree, as described in detail above.

[0148] As can be imagined, unlike the training phase, in the prediction phase, it is not necessary to know the label set of the sentence. Instead, a label tree of the sentence can be generated first, and then the label set of the sentence can be collected based on the label tree.

[0149] The embodiments of this specification may also relate to machine learning models trained in the manner described above (e.g., trained individually or simultaneously), which can be used to process sentences to generate a label tree (and optionally a set of labels) for the sentences. Details of this machine learning model have been described in detail above and will not be repeated here.

[0150] See Figure 10 The diagram shows a schematic block diagram of an example system 1000 for training a machine learning model according to an embodiment of this specification.

[0151] like Figure 10 As shown, the system 1000 may include a training set acquisition module 1002, which can be used to acquire a training set, which includes multiple training samples, each training sample including a sentence and a set of labeled tags for that sentence.

[0152] System 1000 may also include a parse tree generation module 1004, which can be used to generate a parse tree of the sentence using a first machine learning model, the parse tree having a binary tree structure.

[0153] System 1000 may further include a label tree generation module 1006, which can be used to generate a latent label tree for the sentence using a second machine learning model. This latent label tree has the same structure as the parse tree. As described above, the nodes in the label tree include non-terminating nodes, terminating label nodes, and terminating empty label nodes. The label of the terminating label node is the task label. The predefined collection rule is: the task label of a node is collected only if the node in the label tree has a task label and all its ancestor nodes are non-terminating nodes. For example, the label can be collected using the yield function.

[0154] System 1000 may also include a training module 1008, which can be used to determine the label tree generation loss for generating the label tree, wherein the label tree generation loss is negatively correlated with the probability that the set of labels collected from the potential label tree based on predefined collection rules is the same as the labeled set of the sentence, and to train the second machine learning model using the training set to minimize the label tree generation loss.

[0155] It needs to be understood, as mentioned above. Figure 8-10 The description omits some details that can be found in the preceding text (e.g., references). Figure 1-7 (found in the description).

[0156] Figure 11A schematic block diagram of an apparatus 1100 for implementing a system (such as system 1000 above) according to one or more embodiments of this specification is shown. The apparatus may include a processor 1110 and a memory 1115, the processor being configured to perform operations using any of the methods described above. The memory may store, for example, acquired data, algorithms used, and intermediate data generated during operation, etc.

[0157] The device 1100 may include a network connectivity element 1125, such as a network connectivity device that can connect to other devices via a wired or wireless connection. The wireless connection may be, for example, a WiFi connection, a Bluetooth connection, or a 3G / 4G / 5G network connection. It may also receive user input from other devices or transmit data to other devices for display via the network connectivity element.

[0158] The device may also optionally include other peripheral components 1120, such as input devices (e.g., keyboard, mouse), output devices (e.g., monitor), etc. It can also output relevant information to the user via the output devices.

[0159] Each of these modules can communicate with each other directly or indirectly, for example, via one or more buses (e.g., bus 1105).

[0160] Furthermore, this application also discloses a computer-readable storage medium including computer-executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods of the embodiments described herein.

[0161] In addition, this application also discloses an apparatus including a processor and a memory storing computer-executable instructions that, when executed by the processor, cause the processor to perform the methods of the embodiments described herein.

[0162] In addition, this application also discloses a system including means for implementing the methods of the various embodiments described herein.

[0163] It is understood that the methods according to one or more embodiments of this specification can be implemented in software, firmware, or a combination thereof.

[0164] It should be understood that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the apparatus and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. Unless there is a conflict, the features described in one or more embodiments can be implemented in other embodiments.

[0165] It should be understood that the foregoing description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0166] It should be understood that the use of a singular form to describe an element or to show only one element in the accompanying drawings does not imply that the number of such element is limited to one. Furthermore, modules or elements described or shown as separate herein may be combined into a single module or element, and modules or elements described or shown as single herein may be broken down into multiple modules or elements.

[0167] Throughout this instruction manual, "nearly", "almost", and "roughly" mean a deviation of no more than 10%.

[0168] It should also be understood that the terminology and expressions used herein are for descriptive purposes only, and one or more embodiments described herein should not be limited to these terms and expressions. The use of these terms and expressions does not exclude any illustrative and descriptive equivalent features (or parts thereof), and it should be recognized that various modifications that may exist should also be included within the scope of the claims. Other modifications, variations, and substitutions may also exist. Accordingly, the claims should be considered to cover all such equivalents.

[0169] Similarly, it should be noted that although specific embodiments have been described with reference to the present invention, those skilled in the art should recognize that the above embodiments are merely illustrative of one or more embodiments of this specification, and various equivalent changes or substitutions can be made without departing from the spirit of the invention. Therefore, any changes or modifications to the above embodiments within the scope of the essential spirit of the invention will fall within the scope of the claims of this application.

Claims

1. A method for training a machine learning model, the method comprising: Obtain a training set, which includes multiple training samples. Each training sample includes a sentence and a set of labeled tags for the sentence. The sentence is in text form. The first machine learning model is used to generate a parse tree of the sentence, the parse tree having a binary tree structure; A second machine learning model is used to generate a latent label tree for the sentence, the latent label tree having the same structure as the parse tree, wherein generating the latent label tree for the sentence includes predicting the label of each node in the latent label tree, wherein the label is associated with a node in the parse tree to form the latent label tree; Determine the label tree generation loss for generating the potential label tree, wherein the label tree generation loss is negatively correlated with the probability that the set of labels collected from the potential label tree based on predefined collection rules is the same as the labeled set of the sentence; as well as The second machine learning model is trained using the training set to minimize the label tree generation loss.

2. The method of claim 1, wherein the probability that the tag set collected based on predefined collection rules is the same as the labeled tag set of the sentence is equal to the sum of the probabilities of all possible subspaces in the latent tag tree that the tag set collected based on predefined rules is the same as the labeled tag set.

3. The method of claim 1, wherein the first machine learning model is untrained, and the method further comprises: Determine the total loss of the sentence, which is equal to the weighted sum of the parse tree generation loss for generating the parse tree and the label tree generation loss for generating the latent label tree; as well as Using the training set, the first machine learning model and the second machine learning model are trained simultaneously to minimize the total loss.

4. The method of claim 1, wherein the Fast-R2D2 model is used to generate the parse tree of the sentence.

5. The method of claim 1, wherein the nodes in the potential label tree include non-terminating nodes and terminating label nodes, wherein the label of the terminating label node is a task label.

6. The method of claim 5, wherein the potential label tree further includes terminating empty label nodes.

7. The method of claim 6, wherein the predefined collection rule is: the task label of a node is collected only when a node in the potential label tree has a task label and all of the node's ancestor nodes are non-terminating nodes.

8. The method of claim 6, wherein dynamic programming is used to traverse the parse tree.

9. The method of claim 8, wherein using dynamic programming to traverse the parse tree includes stopping traversal of the child nodes of the terminal node when a terminal node is encountered.

10. The method of claim 8, wherein using dynamic programming to traverse the parse tree comprises: The probability is calculated assuming that two non-overlapping spans of the parse tree are not associated with the same task label.

11. The method of claim 1, wherein the representation of each span of the parse tree is converted from bottom-up encoding to top-down encoding using a Transformer model.

12. A method for performing processing on a sentence, the method comprising: Obtain the sentence to be processed; The first machine learning model, which has been trained, is used to predict the parse tree of the sentence, which has a binary tree structure. as well as A trained second machine learning model is used to predict the label tree of the sentence, the label tree having the same structure as the parse tree, wherein predicting the label tree of the sentence includes predicting the label of each node in the label tree, wherein the second machine learning model is trained according to the method of any one of claims 1-11.

13. The method of claim 12, further comprising: Tags from the tag tree are collected using predefined collection rules to generate a tag set for the sentence.

14. A system for training a machine learning model, comprising: The training set acquisition module is used to acquire a training set, which includes multiple training samples. Each training sample includes a sentence and a set of labeled tags for the sentence. The sentence is in text form. A parse tree generation module is used to generate a parse tree of the sentence using a first machine learning model, the parse tree having a binary tree structure; A label tree generation module is used to generate a potential label tree for the sentence using a second machine learning model. The potential label tree has the same structure as the parse tree. Generating the potential label tree for the sentence includes predicting the label of each node in the potential label tree, wherein the label is associated with a node in the parse tree to form the potential label tree. as well as A training module is configured to determine a label tree generation loss for generating the label tree, wherein the label tree generation loss is negatively correlated with the probability that the set of labels collected from the potential label tree based on predefined collection rules is the same as the labeled set of the sentence, and to train a second machine learning model using the training set to minimize the label tree generation loss.

15. The system of claim 14, wherein the nodes in the tag tree include non-terminating nodes, terminating tag nodes, and terminating empty tag nodes, wherein the tag of the terminating tag node is a task tag, and wherein the predefined collection rule is: the task tag of a node is collected only when a node in the tag tree has a task tag and all of the node's ancestor nodes are non-terminating nodes.

16. An apparatus for training a machine learning model, comprising: Memory; as well as A processor configured to perform the method as described in any one of claims 1-13.

17. A computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method as claimed in any one of claims 1-13.

Images