Training method of multi-field multi-intention spoken language semantic understanding model

Abstract

Embodiments of the present application provide a kind of multi-domain multi-intention spoken language semantic understanding model training method.The method comprises: inputting the ontology item data with annotation to spoken language semantic understanding model;Based on the ontology coding module in spoken language semantic understanding model, the semantic coding of dialogue text is carried out, and initial ontology feature and initial vector are obtained;Based on the graph encoder of bidirectional relationship graph attention network, graph neural network coding is carried out to initial ontology feature and initial vector, and semantic coding and structure coding are obtained;The probability distribution of spoken language semantic understanding result is obtained by inputting semantic coding and structure coding to the decoder in spoken language semantic understanding model.The embodiments of the present application start from multi-intention spoken language semantic understanding task, and expand spoken language semantic understanding task to multi-domain setting and hierarchical semantic framework output.A multi-intention data set originated from real scene in industry is constructed, and the multi-domain multi-intention spoken language semantic understanding task is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent speech, and more particularly to a training method for a multi-domain, multi-intent spoken semantic understanding model and the multi-domain, multi-intent spoken semantic understanding model itself. Background Technology

[0002] Existing Spoken Language Understanding (SLU) is typically decoupled into two subtasks: intent detection and slot filling. These subtasks are trained jointly using classic models of sentence classification and sequence labeling, respectively. However, these SLU studies often focus on a single-intent setting, where each input dialogue contains only one user intent. This configuration severely limits the expressiveness of user dialogue and the semantic capabilities of the output.

[0003] To identify multiple intents from the user's spoken semantic understanding, the following approach is typically used:

[0004] 1. Intent combination enumeration: Enumerate different combinations of intents and treat each combination as a single intent;

[0005] 2. Input Clause Segmentation: First, the complete user input is split into clauses, assuming that each clause contains at most one intent. Then, each clause is parsed separately according to the traditional single-intent task.

[0006] 3. Multi-label sentence classification: Replace the multi-class sentence classifier (which selects only one class) in the single intent case with a multi-label sentence classifier to determine whether each intent label exists in the user input.

[0007] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0008] For intent combination enumeration, due to the limitations of a single domain, it is necessary to enumerate all possible intent combinations. When the number of single intents increases, it is easy to cause the number of combined intents to explode exponentially. Furthermore, it is impossible to generalize from some combinations of intents to other combinations, and there is a lack of multi-intent benchmark datasets that can be used for model training.

[0009] This assumption regarding input clause segmentation is too strong and may contain unreasonable presuppositions regarding sentence segmentation. In real-world scenarios, users' arbitrary statements may not be able to be perfectly segmented into clauses.

[0010] For multi-label sentence classification, the oversimplification of the multi-intent problem results in a flattened semantic framework output by the model, which cannot determine which intent a slot pair belongs to. For example, in the sentence "Help me turn on Bluetooth and turn off the air conditioner," the slot pairs "operation = turn on" and "object = Bluetooth" belong to the intent "media control," while the slot pairs "operation = turn off" and "object = air conditioner" belong to the intent "body control." Therefore, this approach cannot handle the hierarchical relationship between "slots" and "intents." This leads to relatively low accuracy of the aforementioned existing technologies in multi-domain, multi-intent spoken semantic understanding. Summary of the Invention

[0011] In order to at least address the problem of relatively low accuracy in multi-domain, multi-intent spoken semantic understanding in existing technologies.

[0012] In a first aspect, embodiments of the present invention provide a training method for a multi-domain, multi-intent spoken semantic understanding model, comprising:

[0013] Annotated ontology data is input into the spoken semantic understanding model, wherein the annotated ontology data includes: dialogue text with a structured semantic framework of labeled domain-intent-slot pairs;

[0014] Based on the ontology encoding module within the spoken semantic understanding model, the dialogue text is semantically encoded to obtain the initial ontology features of the dialogue text and the initial vectors of each word in the dialogue text.

[0015] Based on the graph encoder based on the bidirectional relation graph attention network in the spoken semantic understanding model, the initial ontology features and the initial vector are encoded by a graph neural network to obtain the semantic encoding of the dialogue text and the structural encoding used to represent the structural hierarchical relationship between words.

[0016] The semantic encoding and structural encoding are input into the decoder within the spoken language semantic understanding model to obtain the probability distribution of the spoken language semantic understanding results;

[0017] If the probability distribution of the spoken language semantic understanding result does not reach the preset training target, then the probability distribution is used as the input of the decoder at the next time step to train the spoken language semantic understanding model until the probability distribution reaches the preset training target.

[0018] Secondly, embodiments of the present invention provide a multi-domain, multi-intent spoken semantic understanding model, including:

[0019] The ontology encoding module is used to perform semantic encoding on the input dialogue text to obtain the initial ontology features of the dialogue text and the initial vectors of each word in the dialogue text.

[0020] A graph encoder based on a bidirectional relational graph attention network is used to encode the initial ontology features and the initial vectors using a graph neural network to obtain the semantic encoding of the dialogue text and the structural encoding for representing the hierarchical relationship between words.

[0021] A pointer-generator decoder based on a bidirectional relational graph attention network is used to decode the semantic and structural encodings to obtain the spoken semantic understanding results of the domain-intent-slot pairs predicted from the dialogue text.

[0022] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of a training method for a multi-domain, multi-intent spoken semantic understanding model according to any embodiment of the present invention.

[0023] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of a training method for a multi-domain, multi-intent spoken semantic understanding model according to any embodiment of the present invention.

[0024] The beneficial effects of this invention are as follows: This method starts from the multi-intent spoken semantic understanding task and extends the spoken semantic understanding task to multi-domain settings and hierarchical semantic framework output. It not only constructs a multi-intent dataset (ontology item data) derived from real-world industrial scenarios, but also proposes the BiRGAT model to effectively solve the problems that traditional single-intent settings cannot address with sequence labeling and sentence classification, enabling it to achieve relatively superior results in multi-domain, multi-intent spoken semantic understanding tasks. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a training method for a multi-domain, multi-intent spoken semantic understanding model provided in an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of single intent, multiple intents, and multiple domains in the MIVS dataset, which is a training method for a multi-domain, multi-intent spoken semantic understanding model provided in an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the BiRGAT model structure of a training method for a multi-domain, multi-intent-based spoken semantic understanding model provided in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram illustrating the relationships between words in a training method for a multi-domain, multi-intent-based spoken semantic understanding model provided by an embodiment of the present invention.

[0030] Figure 5 This is a schematic diagram showing the test results of a training method for a multi-domain, multi-intent spoken semantic understanding model provided by an embodiment of the present invention on the MVIS and TOPv2 datasets;

[0031] Figure 6 This is a schematic diagram of the BiRGAT encoder ablation study of a training method for a multi-domain, multi-intent spoken semantic understanding model provided in an embodiment of the present invention.

[0032] Figure 7 This is a schematic diagram of the intent training results in the field of vehicle control, based on a training method for a multi-domain, multi-intent spoken semantic understanding model provided by an embodiment of the present invention.

[0033] Figure 8 This is a schematic diagram of a training experiment of a multi-domain, multi-intent spoken semantic understanding model provided by an embodiment of the present invention in the map + weather domain;

[0034] Figure 9 This is a schematic diagram of the structure of a multi-domain, multi-intent spoken semantic understanding model provided in an embodiment of the present invention;

[0035] Figure 10 This is a schematic diagram of an electronic device embodiment of a training method for a multi-domain, multi-intent spoken semantic understanding model provided by an embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] like Figure 1 The diagram shows a flowchart of a training method for a multi-domain, multi-intent spoken semantic understanding model according to an embodiment of the present invention, including the following steps:

[0038] S11: Input the labeled ontology item data into the spoken semantic understanding model, wherein the labeled ontology item data includes: dialogue text with a structured semantic framework labeled with domain-intent-slot pairs;

[0039] S12: Based on the ontology encoding module within the spoken semantic understanding model, perform semantic encoding on the dialogue text to obtain the initial ontology features of the dialogue text and the initial vectors of each word in the dialogue text;

[0040] S13: Based on the graph encoder based on the bidirectional relation graph attention network in the spoken semantic understanding model, the initial ontology features and the initial vector are encoded by a graph neural network to obtain the semantic encoding of the dialogue text and the structural encoding used to represent the structural hierarchical relationship between words.

[0041] S14: Input the semantic encoding and structural encoding into the decoder in the spoken language semantic understanding model to obtain the probability distribution of the spoken language semantic understanding result;

[0042] S15: If the probability distribution of the spoken language semantic understanding result does not reach the preset training target, then the probability distribution is used as the input of the decoder at the next time step to train the spoken language semantic understanding model until the probability distribution reaches the preset training target.

[0043] In this embodiment, the method finds that existing technologies lack corresponding real-world multi-intent, or even multi-domain, datasets suitable for use in spoken semantic understanding models for experimentation. Simply put, this method serializes a structured semantic framework into a sequence of symbols or words, and then treats multi-intent spoken semantic understanding as a text generation task. Under a single-intent setting, in addition to semantic understanding, the model also needs to learn the underlying triplet format, i.e., the output format specification. For example, the semantic framework is serialized as "[body control (operation = open)(object = Bluetooth)]". However, under a multi-intent setting, how to serialize the output and ensure that the model's output conforms to the target's specified format requirements (constrained decoding) are two key issues.

[0044] For step S11, the spoken semantic understanding model is trained using the prepared labeled ontology item data for training the spoken semantic understanding model.

[0045] As one implementation method, the labeled ontology item data is collected and obtained in real-world scenarios, including:

[0046] Collect user dialogue data in a single domain with a single intent, as well as dialogue data in a single domain with multiple intents. The single domain includes: vehicle control, map, music, weather, and phone call.

[0047] Select multiple single-domain single-intent dialogue data or single-domain multi-intent dialogue data from different domains to construct multi-domain multi-intent dialogue data for the user.

[0048] In this embodiment, the method constructs a multi-intent dataset (MIVS) for training the spoken semantic understanding model. MIVS is a multi-intent dataset collected from a realistic in-vehicle dialogue system (e.g., an in-vehicle dialogue system).

[0049] The Multi-Intent MIVS dataset comprises five distinct domains: maps, weather, phone calls, vehicle control, and music. The entire dataset can be divided into two parts: single-domain and multi-domain. The single-domain portion contains single-intent and multi-intent dialogue examples, manually labeled and collected from real-world industrial vehicle environments.

[0050] For cross-domain samples (i.e., multi-domain, multi-intent dialogue data), two utterances can be extracted from two different domains (e.g., "Navigate to Chengdu" from the "Map" domain and "Play Chengdu" from the "Music" domain), and then connected using conjunctions such as "and" (to obtain "Navigate to Chengdu and then play Chengdu"). This results in multi-domain, multi-intent dialogue data composed of such imperative dialogues. However, due to the arbitrary diversity of user input, the dialogue input by the user may also be a question dialogue, such as "How is the weather today?" from the "Weather" domain and "Navigate to Changting Road" from the "Navigation" domain. By combining these, we get "How is the weather today? By the way, navigate to Changting Road."

[0051] Furthermore, the sequence can be serialized into a labeled sequence to prepare for model training by inserting sentinel markers (such as brackets for clustering). The serialized format is the output y of the model in this method, which will be used in subsequent steps. Figure 2 Examples of labeled domain-intent-slot value structured semantic frameworks for single-domain and multi-domain datasets are provided. To ensure the privacy of personal information, privacy-preserving data such as names in the multi-intent dataset are replaced.

[0052] For step S12, the spoken semantic understanding model of this method can be called BiRGAT (bidirectional Relational Graph Attention Network). The model is divided into three parts, including: ontology encoding module, BiRGAT graph encoder and BiRGAT symbol decoder.

[0053] Regarding the encoder of the spoken language semantic understanding model, this method introduces ontology items, including semantic and structural encodings of domain, intent, and slots. Given ontology items from a specific domain d (e.g., music). Ontology item embeddings can be initialized from SWV (static word vectors) or BERT (Bidirectional Encoder Representation from Transformers) PLM (pretrained language model). (m is the dimension) is obtained, where |O d | represents the number of ontology items in domain d. The corresponding structure is as follows: Figure 3 The ontology encoding module is shown.

[0054] As one implementation method, the semantic encoding of the dialogue text based on the ontology encoding module within the spoken language semantic understanding model includes:

[0055] Insert corresponding category features into each word in the dialogue text to avoid overfitting and arrangement bias during model training. The category features include: domain, intent, and slot.

[0056] A sequence of dialogue texts is generated by using the domain names, all intent names, and all slot names under the same domain conditions in the dialogue text.

[0057] The dialogue text sequence is input into an autoencoder pre-trained model to obtain a word vector sequence with domain context information;

[0058] The word vector sequence is input into a bidirectional long short-term memory network to output the initial ontology features of the dialogue text;

[0059] Each word in the dialogue text is input into the autoencoder pre-trained model or the bidirectional long short-term memory network to obtain the initial vector of each word.

[0060] In this embodiment, it is assumed that From several words Composed of (l being the length), their vectors are passed to a single-layer BiLSTM (bidirectional long short-term memory network). To inform the BiLSTM of type information, in its name ( {DOMAIN,INTENT,SLOT})({domain, intent, slot}) preceded by the type The forward and backward hidden states from the BiLSTM are concatenated to form the initial embedding.

[0061] Before PLM, all ontology items in a domain d are first flattened into a sequence, following the order of domain (only one) → intent → slot. Type information is also inserted before each ontology item. ([DOMAIN] / [INTENT] / [SLOT]). To avoid overfitting permutation bias, the internal order of intent and slot is randomly adjusted in each training iteration. This is done when extracting each ontology item. After the PLM outputs the hidden state, they are also fed into a type-aware BiLSTM to obtain...

[0062] Embed all initial bodies Superposition yields the feature matrix. Note that in the case of multiple domains, or if the target domain is uncertain at input, a further step is taken to stack the matrix O from all possible domains d. d And obtain the entire matrix. Where O=∪ d O d Otherwise, O d The feature O is directly used as the input feature of the next encoder. (0) .

[0063] The input question dialogue Q (e.g., "How's the weather today?") is given by Q = (q1, q2, ..., q...). |Q| ), where q is a question word of length |Q|, and the initial features are... It can also be initialized from SWV or PLM. This leads to the initialized ontology features. i ∈O, and the initial vector q for each word. i ∈Q.

[0064] For step S13, regarding the BiRGAT graph encoder, as follows: Figure 3 The middle part, obtained through improvements to the Transformer encoder architecture, is where the module obtains the initial matrix Q of the question words and ontology items. (0) and O (0) Then, structural knowledge and transverse segment information are used to further enrich the features. The BiRGAT encoder consists of L layers, and the calculation of the l-th layer is as follows:

[0065] Q (l+1) O (l+1) =BiRGAT(Q (l) O (l) )

[0066] In this layer, 0 ≤ l ≤ L-1, each layer consists of three sub-modules: (1) Dual Multi-head Self-Attention, (2) Dual Multi-head CrossAttention, and (3) Feedforward Network. Each sub-module is also wrapped with residual connections and Layer Norm functions.

[0067] Multi-head self-attention mechanisms are invoked between question words and ontology items for computation, and different types of hierarchical relationships of edge feature structures are introduced through relative position embedding. ij As one implementation, the structural hierarchy relationships between words include: subordinate relationships in word structure and sequential relationships in word order. In this implementation, for example, there is a "domain-intent" subordinate relationship between two words, or a "word-before-after" relationship. Essentially, it is a variant of the RGAT (Relational Graph Attention Network) model, where the attention weights e ij and attention vector The calculation method is as follows: (based on the ontology item o) i For example, N(i) represents the set of neighboring nodes of node i.

[0068]

[0069]

[0070] Among them, a jj It is e ij The softmax version.

[0071] This method constructs a complete graph among the words in the question and uses the relative distance between words as the relation z. ij .like Figure 4 All relation types are provided. To avoid over-parameterization, the marginal feature z... ij Shared between different layers and headers.

[0072] Then, a bidirectional multi-head cross-attention module is invoked to calculate the weighted features of all ontology items for each question word and the features of the current question for each ontology item. This module is analogous to the cross-attention module in the Transformer decoder. Finally, a feedforward neural network is invoked for further encoding of both the question words and ontology items. The output Q after stacking L layers of RGAT is... (L) and O (L)The final encoded state (semantic encoding and structural encoding used to represent the hierarchical relationships between words) (Q and O) is saved for use by the decoder.

[0073] For step S14, the Q and O determined by the encoder in step S13 are input into the BiRGAT decoder. The decoder in the spoken semantic understanding model includes a pointer-generator decoder based on a bidirectional relation graph attention network. In this method, it can be a three-gate pointer-generator decoder.

[0074] As one implementation, the probability distribution of inputting the semantic encoding and structural encoding into the decoder within the spoken language semantic understanding model to obtain the spoken language semantic understanding result includes:

[0075] The semantic and structural encodings are converted into string sequences, and dynamically decoded at each time step using an autoregressive text generation method to obtain the probability distribution of the spoken semantic understanding results.

[0076] In this embodiment, taking BiRGAT's three-gate pointer-generator decoder as an example, the Q and O of the encoder are input, first serialized into a string sequence using sentinel characters, and the output token sequence y = (y1, y2, ..., y3) is then used. |y| This is generated autoregressively using a single-layer Transformer decoder. The hidden state of the decoder at time step t is:

[0077] s t =TransformerDecoder(y ＜t [Q;O])

[0078] Using the serialized string as the output target, this module is an improvement on the Transformer decoder. It utilizes a pointer-generator network structure at the output, and the decoding space for each time step can be selected from three aspects:

[0079] (1) Generated directly from the vocabulary, such as sentinel characters “{”, “[”, “(” and some unaligned slot values ​​(unaligned means that the slot value does not appear in the question, so it cannot be sequence-tagged or copied);

[0080] (2) Copy a word from the question, mainly to handle the aligned slot values;

[0081] (3) Select one item directly from the encoded ontology item memory O.

[0082] Assume the hidden state of the decoder at time t is s t , with g t As the chosen gate function (containing three dimensions), φ(w) i ) represents the word wi The word vector function outputs y t The probability distribution is calculated as follows:

[0083]

[0084] P gen (w i ) = softmax i (s t W gen φ(w i ) 1 ),

[0085]

[0086] P select (o i ) = PtrNet(s t ,O)[i],

[0087] P(y t |s t (Q, O) = g t1 P gen +g t2 P copy +g t3 P select ,

[0088] Wherein, φ(w) i Return w i In the word embedding of a fixed vocabulary, PtrNet(st, Q)[k] represents the probability of selecting the k-th entry (row) in memory Q, which is implemented as the average weight of different heads of the multi-head cross-attention module. Then, through training at the current time step, the probability distribution of spoken semantic understanding results is obtained, achieving sequence labeling and sentence classification.

[0089] For step S15, training typically involves a pre-set training objective, which can be determined empirically. If the probability distribution of yt does not reach the pre-set training objective at this point, training proceeds to the next time step. Figure 3 As can be seen from the right part, in the next training round, in addition to the Q and O inputs of the encoder, the probability distributions from the previous rounds are also received for training. The decoder's input features y at the next time step... t It can be defined as:

[0090]

[0091] The training objectives are decoupled as follows:

[0092]

[0093] The training continues until the probability distribution reaches the preset training target, thus obtaining a multi-domain, multi-intent spoken semantic understanding model.

[0094] As can be seen from this implementation, this method starts with the multi-intent spoken semantic understanding task and extends it to multi-domain settings and hierarchical semantic framework output. It not only constructs a multi-intent dataset (ontology item data) derived from real-world industrial scenarios, but also proposes the BiRGAT model to effectively solve the problems that traditional single-intent settings cannot address with sequence labeling and sentence classification, enabling it to achieve relatively superior results in multi-domain, multi-intent spoken semantic understanding tasks.

[0095] This paper provides a detailed experimental description of the method. Regarding datasets, the method was tested on two multi-intent SLU (Spoken Language Understanding) datasets: MIVS (mentioned earlier) and the English TOPv2 dataset. Because the original output format of TOPv2 differs from the MIVS annotation used in this method, the output labels of TOPv2 need to be converted to the three-layer hierarchical structure of this method.

[0096] The hidden dimensions *m* of SWV and PLM were 256 and 512, respectively. The BiRGAT encoder had 2 layers. For the pointer generator decoder, the number of layers was fixed at 1. The number of heads and the dropout rate were set to 8 and 0.2, respectively. Throughout the experiments, the AdamW optimizer and a linear warm-up scheduler were used. The warm-up ratio for the total training steps was 0.1. The learning rate and weight decay rate for SWV were 5e-4 / 1e-4, and for PLM, they were 2e-4 / 0.1. The PLM parameters were further optimized with a layer-wise decay rate (coefficient 0.8). The batch size was 20, and the number of training iterations was 100k. For inference, a beam search of size 5 was used.

[0097] The baseline methods compared with this method are:

[0098] SL: Slot-filled, a classic sequence labeling method. Since the output is a three-level tree structure, the label "B-[slot_name]" needs to be expanded to "B-[domain_name][intent_name]-[slot_name]" in order to recover the output structure from the list of labeled tuples.

[0099] SL+CLF: Based on the SL method, it first accumulates frequent four-tuples "(domain, intent, slot, value)" that are misaligned in the input question and appear at least 5 times throughout the training set. These four-tuples are then used as utterance-level labels, and a multi-label classifier is added to solve the misaligned slot-value problem.

[0100] BART: A fine-tuned end-to-end BART model without introducing any task-specific parameters. Each ontology item is also generated token-by-token.

[0101] BART / Copy: To ensure good formatting of ontology items, the original BART model was extended using an ontology copying scheme. Specifically, the original BART vocabulary was expanded by all ontology items.

[0102] LLM+ICL: Large Language Model (LLM) and Context Learning (ICL).

[0103] In the experiments, all data samples were merged, including single-domain and multi-domain samples, and ontology items were provided as input from all domains. That is, the model also needs to determine the specific domain of the current utterance. The performance of different models varies depending on the initialization and decoding methods, as shown below. Figure 5 As shown. We can observe:

[0104] (1) Replacing the static word vectors SWV with the more powerful autoencoder PLM can steadily improve performance. For example, the accuracy of the test set improved from 85.6 to 89.3 on MIVS and from 85.9 to 87.8 on TOPv2.

[0105] (2) Sequence generation is better suited for handling hierarchical semantic frameworks compared to traditional SL and SL+CLF methods. Even with multi-label classification (+CLF) to address slot misalignment, complex output structures cannot be recovered from flat sequences or quadruple lists. This inherent limitation results in poor performance on both datasets.

[0106] (3) The ontology copying mechanism is crucial for ensuring consistency between ontology item names and predefined names. This conclusion was verified not only by the BiRGAT decoder, a pointer generator customized in this method, but also by the end-to-end BART model. Notably, on the MIVS test set using BART, the performance gap between w / o Copy and w / Copy exceeds 35 points (26.8% vs. 63.3%).

[0107] (4) The model using only SWV in this method can already compete with the generative PLM BART, and even outperform it on MIVS. This phenomenon can be explained by two factors. First, the internal structure of the output sequence 4 is different from the pre-training corpus of the BART model, which is mainly free-form natural language. This difference in pre-training / fine-tuning is particularly evident on the Chinese dataset MIVS, where the accuracy on the test set based purely on Chinese is only 26.8%. Another potential reason is that directly fine-tuning the BART model lacks the structural and semantic encoding of ontology items. To further analyze the fine-grained effects of each component in the BiRGAT encoder, this method further conducts an ablation study on the encoder.

[0108] This method attempts to identify the contribution of each component in the BiRGAT encoder, including whether an ontology encoding module (w / OE) is used and whether relational features z are included. ij Incorporate structural encoding (GAT vs. RGAT), and whether to insert a dual-headed cross-attention submodule into the GNN layer (w / DCA). According to... Figure 6 We can conclude that:

[0109] (1) Utilizing textual descriptions (w / OE) of ontology items can effectively enrich the semantics of ontology embeddings. This observation is more evident on English datasets. For example, on the TOPv2 test set, SWV's performance improved from 82.6% to 85.3%.

[0110] (2) Both structure encoding (GAT) and relational encoding (RGAT) can improve the performance on the test set. Compared with English TOPv2, the MIVS dataset proposed in this method seems to benefit more from the hierarchical structure of ontology items. This can be attributed to the fact that the data samples in MIVS contain relatively more intents and exhibit a more complex output tree shape.

[0111] (3) Although separate encoding of question words and ontology items has yielded significant results, the integration of cross-segment attention (w / DCA) can still bring stable performance improvements on both datasets.

[0112] (4) Furthermore, the above conclusions remain consistent regardless of the initialization method used (SWV or PLM). In other words, the benefits of the structured BiRGAT encoder and the contextual PLM BERT are compatible and complementary.

[0113] (5) The performance gain of the structured BiRGAT encoder is relatively small compared to the improvements brought by the three-gate pointer generator decoder. This is attributed to the large-scale training data, which makes it easier to explore the underlying patterns of the output sequence and the hierarchical structure of the ontology items.

[0114] For experiments involving multiple intentions, train the model on data samples with 3 or fewer intentions, but evaluate the model on samples containing more intentions (>3). The model can be further fine-tuned using a smaller number of samples containing >3 intentions. Re-split the data and conduct experiments in the "in-vehicle control" domain, as it contains more intentions on average. Figure 7 The performance of different models was demonstrated.

[0115] For experiments in different domains, the model was trained on discourses including domains A, B, C, A+B, and B+C, and evaluated on samples involving domain A+C. Let A = "map", B = "music", and C = "weather". The experimental results are as follows: Figure 8 As shown in the figure, the performance improvement of this method can be seen from the change curve.

[0116] In summary, this method proposes a large-scale, multi-domain, multi-intent dataset, MIVS, collected from a real-world in-vehicle dialogue system. The target semantic framework is organized into a three-layer tree. Therefore, this method further designs a BiRGAT model, integrating the hierarchical structure of ontology items into the graph encoder and introducing a three-way copy mechanism in the decoder. Experiments on the MIVS and TOPv2 datasets demonstrate the advantages of our BiRGAT model over various existing baseline methods.

[0117] like Figure 9 The diagram shown is a structural schematic of a multi-domain, multi-intent spoken semantic understanding model provided in an embodiment of the present invention. The system is trained by the training method of the multi-domain, multi-intent spoken semantic understanding model described in any of the above embodiments and configured in a terminal.

[0118] This embodiment provides a multi-domain, multi-intent spoken semantic understanding model 10, which includes: an ontology encoding module 11, a graph encoder 12 based on a bidirectional relation graph attention network, and a pointer-generator decoder 13 based on a bidirectional relation graph attention network.

[0119] Specifically, the ontology encoding module 11 performs semantic encoding on the input dialogue text to obtain initial ontology features of the dialogue text and initial vectors for each word in the dialogue text; the graph encoder 12 based on a bidirectional relational graph attention network performs graph neural network encoding on the initial ontology features and the initial vectors to obtain semantic encoding of the dialogue text and structural encoding for representing the hierarchical relationship between words; and the pointer-generator decoder 13 based on a bidirectional relational graph attention network decodes the semantic encoding and structural encoding to obtain the spoken semantic understanding result of the predicted domain-intent-slot pairs of the dialogue text.

[0120] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the training method of the multi-domain, multi-intent spoken semantic understanding model in any of the above method embodiments.

[0121] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0122] Annotated ontology data is input into the spoken semantic understanding model, wherein the annotated ontology data includes: dialogue text with a structured semantic framework of labeled domain-intent-slot pairs;

[0123] Based on the ontology encoding module within the spoken semantic understanding model, the dialogue text is semantically encoded to obtain the initial ontology features of the dialogue text and the initial vectors of each word in the dialogue text.

[0124] Based on the graph encoder based on the bidirectional relation graph attention network in the spoken semantic understanding model, the initial ontology features and the initial vector are encoded by a graph neural network to obtain the semantic encoding of the dialogue text and the structural encoding used to represent the structural hierarchical relationship between words.

[0125] The semantic encoding and structural encoding are input into the decoder within the spoken language semantic understanding model to obtain the probability distribution of the spoken language semantic understanding results;

[0126] If the probability distribution of the spoken language semantic understanding result does not reach the preset training target, then the probability distribution is used as the input of the decoder at the next time step to train the spoken language semantic understanding model until the probability distribution reaches the preset training target.

[0127] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they execute the training method and the multi-domain multi-intent spoken semantic understanding model of any of the above method embodiments.

[0128] Figure 10 This is a schematic diagram of the hardware structure of an electronic device for training a multi-domain, multi-intent spoken semantic understanding model according to another embodiment of this application, as shown below. Figure 10 As shown, the device includes:

[0129] One or more processors 1010 and memory 1020, Figure 10 Taking a processor 1010 as an example, the device for training a multi-domain, multi-intent spoken semantic understanding model may further include an input device 1030 and an output device 1040.

[0130] The processor 1010, memory 1020, input device 1030, and output device 1040 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0131] The memory 1020, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the training method of the multi-domain, multi-intent spoken semantic understanding model in the embodiments of this application. The processor 1010 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1020, thereby implementing the training method of the multi-domain, multi-intent spoken semantic understanding model in the above-described method embodiments.

[0132] The memory 1020 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1020 may optionally include memory remotely located relative to the processor 1010, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0133] Input device 1030 can receive input numerical or character information. Output device 1040 may include display devices such as a display screen.

[0134] The one or more modules are stored in the memory 1020. When executed by the one or more processors 1010, they execute the training method of the multi-domain, multi-intent spoken semantic understanding model in any of the above method embodiments.

[0135] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0136] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0137] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the training method for a multi-domain, multi-intent spoken semantic understanding model according to any embodiment of this invention.

[0138] The electronic devices described in this application exist in various forms, including but not limited to:

[0139] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0140] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0141] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0142] (4) Other electronic devices with data processing functions.

[0143] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a multi-domain, multi-intent-based spoken semantic understanding model, comprising: Annotated ontology data is input into the spoken semantic understanding model, wherein the annotated ontology data includes: dialogue text with a structured semantic framework of labeled domain-intent-slot pairs; Based on the ontology encoding module within the spoken semantic understanding model, the dialogue text is semantically encoded to obtain the initial ontology features of the dialogue text and the initial vectors of each word in the dialogue text. Based on the graph encoder based on the bidirectional relation graph attention network in the spoken semantic understanding model, the initial ontology features and the initial vector are encoded by a graph neural network to obtain the semantic encoding of the dialogue text and the structural encoding used to represent the structural hierarchical relationship between words. The semantic encoding and structural encoding are input into the decoder within the spoken language semantic understanding model to obtain the probability distribution of the spoken language semantic understanding results; If the probability distribution of the spoken language semantic understanding result does not reach the preset training target, then the probability distribution is used as the input of the decoder at the next time step to train the spoken language semantic understanding model until the probability distribution reaches the preset training target.

2. The method according to claim 1, wherein, The semantic encoding of the dialogue text based on the ontology encoding module within the spoken language semantic understanding model includes: A sequence of dialogue texts is generated by using the domain names, all intent names, and all slot names under the same domain conditions in the dialogue text. The dialogue text sequence is input into an autoencoder pre-trained model to obtain a word vector sequence with domain context information; The word vector sequence is input into a bidirectional long short-term memory network to output the initial ontology features of the dialogue text; Each word in the dialogue text is input into the autoencoder pre-trained model or the bidirectional long short-term memory network to obtain the initial vector of each word.

3. The method according to claim 2, wherein, Before generating the dialogue text sequence using domain names, all intent names, and all slot names under the same domain condition in the dialogue text, the method further includes: Insert corresponding category features into each word in the dialogue text to avoid overfitting and arrangement bias during model training. The category features include: domain, intent, and slot.

4. The method according to claim 1, wherein, The structural relationships between the words include: the subordinate relationship of word structure and the sequential relationship of word order.

5. The method according to claim 1, wherein, The decoder within the spoken language semantic understanding model includes a pointer-generative decoder based on a bidirectional relation graph attention network; The probability distribution of inputting the semantic encoding and structural encoding into the decoder within the spoken language semantic understanding model to obtain the spoken language semantic understanding result includes: The semantic and structural encodings are converted into string sequences, and dynamically decoded at each time step using an autoregressive text generation method to obtain the probability distribution of the spoken semantic understanding results.

6. The method according to claim 5, wherein, The conversion methods for the string sequence include: sentinel character serialization.

7. The method according to claim 1, wherein, The labeled ontology item data was collected in real-world scenarios and includes: Collect user dialogue data in a single domain with a single intent, as well as dialogue data in a single domain with multiple intents. The single domain includes: vehicle control, map, music, weather, and phone call. Select multiple single-domain single-intent dialogue data or single-domain multi-intent dialogue data from different domains to construct multi-domain multi-intent dialogue data for the user.

8. A multi-domain, multi-intent spoken language semantic understanding system, comprising: The ontology encoding module is used to perform semantic encoding on the input dialogue text to obtain the initial ontology features of the dialogue text and the initial vectors of each word in the dialogue text. A graph encoder based on a bidirectional relational graph attention network is used to encode the initial ontology features and the initial vectors using a graph neural network to obtain the semantic encoding of the dialogue text and the structural encoding for representing the hierarchical relationship between words. A pointer-generator decoder based on a bidirectional relational graph attention network is used to decode the semantic and structural encodings to obtain the spoken semantic understanding results of the domain-intent-slot pairs predicted from the dialogue text.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

Images