Online spoken language understanding system based on bert-bjnn-emgcrf model

Abstract

The application discloses an online oral language understanding system based on a BERT-BJNN-EMGCRF model, comprising the following steps: S1, constructing a model framework composed of a shared network, a bidirectional joint network and a conditional random field of an end-to-end shielding graph; S2, training the model in S1, updating the parameters of the entire network model, and obtaining a semantic slot filling and intention detection model; S3, building an online system composed of a user end and a server end; S4, importing the semantic slot filling and intention detection model in S2 into the online system based on S3 to form an online oral language understanding system. The application greatly utilizes the associated information between the intention and the semantic slot, sufficiently learns the compatibility implicit connection of the label pairs in the semantic slot and between the intention and the semantic slot, improves the overall performance of the two tasks, and realizes the extraction of the semantic slot and the detection of the intention in the detected dialogue text sentence in an online mode.

Description

Technical Field

[0001] This invention relates to the field of spoken language understanding in dialogue systems, and in particular to an online spoken language understanding system based on the BERT-BJNN-EMGCRF model. Specifically, it is a joint model for semantic slot filling and intent detection based on BERT-BJNN-EMGCRF, and an online semantic slot extraction and intent detection system with this model as the core. It can be applied to fields such as online smart ordering and personal voice assistants. Background Technology

[0002] Intent detection and slot filling are crucial components of spoken language understanding (SLU) in dialogue systems; therefore, their performance significantly impacts the overall performance of the dialogue system. The rapid development of technologies such as Artificial Intelligence (AI), Deep Learning (DL), and Big Data has provided technical support for intent detection and slot filling. To improve the accuracy of slot filling and intent detection, the most widely used approach is a joint method, which aims to propagate the errors of intent and slots simultaneously to train the network. However, existing joint models still face many challenges. For example, insufficient fusion of information between the corresponding hidden layer slots and intents in the outputs of the two tasks leads to decreased accuracy for both tasks. Furthermore, mismatches in label pairs within slots and between intents and slots also contribute to accuracy degradation.

[0003] Furthermore, with the increasing demand for convenient human-computer interaction, people hope to be able to execute their requests by conversing with machines in simple spoken language. Traditional systems require users to manually select certain constraints or consult staff online to achieve their goals such as searching or booking tickets, including semantic slot values ​​such as query and booking intent, time, departure point, and destination. This approach cannot intelligently understand the user's rough intent and the specific details of semantic slot values ​​through simple spoken dialogue, thus reducing user operations and alleviating staff pressure. Summary of the Invention

[0004] To address the shortcomings and limitations of existing technologies, this invention provides an online spoken language comprehension system based on the BERT-BJNN-EMGCRF model.

[0005] An online spoken language understanding system based on the BERT-BJNN-EMGCRF model includes the following steps: S1, constructing a model framework consisting of a shared network, a bidirectional joint network, and a conditional random field with an end-to-end masking graph, wherein the shared network is composed of "BERT", the bidirectional joint network is composed of "BJNN", and the conditional random field with the end-to-end masking graph is composed of "EMGCRF"; S2, training the model described in S1, calculating the loss based on the error between the two prediction vectors and the true values ​​obtained from the output of the model described in S1, and updating the parameters of the entire network model to obtain the semantic slot filling and intent detection model; S3, building an online system consisting of a user terminal and a server terminal; S4, importing the semantic slot filling and intent detection model described in S2 into the online system based on S3 to form an online spoken language understanding system, wherein the server of the system processes the dialogue text sentences to be detected submitted by the user from the web page and asynchronously returns the results to the web page, realizing the extraction of semantic slots and the detection of intent in the dialogue text sentences to be detected online.

[0006] Optionally, the shared network adds an intent classification word to the beginning of the given dialogue text sentence data and inputs it into the BERT model to obtain the contextual semantic embedding feature vector of the word. Before the text sentence is input into the word embedding model, the word '[CLS]' is added to the beginning of the sentence as the input word for intent classification, thereby unifying the two tasks of semantic slot filling and intent detection into a sequence labeling task. Then, the sentence is segmented to obtain the word index X′ after segmentation, and then embedding is performed to obtain the word embedding feature vector sequence E′. Finally, the final word semantic feature sequence E is output through a selection operation.

[0007] X′=Token(X)

[0008] E′=BERT(X′)

[0009] E = select(E′)

[0010] Where X = {x1, x2, ..., x} N} represents the sequence of words in the input sentence with '[CLS]' added, and N represents the maximum number of words in the input sentence. For some sentences with fewer than N words, '[PAD]' will be added to the end of the sentence for padding. Here, e1 is the encoding of the '[CLS]' token, which is trained to classify the entire sequence. `select` replaces the semantic feature vector corresponding to the input word '[PAD]' with a zero vector, thus preventing the filler word from being considered as a message. Furthermore, due to the atomistic, fine-grained nature of the dictionary words in the BERT model, using it for semantic feature extraction of sentence words can handle complex input words like x. iDuring word segmentation, the word is broken down into multiple basic words and embedded to obtain a rich semantic expression. Therefore, in the Token stage, select will sum the fine-grained word vectors to obtain the final semantic feature vector e of the word. i .

[0011] Furthermore, the bidirectional joint network comprises two parts: intent2slot for semantic slot filling and slot2intent for intent classification;

[0012] Let D i and D s These represent the number of different intent labels and semantic slot labels, respectively. To achieve bidirectional jointing of semantic slots and intent probability distributions, e1 is first passed through a network consisting of W... ID The constructed fully connected feedforward neural network layer, and then the final semantic feature vector H = {e2,…,e2} excluding the word '[CLS]'. N} By W SD The constructed fully connected feedforward neural network layer is finally normalized by applying a softmax function to the semantic slots and intent probability distribution.

[0013] P ID =softmax(e1×W) ID +b ID )

[0014] P SD =softmax(H×W) SD +b SD )

[0015] Where the parameter matrix bias matrix It is the probability distribution of intent. It is the semantic slot probability distribution;

[0016] intent2Slot extracts semantic information from e1 to plot the intent probability distribution, and then plots P... ID The algorithm is copied N-1 times and concatenated with the semantic slot probability distribution, ensuring that the prediction of each semantic slot label incorporates information from the intent probability distribution. Then, a fully connected feedforward neural network is used to output the semantic slot prediction.

[0017] P Ir = repeat(P ID ),(N-1)times

[0018] P S =[P SD ;P Ir ]×W S +bS

[0019] in Parameter matrix The text represents the predicted semantic slot labels, and [;] represents matrix concatenation.

[0020] slot2intent describes the probability distribution of semantic slots using semantic feature vectors H. First, it uses P... SD For P SD For each word's semantic slot probability distribution, a weight is calculated. This weight determines the proportion of each semantic slot probability in the supplementary information. A semantic slot probability distribution with a weak connection to the intent receives a smaller weight. Then, the weighted probability distributions of each semantic slot are summed to obtain the supplementary information from the semantic slots and concatenated with the intent probability distribution. This allows the semantic slot probability distribution P to be used to generate the supplementary information. SD This is used to detect intent. Finally, a fully connected feedforward neural network is used to calculate the predicted intent.

[0021] a = sigmoid(tanh(P) SD ×W a )×V a )

[0022] P I =[P ID ;(a T ×P SD )]×W I +b I

[0023] The parameter matrix for calculating weight a is... Predictive Intent

[0024] The core of the conditional random field of the end-to-end masking graph is a connection matrix W composed of semantic slots and intent nodes. crf Its function is to correct the discrepancy between the intention and semantic slots predicted by the bidirectional joint network and the actual labels caused by compatibility issues.

[0025]

[0026] This indicates that a masking mechanism, i.e., a negative infinity -inf mask, is applied to node i; i∈[1,N] where 1 to N-1 are semantic slot nodes, and N is an intent node. Since intent nodes only have intent class labels, a masking mechanism is needed to mask the semantic slot correction parameters when predicting the corrected intent, thus retaining only D. i The intent correction parameter for length modifies the intent prediction. Similarly, semantic slot nodes are masked.

[0027]

[0028] Among them una i Given the original predicted probability of the bidirectional joint network for the i-th node, according to the masking mechanism, una i The original predicted probabilities need to be filled with zero vectors;

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] Let represent the predicted label of the i-th node in the t-th iteration, where i∈[1,N],t∈[0,T]. It is the label predicted in the t-th iteration of the intent node. This represents the final predicted label of the i-th node, i.e., the predicted label of the intent or semantic slot. When i∈[1,N-1], This represents the information about the predicted label distribution from the i-th semantic slot node and intent node. When i = N, it represents the label distribution information of all nodes. It is a transformation matrix that reflects the implicit connection of the tag for compatibility. By transforming matrix W crf Calculate the correction vector Then let the correction vector and the original predicted probability una i Add them together to obtain the corrected prediction probability Then, the predicted label for the (t+1)th iteration is obtained using a masking mechanism and normalization operations. arrive The calculation process iterates T times to obtain the final intent and semantic slot prediction labels.

[0035] Optionally, the overall network loss during model training consists of semantic slot and intent losses, treating each intent or semantic slot as a distinct category. Label smoothing recognition loss is used to supervise the training of various parameters in the shared network, bidirectional joint network, and end-to-end masking graph conditional random field. Their losses are defined as follows:

[0036]

[0037]

[0038]

[0039]

[0040] L=δL I +(1-δ)L S

[0041] Where q i The true label representing intent i, p i It is a predicted intent label; This represents the predicted label for the output semantic slot s corresponding to the j-th word. It is a real semantic slot tag; L I and L S δ represents the intent loss and semantic slot loss, respectively, and δ∈[0,1] adjusts the degree of attention this method pays to these two losses;

[0042] The current loss is fed back into the network model, and the Adam optimizer is used to iteratively optimize the model by adjusting the parameters to reduce the loss and improve the accuracy of target prediction. The trained model is then tested on the test dataset to obtain the detection accuracy of the current model. If the preset accuracy is not achieved, the hyperparameters or optimizer in the network are reset until a model that meets the preset accuracy requirements is obtained. This model is then saved as the final semantic slot filling and intent detection model.

[0043] Step S3 specifically includes: S301, the user interface consists of a system homepage, a semantic slot extraction interface, and an intent detection interface. The system homepage includes two parts: head and body. The head section first sets the encoding format and compatible browsers using meta tags. Then, when setting the webpage title using the title tag, it embeds the Flask Jinja syntax template block title block and sets the title to 'home' within it. The purpose of this block is to allow for content replacement when switching between different webpages, while the content outside the syntax block remains unchanged through inheritance. Then, the link and script tags are used to import the Bootstrap and jQuery libraries from external sources. Finally, a reserved head block provides an interface for adding settings to other webpages. The body section uses HTML controls to divide the entire webpage into four areas: header, side navigation bar, main content, and footer, corresponding to the header, aside, main, and footer controls, respectively. The header contains `` controls that hyperlink to the current page. The `<aside>` section contains hyperlinks to the 'System Homepage', 'Semantic Slot Extraction', and 'Intent Detection'. The `main` section contains a `blockbody` block with embedded `` and `` controls to display the title and text. After arranging the controls, the size, position, color, and dimensions of the controls are statically rendered using the imported Bootstrap library, and JavaScript is used for dynamic rendering.<h1>

[0044] The semantic slot extraction interface inherits the layout of the system homepage using the extend syntax, modifying only the content of the blocks within the system homepage. This interface changes the content of the block title to 'slot' to replace the webpage title, and adds a script control within the block head to import the slot.js file to enable communication between this webpage and the backend. The content in the block body is replaced with the title and text corresponding to the semantic slot extraction, and also includes a form control and a div control to display the extraction results. A form with the network response parameter method set to GET or POST includes input and button controls to implement dialogue text input and submission. The intent detection interface can be obtained using the same method.

[0045] S302. On the server side, a local server object for the app is created using the Flask framework. Routes are created using the `route` function of the decorator view function. The URL is set to ' / ' as the root route, registered, and bound to the system homepage view. The system homepage is loaded using the `render_template` function, which returns the system homepage HTML file as a parameter. The semantic slot extraction interface and intent detection interface are configured in the same way. Additionally, an event handling route ' / spoke_process' is set to handle interaction requests within the semantic slot extraction and intent detection interfaces, with the `methods` parameter set to GET or POST to match the frontend's request method. Finally, the `run` function is used to start the local server, with the port number set to 2023 and the host number set to 0.0.0.0, allowing devices on the same network segment (i.e., the same local area network) as the local host to access the local server.

[0046] S303, the intermediate communication mechanism uses asynchronous Ajax technology. This method finds the corresponding control in the .js file using `#` followed by the control's ID and uses an Ajax function to achieve communication between the front-end and back-end. First, it retrieves the submit button control and sets a click listener event. When a click is detected, the click event function executes. Within the function, it retrieves the input control using its ID and uses the `val` function to retrieve the content entered in the input control. Next, in the Ajax function, the request type parameter `type` is set to 'get', and the data is passed in dictionary form with text data format. Furthermore, in the `slot.js` file, the Ajax URL parameter is set to " / spoke_process?intent_slot=slot" to find the event handling route and implement semantic slot extraction. In `intent.js`, it's set to " / spoke_process?intent_slot=intent" to implement intent extraction. Finally, after a successful request, the result is received from the server. The display control is found using its ID, and the result is displayed within the control using an HTML function.

[0047] In the server's event handling routing function, upon receiving a GET request from the frontend, the `args` method of the `request` class retrieves the value of the `intent_slot` variable from the AJAX URL parameters. It then determines whether this is intent detection or semantic slot extraction. Finally, the `values` method of the `request` class obtains the text from the input fields in the data dictionary. The function also reserves an interface for processing the text and returns the processed result to the corresponding interface via `return`.

[0048] The beneficial effects of this invention are as follows: Addressing the issues of insufficient fusion of hidden layer semantic slots and intent information in the joint intent detection and semantic slot filling method for spoken language understanding, leading to decreased accuracy due to mismatches between label pairs within semantic slots and between intent and semantic slots, as well as insufficient model application, this invention proposes an online spoken language understanding system based on the BERT-BJNN-EMGCRF model. Its main advantages are: 1) BJNN provides associated semantic information and supplementary semantic information for semantic slot and intent classification respectively through intent probability distribution and semantic slot probability distribution, enabling full fusion of hidden layer semantic slot and intent information. 2) The two-dimensional matrix connection graph composed of semantic slot and intent nodes in EMGCRF learns the implicit compatibility connections between label pairs within semantic slots and between intent and semantic slots, thereby correcting the deviation between the intent and semantic slots predicted by BJNN and the true labels. 3) Embedding the joint intent detection and semantic slot filling model into an online system demonstrates the model's practicality and provides a reference for realizing complex online spoken language understanding systems in the field. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention.

[0050] Figure 1 This is a schematic diagram of the overall process of the online spoken language comprehension system based on the BERT-BJNN-EMGCRF model provided in this invention.

[0051] Figure 2 This is a schematic diagram of the BERT-BJNN-EMGCRF model structure provided in this invention.

[0052] Figure 3 This is a schematic diagram of the functional structure of the shared network module provided in this invention.

[0053] Figure 4 This is a schematic diagram of the EMGCRF module structure provided in this invention.

[0054] Figure 5 This is a schematic diagram of the homepage of the online spoken language comprehension system provided in this invention.

[0055] Figure 6 This is a schematic diagram of semantic slot extraction in the online spoken language understanding system provided by this invention;

[0056] Figure 7 This is a schematic diagram of the intent understanding system of the online spoken language understanding system provided in this invention; Detailed Implementation

[0057] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings, so that the above and other objects, features, and advantages of the present invention will become clearer. In all the drawings, the same reference numerals indicate the same parts. The drawings are not intentionally drawn to scale; the focus is on illustrating the main points of the invention.

[0058] In summary, this invention proposes a model consisting of BERT, Bi-directional Joint Neural Networks (BJNN), and an End-to-end masked graph-based Conditional Random Field (EMGCRF), and an online spoken language understanding system based on this model. The main features of this system are: 1) Building and training a model composed of BERT, BJNN, and EMGCRF, with the trained model serving as the core processing module for semantic slot extraction and intent detection in the online system; 2) Adding the marker '[CLS]' to the beginning of the dialogue text sentence sequence to form the input sequence (x1, x2, ..., x...). N ) and word embedding is performed. During word segmentation, the BERT semantic model divides certain coarse-grained or synthetic words into fine-grained basic words and performs word embedding. Then, the fine-grained word embedding vectors of the word are summed to obtain the final word embedding vector of the word; 3) In the bidirectional joint BJNN, the intent probability distribution and semantic slot probability distribution provide associated semantic information and supplementary semantic information for semantic slot and intent classification, respectively; 4) The EMGCRF module is introduced after BJNN. Its core is a connection graph composed of semantic slots and intent nodes, represented by a two-dimensional matrix. This matrix corrects the deviation between the intent and semantic slot predicted by the bidirectional joint network and the real labels by learning the compatibility implicit connection of label pairs within semantic slots and between intent and semantic slots; 5) The online system is built using Flask, Bootstrap, jQuery library, CSS, JavaScript and other technologies and the PyCharm editor.

[0059] The flowchart of the online spoken language comprehension system based on the BERT-BJNN-EMGCRF model is as follows: Figure 1 As shown, it includes the following steps:

[0060] S1. Building and Training the Model: A model framework consisting of a shared network, a bidirectional joint network, and a conditional random field with an end-to-end shielding graph, such as... Figure 2 As shown, the shared network is composed of "BERT", the bidirectional joint network is composed of "BJNN", and the conditional random field of the end-to-end shielded graph is composed of "EMGCRF".

[0061] S2. Train the model, calculate the loss based on the error between the two predicted vectors and the true value obtained from the model output, and update the parameters of the entire network model to obtain the semantic slot filling and intent detection model.

[0062] S3. Build an online system consisting of a client and a server.

[0063] S4. Import the model: Import the trained semantic slot filling and intent detection model into the established online system to form an online spoken language understanding system. The system's server processes the dialogue text sentences to be detected submitted by the user from the web page and returns the results asynchronously to the web page, realizing the extraction of semantic slots and the detection of intent in the dialogue text sentences to be detected online.

[0064] The above model consists of three parts: a shared network, a bidirectional joint network, and a conditional random field for an end-to-end shielded graph;

[0065] 1) The shared network model selection includes a case-insensitive dictionary, an embedding layer, 12 encoder layers, and a final hidden layer dimension of D. e =768 and 12 attention heads pre-trained BERT model "bert_uncased_L-12_H-768_A-12". Given dialogue text sentences, intent classification words are added to the beginning of the sentence before being input into the shared network model to obtain the contextual semantic embedding feature vectors of the words. Before the text sentence is input into the shared network model, the word '[CLS]' is added to the beginning of the sentence as the input word for intent classification, thus unifying the semantic slot filling and intent detection tasks into a single sequence labeling task. Next, the sentence is segmented to obtain the word index X′, then embedded to obtain the word embedding feature vector sequence E′, and finally, a selection operation is performed to output the final word semantic feature sequence E.

[0066] X′=Token(X) (1)

[0067] E′=BERT(X′) (2)

[0068] E = select(E′) (3)

[0069] Where X = {x1, x2, ..., x} N} represents the sequence of words in the input sentence with '[CLS]' added, and N represents the maximum number of words in the input sentence. For some sentences with fewer than N words, '[PAD]' will be added to the end of the sentence for padding. Here, e1 is the encoding of the '[CLS]' token, which is trained to classify the entire sequence. `select` replaces the semantic feature vector corresponding to the input word '[PAD]' with a zero vector, thus preventing the filler word from being considered as a message. Furthermore, due to the atomistic, fine-grained nature of the dictionary words in the BERT model, using it for semantic feature extraction of sentence words can handle complex input words like x. i During word segmentation, the word is broken down into multiple basic words and embedded to obtain a rich semantic expression. Therefore, in the Token stage, select will sum the fine-grained word vectors to obtain the final semantic feature vector e of the word. i The process is as follows: Figure 3 As shown.

[0070] 2) The bidirectional joint network, built upon the shared network, comprises two parts: intent2slot for semantic slot filling and slot2intent for intent classification. intent2slot integrates the intent probability distribution with the semantic slot probability distribution when labeling each semantic slot, providing semantic information about the association between intent and semantic slots for slot filling. slot2intent uses the semantic slot label probability distribution as supplementary information for intent classification.

[0071] Let Di and Ds be the number of different intent labels and semantic slot labels, respectively. To achieve bidirectional joint semantic slots and intent probability distributions, e1 is first passed through a fully connected feedforward neural network layer constructed using WID, and then the final semantic feature vector H = {e2,…,e...} excluding the word '[CLS]' is generated. N} By W SD The constructed fully connected feedforward neural network layer is finally normalized by applying a softmax function to the semantic slots and intent probability distribution.

[0072] P ID =softmax(e1×W) ID +b ID (4)

[0073] P SD =softmax(H×W) SD +b SD (5)

[0074] Where the parameter matrix bias matrix It is the probability distribution of intent. It is the semantic slot probability distribution;

[0075] intent2Slot extracts semantic information from e1 to plot the intent probability distribution and uses it to help detect the semantic slot label for each word. To utilize the intent probability distribution P... ID To fill the semantic slots, P ID The algorithm is copied N-1 times and concatenated with the semantic slot probability distribution, ensuring that the prediction of each semantic slot label incorporates information from the intent probability distribution. Then, a fully connected feedforward neural network is used to output the semantic slot prediction.

[0076] P Ir = repeat(P ID ),(N-1)times (6)

[0077] P S =[P SD ;P Ir ]×W S +b S (7)

[0078] in Parameter matrix The text represents the predicted semantic slot labels, and [;] represents matrix concatenation.

[0079] slot2intent uses a semantic feature vector H to depict the probability distribution of semantic slots and utilizes it to provide supplementary information about semantic slots during intent detection. First, it uses P... SD For P SD For each word's semantic slot probability distribution, a weight is calculated. This weight determines the proportion of each semantic slot probability in the supplementary information. A semantic slot probability distribution with a weak connection to the intent receives a smaller weight. Then, the weighted probability distributions of each semantic slot are summed to obtain the supplementary information from the semantic slots and concatenated with the intent probability distribution. This allows the semantic slot probability distribution P to be used to generate the supplementary information. SD This is used to detect intent. Finally, a fully connected feedforward neural network is used to calculate the predicted intent.

[0080] a = sigmoid(tanh(P) SD ×W a )×V a (8)

[0081] P I =[P ID ;(a T ×P SD )]×W I +b I (9)

[0082] The parameter matrix for calculating weight a is... Predictive Intent

[0083] 3) The end-to-end shielded graph conditional random field follows the bidirectional joint network. Its core is a connection graph composed of semantic slots and intent nodes, which is represented by the connection matrix W. crf ,like Figure 4 As shown. This matrix corrects the discrepancy between the predicted intents and semantic slots and the actual labels caused by compatibility issues by learning the implicit connectivity of label pairs within semantic slots and between intents and semantic slots;

[0084]

[0085] This indicates that a masking mechanism, i.e., a negative infinity -inf mask, is applied to node i; i∈[1,N] where 1 to N-1 are semantic slot nodes, and N is an intent node. Since intent nodes only have intent class labels, a masking mechanism is needed to mask the semantic slot correction parameters when predicting the corrected intent, thus retaining only D. i The intent correction parameter for length modifies the intent prediction. Similarly, semantic slot nodes are masked.

[0086]

[0087] Among them una i Given the original predicted probability of the bidirectional joint network for the i-th node, according to the masking mechanism, una i The original predicted probabilities need to be filled with zero vectors;

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] Let represent the predicted label of the i-th node in the t-th iteration, where i∈[1,N],t∈[0,T]. It is the label predicted in the t-th iteration of the intent node. This represents the final predicted label of the i-th node, i.e., the predicted label of the intent or semantic slot. When i∈[1,N-1], This represents the information about the predicted label distribution from the i-th semantic slot node and intent node. When i = N, it represents the label distribution information of all nodes. It is a transformation matrix that reflects the implicit connection of the tag for compatibility. By transforming matrix W crf Calculate the correction vector Then let the correction vector and the original predicted probability una i Add them together to obtain the corrected prediction probability Then, the predicted label for the (t+1)th iteration is obtained using a masking mechanism and normalization operations. arrive The calculation process iterates T times to obtain the final intent and semantic slot prediction labels.

[0094] The model training steps are as follows:

[0095] The aforementioned model was trained and tested on the PyTorch framework using a text training and testing dataset from a specific domain-specific dialogue system. The overall network loss consists of semantic slot loss and intent loss. Following previous work, each intent or semantic slot is treated as a distinct category. Label smoothing recognition loss is used to supervise the training of individual parameters in the shared network, bidirectional joint network, and end-to-end masking graph conditional random field. Their losses are defined as follows:

[0096]

[0097]

[0098]

[0099]

[0100] L=δL I +(1-δ)L S (twenty one)

[0101] Where q i The true label representing intent i, p i It is a predicted intent label; This represents the predicted label for the output semantic slot s corresponding to the j-th word. It is a real semantic slot tag; L I and L S δ represents the intent loss and semantic slot loss, respectively, and δ∈[0,1] adjusts the degree of attention this method pays to these two losses.

[0102] Finally, the obtained loss is fed back into the network model, and the Adam optimizer is used to iteratively optimize the model by adjusting the parameters to reduce the loss and improve the accuracy of target prediction. The trained model is then tested on the test dataset to obtain the detection accuracy of the current model. If the preset accuracy is not achieved, the hyperparameters or optimizer in the network are reset until a model that meets the preset accuracy requirements is obtained. This model is then saved as the final semantic slot filling and intent detection model.

[0103] The aforementioned online system consists of three parts: the client, the server, and the intermediate communication mechanism;

[0104] 1) The user interface consists of three parts: the system homepage, the semantic slot extraction interface, and the intent detection interface;

[0105] like Figure 5 As shown, the system homepage consists of two parts: head and body. The head section first sets the encoding format and compatible browsers using meta tags. Then, it uses the title tag to set the page title, embedding a Flask Jinja syntax template block (`block title`) within the tag and setting the title to 'home'. This block allows for content replacement between different pages, while content outside the syntax block remains unchanged through inheritance. Next, the `link` and `script` tags import the Bootstrap and jQuery libraries from external sources. Finally, a `blockhead` is reserved to provide an interface for adding settings to other pages. The body section uses HTML controls to divide the entire page into four areas: header, sidebar, main content, and footer, corresponding to the header, aside, main, and footer controls, respectively. The header contains `` controls that hyperlink to the current page. The aside section contains hyperlinks to the 'System Homepage' page, the 'Semantic Slot Extraction' page, and the 'Intent Detection' page. The main section contains a `blockbody` block, embedding `h1` and `p` controls to display the content title and text. After the controls are arranged, the size, position, color and size of the controls are statically rendered using the imported Bootstrap library, and the controls are dynamically rendered using JavaScript.

[0106] like Figure 6As shown, the semantic slot extraction interface inherits the layout of the system homepage using the extend syntax, modifying only the content of the blocks within the system homepage. This interface modifies the content of the block title to 'slot' to replace the webpage title, and adds a script control within the block head to import the slot.js file to enable communication between this webpage and the backend. The content in the block body is replaced with the title and text corresponding to the semantic slot extraction, and also includes a form control and a div control to display the extraction results. A form with the network response parameter method set to GET or POST includes input and button controls to implement dialogue text input and submission. The intent detection interface can be obtained using the same method, such as... Figure 7 As shown.

[0107] 2) On the server side, a local server object is created using the lightweight, customizable web application framework Flask. Routes are created using the `route` method of decorator view functions, and the URL is set to ' / ' as the root route. The system homepage view is registered and bound to this route. The system homepage is loaded using the `render_template` function, which returns the system homepage HTML file as a parameter. The semantic slot extraction and intent detection interfaces are configured in the same way. Additionally, an event handling route ` / spoke_process` is set to handle interaction requests within the semantic slot extraction and intent detection interfaces, with the `methods` parameter set to `get` or `post` to match the frontend's request method. Finally, the `run` function is used to start the local server, with the port number set to 2023 and the host number set to 0.0.0.0, allowing devices on the same network segment (i.e., the same local area network) as the local host to access the local server.

[0108] 3) The intermediate communication mechanism uses asynchronous Ajax technology. This method uses the `#` followed by the control's ID in the .js file to find the corresponding control and then uses an Ajax function to achieve communication between the front-end and back-end. First, the submit button control is retrieved and a click listener is set for it. When a click is detected, the click event function is executed. Within the function, the input control is retrieved using its ID, and the content entered in the input control is retrieved using the `val` function. Next, the request type parameter `type` is set to 'get' in the Ajax function, and the data is passed in dictionary form with text data format. Furthermore, in the `slot.js` file, the Ajax URL parameter is set to " / spoke_process?intent_slot=slot" to find the event handling route and implement semantic slot extraction. In `intent.js`, it is set to " / spoke_process?intent_slot=intent" to implement intent extraction. Finally, after a successful request, the result is received from the server. The display control is found using its ID, and the result is displayed within the control using an HTML function.

[0109] In the server's event handling routing function, upon receiving a GET request from the frontend, the `args` method of the `request` class retrieves the value of the `intent_slot` variable from the AJAX URL parameters. It then determines whether this is intent detection or semantic slot extraction. Finally, the `values` method of the `request` class obtains the text from the input fields in the data dictionary. The function also reserves an interface for processing the text and returns the processed result to the corresponding interface via `return`.

[0110] The trained network model is imported into an online system to form an online spoken language understanding system, enabling online intent detection and semantic slot extraction from dialogue text. First, the model is imported from the model building file in the backend and a model object is created. Then, the saved model parameters are loaded into the model object to complete the import of the trained model. Next, within the interface provided by the event handling routing function, a preprocessing function is used to convert the obtained text into a list of word indices required by the model. This list is then input into the model to complete intent detection and semantic slot extraction. Finally, the results are processed into the required text format and returned to the interface.

[0111] While the technology has been described and illustrated with respect to one or more embodiments, changes and / or modifications may be made to the illustrated examples without departing from the spirit and scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components or structures (components, devices, circuits, systems, etc.), the terminology used to describe such components (including references to “apparatus”) is intended to correspond to any component or structure performing the specified function of the described component (e.g., functionally equivalent), even if structurally not equivalent to the disclosed structure performing the function of the illustrated embodiments described herein, unless otherwise specified. Furthermore, while a particular feature may have been disclosed with respect to one of several embodiments, such feature may be combined with one or more other features in other embodiments as may be desired and advantageous for any given or particular application. Moreover, with regard to the use of the terms “comprising,” “including,” “having,” “containing,” “comprising,” or variations thereof in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “comprising.”

[0112] Many specific details have been set forth in the foregoing description to provide a thorough understanding of the present invention. However, the above description is merely a preferred embodiment of the present invention, and the present invention can be implemented in many other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed above. Furthermore, any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention, or modify them into equivalent embodiments, using the methods and techniques disclosed above, without departing from the scope of the present invention. Any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention, without departing from the content of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. An online spoken language comprehension system based on the BERT-BJNN-EMGCRF model, characterized in that, Includes the following steps: S1. Construct a model framework consisting of a shared network, a bidirectional joint network, and a conditional random field with an end-to-end shielded graph. The shared network is composed of "BERT", the bidirectional joint network is composed of "BJNN", and the conditional random field with an end-to-end shielded graph is composed of "EMGCRF". S2. Train the model described in S1, calculate the loss using the error between the two predicted vectors and the true value obtained from the output of the model described in S1, and update the parameters of the entire network model to obtain the semantic slot filling and intent detection model. S3. Build an online system consisting of a client and a server; S4. The semantic slot filling and intent detection model described in S2 is imported into the online system described in S3 to form an online spoken language understanding system. The system's server processes the dialogue text sentences to be detected submitted by the user from the web page and returns the results asynchronously to the web page, thereby realizing the extraction of semantic slots and the detection of intent in the dialogue text sentences to be detected online.

2. The online spoken language comprehension system based on the BERT-BJNN-EMGCRF model according to claim 1, characterized in that, The shared network takes a given dialogue text sentence, adds an intent classification word at the beginning of the sentence, and inputs it into the BERT model to obtain the contextual semantic embedding feature vector of the word. Before the text sentence is input into the word embedding model, the word '[CLS]' is added to the beginning of the sentence as the input word for intent classification, thus unifying the two tasks of semantic slot filling and intent detection into a single sequence labeling task. Then, the sentence is segmented to obtain the word index X after segmentation. ′ Then, embedding is performed to obtain the word embedding feature vector sequence E. ′ Finally, the semantic feature sequence E of the final word is output through a selection operation; X′=Token(X) E′=BERT(X′) E = select(E′) Where X = {x1, x2, ..., x} N } represents the sequence of words in the input sentence with '[CLS]' appended, N represents the maximum number of words in the input sentence, and for sentences with fewer than N, '[PAD]' will be appended to the end of the sentence for padding. E = {e1, e2, ..., e N }; Here, e1 is the encoding of the '[CLS]' token, which is trained to classify the entire sequence. `select` replaces the semantic feature vector corresponding to the input word '[PAD]' with a zero vector, thus preventing the filler word from being used for information transmission. Furthermore, `select` sums the fine-grained word vectors during the tokenization stage to obtain the final semantic feature vector e of the word. i .

3. The online spoken language comprehension system based on the BERT-BJNN-EMGCRF model according to claim 2, characterized in that, The bidirectional joint network consists of two parts: intent2slot for semantic slot filling and slot2intent for intent classification; Let D i and D s These represent the number of different intent labels and semantic slot labels, respectively. To achieve bidirectional jointing of semantic slots and intent probability distributions, e1 is first passed through a network consisting of W... ID The constructed fully connected feedforward neural network layer, and then the final semantic feature vector H = {e2,…,e2} excluding the word '[CLS]'. N } By W SD The constructed fully connected feedforward neural network layer is finally normalized by applying a softmax function to the semantic slots and intent probability distribution. P ID =softmax(e1×W ID +b ID ) P SD =softmax(H×W SD +b SD ) Where the parameter matrix bias matrix It is the probability distribution of intent. It is the semantic slot probability distribution.

4. The online spoken language comprehension system based on the BERT-BJNN-EMGCRF model according to claim 3, characterized in that, The intent2Slot is used to extract semantic information from e1 to draw the intent probability distribution, and P ID The algorithm is copied N-1 times and concatenated with the semantic slot probability distribution. Then, the semantic slot prediction is output through a fully connected feedforward neural network. P Ir =repeat(P ID ),(N-1)times P S =[P SD ;P Ir ]×W S +b S in Parameter matrix The text represents the predicted semantic slot labels, and [;] represents the concatenation of matrices.

5. The online spoken language comprehension system based on the BERT-BJNN-EMGCRF model according to claim 3, characterized in that, The slot2intent first passes through P SD For P SD For each word in the semantic slot probability distribution, a weight is calculated. Then, the probability distributions of each semantic slot multiplied by the weight are summed to obtain supplementary information from the semantic slots. This information is then concatenated with the intent probability distribution to utilize the semantic slot probability distribution P. SD The intent is detected, and the predicted intent is finally calculated through a fully connected feedforward neural network. a=sigmoid(tanh(P SD ×W a )×V a ) P I =[P ID ;(a T ×P SD )]×W I +b I The parameter matrix for calculating weight a is... Predictive Intent 6. The online spoken language comprehension system based on the BERT-BJNN-EMGCRF model according to claim 5, characterized in that, The core of the conditional random field of the end-to-end masking graph is a connection matrix W composed of semantic slots and intent nodes. crf ; It indicates that a masking mechanism is applied to node i, namely the negative infinity-inf mask, i∈[1,N] where 1~N-1 are semantic slot nodes and N is an intent node. Similarly, the semantic slot nodes are masked. Among them una i Given the original predicted probability of the bidirectional joint network for the i-th node, according to the masking mechanism, una i The original predicted probabilities need to be filled with zero vectors; Let represent the predicted label of the i-th node in the t-th iteration, where i∈[1,N],t∈[0,T]. It is the label predicted in the t-th iteration of the intent node. The final predicted label for the i-th node is the predicted label of the intent or semantic slot, when i∈[1,N-1]. This represents the information about the predicted label distribution from the i-th semantic slot node and the intent node. When i = N, it represents the label distribution information of all nodes. It is a transformation matrix that reflects the implicit connection of the tag for compatibility. By transforming matrix W crf Calculate the correction vector Then let the correction vector and the original predicted probability una i Add them together to obtain the corrected prediction probability Then, using a masking mechanism and normalization operation, the predicted label for the (t+1)th iteration is obtained. arrive The calculation process iterates T times to obtain the final intent and semantic slot prediction labels.

7. The online spoken language comprehension system based on the BERT-BJNN-EMGCRF model according to claim 6, characterized in that, The overall network loss during model training consists of semantic slot loss and intent loss, treating each intent or semantic slot as a distinct category. Label smoothing recognition loss is used to supervise the training of various parameters in the shared network, bidirectional joint network, and end-to-end masking graph's conditional random field; their losses are defined as follows: L=δL I +(1-δ)L S Where q i The true label representing intent i, p i It is a predicted intent label; This represents the predicted label for the output semantic slot s corresponding to the j-th word. It is a real semantic slot tag, L I and L S δ represents the intent loss and semantic slot loss, respectively, and δ∈[0,1] adjusts the attention level of these two losses; The current loss is fed back into the network model, and the Adam optimizer is used to adjust the parameters in the model for continuous iterative optimization, so as to reduce the loss and improve the accuracy of target prediction.

8. The online spoken language comprehension system based on the BERT-BJNN-EMGCRF model according to claim 7, characterized in that, Step S3 specifically includes: S301. The user interface consists of a system homepage, a semantic slot extraction interface, and an intent detection interface. The system homepage includes a head and a body. The head first sets the encoding format and compatible browsers using meta tags. Then, when setting the webpage title using the title tag, it embeds a Flask Jinja syntax template block title within the tag and sets the title to 'home'. Next, it uses link and script tags to import Bootstrap and jQuery libraries from external sources. Finally, a reserved block head provides an interface for adding settings to other webpages. The body uses HTML controls to divide the entire webpage into four areas: header, side navigation bar, main content, and footer, corresponding to header, aside, main, and footer controls respectively. The header contains an control that provides hyperlinks to the current page. The aside contains 'System Homepage' hyperlink to the current page, 'Semantic Slot Extraction' hyperlink to the semantic slot extraction interface, and 'Intent Detection' hyperlink to the intent detection interface. The main contains a block. The body block is used to embed h1 and p controls to display the content title and text; after the control layout is completed, the size, position, color and size of the above controls are statically rendered using the imported Bootstrap library, and the controls are dynamically rendered using JavaScript technology. The semantic slot extraction interface inherits the layout of the system homepage using the extend syntax, modifying only the content of the blocks within the system homepage. This interface changes the content of the block title to 'slot' to replace the webpage title, adds a script control to the block head to import the slot.js file to enable communication between this webpage and the backend, and replaces the content of the block body with the title and text corresponding to the semantic slot extraction. In addition, there are form controls and div controls to display the extraction results. The form with the network response parameter method set to get or post contains input and button controls to enable the input and submission of dialog text. S302. On the server side, a local server object for the app is created using the Flask framework. Routes are created using the `route` function of the decorator view function. The URL is set to ' / ' as the root route, registered, and bound to the system homepage view. The system homepage is loaded by the function `render_template`, which returns the system homepage HTML file as a parameter. In addition, an event handling route ` / spoke_process` is set to handle interactive requests within the interface for semantic slot extraction and intent detection. The `methods` parameter is set to `get` or `post` to match the request method of the frontend. Finally, the `run` function is used to start the local server, with the port number `port` set to 2023 and the host number `host` set to 0.0.0.0 so that devices on the same network segment as the local host, i.e., the same local area network, can access the local server. S303, the intermediate communication mechanism uses asynchronous Ajax technology. In the .js file, the corresponding control is found by adding the control's ID number with #, and the communication between the front-end and back-end is achieved through the Ajax function. First, the submit button control is obtained and a click listener event is set for it. When a click action is detected, the click event function is executed. In the function, the input control is obtained by the ID number, and the content entered in the input control is obtained by the val function. Next, the request type parameter is set to 'get' in the Ajax function, and the data is sent in dictionary form with the data type as text. In addition, the URL parameter of Ajax is set to " / spoke_process?intent_slot=slot" in slot.js, which is to find the event handling route and implement the semantic slot extraction function. In intent.js, it is " / spoke_process?intent_slot=intent" to implement the intent extraction function. Finally, after the request is successful, the result is received from the server. The display control is found by the ID, and the result is displayed in the control using the html function. In the event handling routing function of the server, after receiving the GET request from the front end, the value of the intent_slot variable in the URL parameter of the AJAX request is obtained through the get function of the args method in the request. Then, it is determined whether it is intent detection or semantic slot extraction. The text in the input box in the data dictionary is obtained through the values ​​method in the request. Finally, the interface for obtaining the text processing is reserved in the function, and the processing result is returned to the corresponding interface through return.

9. The online spoken language comprehension system based on the BERT-BJNN-EMGCRF model according to claim 8, characterized in that, The online spoken language understanding system imports a trained network model into the online system to achieve online intent detection and semantic slot extraction of the dialogue text. The system imports the model from the model building file in the background file and creates a model object. Then, the saved model parameters are loaded into the model object to complete the import of the trained model.

Images