A natural language understanding method based on parameter sharing

Abstract

The present invention provides a natural language understanding method based on parameter sharing, which relates to the field of audio processing technology. The present invention establishes a natural language understanding model with multiple encoders, and updates and shares the parameters in the encoders in sequence according to a preset order during back propagation, thereby achieving the advantages of low computational complexity, low power consumption, high recognition rate, etc. The parameter sharing method is used to construct a natural language understanding model. By sharing parameters, the network depth is deepened without increasing network parameters, so that the model has better nonlinearity and can fit more responsible training data. After the output intent of the natural language understanding is re-encoded and represented, it interacts with the feature vector of the semantic slot. Thereby achieving the purpose of constraining the output of the semantic slot through the intent list. The present invention is designed so that all algorithm models can be deployed on the same, offline, local storage medium, and there is no need to interact with the cloud server.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to a natural language understanding method based on parameter sharing. Background Art

[0002] Natural language understanding (NLU) refers to the processes, technologies, and methods by which computers analyze and process natural language text to understand it. From a micro perspective, NLU refers to the mapping from natural language to machine content.

[0003] NLU plays a crucial role in text processing systems. Its functions primarily include answering questions, generating summaries, interpreting text, and translating it. It's an essential module for recommendation, question-answering, and search systems. From a broader perspective, natural language understanding refers to the ability of machines to perform certain language functions expected by humans.

[0004] Existing natural language understanding solutions are mainly based on voiceprint models. However, the voiceprint models and voiceprint model training requirements in existing solutions require a very large amount of computing and parameters, making them impossible to deploy on low-power chips without cloud services. For example, some common IoT robots require an Internet connection for natural language understanding. However, many applications currently need to be deployed on offline devices with limited computing power.

[0005] Therefore, it is necessary to provide a natural language understanding method based on parameter sharing to solve the above technical problems. Summary of the Invention

[0006] In order to solve one of the above technical problems, the present invention provides a natural language understanding method based on parameter sharing, establishes a natural language understanding model with multiple encoders, and performs model training, model verification and text data labeling on the natural language understanding model; wherein, the multiple encoders use the same weight parameters and keep the dimensions of the input vector and the output vector unified; the multiple encoders have the same structure as each other, and during back propagation, the parameters in the encoders are updated and shared in sequence according to a preset order, thereby deepening the network depth without increasing the network parameters.

[0007] As a further solution, the natural language understanding model is deployed through a neural network model including a gated recurrent unit and an attention mechanism, including a BIGRU layer, a Dense layer, an Encoder layer, a LN layer, and an Embedding layer.

[0008] Specifically, the natural language understanding model accepts input text and inputs each word in the input text into the BIGRU layer to extract deep-level features of the text, obtains a vector output corresponding to the input text, and sends it to the Dense layer. The features in the vector output undergo nonlinear transformation in the Dense layer, extracts the associations between these features, and finally maps them to the output space; the Encoder layer has multiple encoders, each encoder completes a feature extraction of the input and obtains the intent classification of each word; the intent classification of each word is sent to the LN layer and the Embedding layer respectively; the LN layer performs regularization operation, the Embedding layer performs dimensionality reduction operation, and outputs the slot sequence corresponding to the input text.

[0009] As a further solution, the Encoder layer includes an Attention layer, a Dropout1 layer, a LN+DENSE layer, and a Dropout2 layer.

[0010] Specifically, the Attention layer accepts input features and transmits the processed data to the Dropout1 layer and LN+DENSE layer for processing; the Dropout1 layer transmits the processed data to the LN+DENSE and Dropout2 layers for processing; the LN+DENSE layer transmits the processed data to the Dropout2 layer for processing, and the Dropout2 layer outputs the processed data.

[0011] As a further solution, the natural language understanding model is trained using training text data. The training text data is obtained through the following steps:

[0012] Step A1: collecting initial audio data through a silicon microphone or an electret microphone, wherein the initial audio data corresponds to prior real text data;

[0013] Step A2: performing speech recognition processing on the initial audio data to obtain initial speech recognition data;

[0014] Step A3: Compare the acquired initial speech recognition data with the real text data, and generate a corresponding error correction dictionary for the misrecognized data;

[0015] Step A4: Collect training audio data using a silicon microphone or an electret microphone;

[0016] Step A5: Perform speech recognition processing on the training audio data to obtain training text data.

[0017] As a further solution, a length padding operation is performed on the training text data. The length padding operation is an indefinite length padding operation and is performed through the following steps:

[0018] Step B1: Read the training text data in batches;

[0019] Step B2: Find the maximum text length in each batch;

[0020] Step B3: Read the maximum text length in the same batch of training text data;

[0021] Step B4: Fill the training text data of the current batch with data according to the maximum text length of the same batch until the length is equal to the maximum text length of the same batch.

[0022] As a further solution, the vector output of the BIGRU layer at each moment is processed through three Dense layers to obtain the Query vector, Key vector, and Value vector. The importance of each word in the input text relative to other words in the sentence is calculated from the Query vector and Key vector, and the weight used is obtained by normalization through the Softmax function. The obtained normalized weight is weighted and summed with the Value vector to obtain the attention of the current input text. The attention formula is:

[0023]

[0024] in, Represent the vector representations corresponding to the Query vector, Key vector, and Value vector respectively.

[0025] As a further solution, the multi-head attention concatenates multiple attention output results to obtain the semantic representation of the current input text based on the multi-head attention. The semantic representation of the multi-head attention is:

[0026]

[0027] in, Represent the vector representations corresponding to the Query vector, Key vector, and Value vector respectively. Indicates multiple attentions, () represents the splicing function.

[0028] As a further solution, before the dimensionality reduction operation is performed on the Embedding layer, the intent classification results of each word Perform one-hot encoding and obtain the intent vector after the Embedding layer After the parameters are regularized in the LN layer of the neural network model, the obtained vector is transformed through a weight, and the obtained intention vector replaces the weight in the LN layer; the BIGRU layer outputs the vector matrix , and after passing through the LN layer, we get the vector , by adding the vector and Perform splicing to obtain the feature representation of the input text ,Depend on Perform part-of-speech tagging on the input text and vector Update; the specific calculation formula is:

[0029]

[0030]

[0031] in, is the intent classification result, is the intention vector, Output vector matrix for the BIGRU layer, for The vector obtained after the LN layer is is the feature of the input text, () represents the splicing function.

[0032] Compared with related technologies, the natural language understanding method based on parameter sharing provided by the present invention has the following beneficial effects:

[0033] The existing voiceprint models and voiceprint model training require a very large amount of computation and parameters, making them difficult to deploy on low-power chips without cloud services. The main innovations of this embodiment include:

[0034] (1) A parameter sharing method is used to build a natural language understanding model. By sharing parameters, the network depth is deepened without increasing the network parameters, so that the model has better nonlinearity and can fit more responsible training data.

[0035] (2) The model achieves better performance by using the interaction between intent and semantic slot features. After re-encoding the output intent of natural language understanding, it interacts with the feature vector of the semantic slot. This achieves the goal of constraining the semantic slot output through the intent list.

[0036] (3) In this embodiment, all algorithm models can be deployed on the same, offline, local storage medium without the need to interact with the cloud server. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A preferred training flow chart of a natural language understanding method based on parameter sharing provided by an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the structure of a CNN convolutional neural network provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of the structure of a GRU recurrent neural network provided by an embodiment of the present invention;

[0040] Figure 4 An optimal verification flow chart of a natural language understanding method based on parameter sharing provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] like Figure 1 As shown, this embodiment provides a natural language understanding method based on parameter sharing, establishes a natural language understanding model with multiple encoders, and performs model training, model verification and text data labeling on the natural language understanding model; wherein, the multiple encoders use the same weight parameters and keep the dimensions of the input vector and the output vector unified; the multiple encoders have the same structure as each other, and during back propagation, the parameters in the encoders are updated and shared in sequence according to a preset order, thereby deepening the network depth without increasing the network parameters.

[0043] It should be noted that this embodiment achieves the advantages of low computational complexity, low power consumption, and high recognition rate by establishing a natural language understanding model with multiple encoders and updating and sharing the parameters in the encoders in sequence according to a preset order during back propagation.

[0044] The existing voiceprint models and voiceprint model training require a very large amount of computation and parameters, making them difficult to deploy on low-power chips without cloud services. The main innovations of this embodiment include:

[0045] (1) A parameter sharing method is used to build a natural language understanding model. By sharing parameters, the network depth is deepened without increasing the network parameters, so that the model has better nonlinearity and can fit more responsible training data.

[0046] (2) The model achieves better performance by using the interaction between intent and semantic slot features. After re-encoding the output intent of natural language understanding, it interacts with the feature vector of the semantic slot. This achieves the goal of constraining the semantic slot output through the intent list.

[0047] (3) In this embodiment, all algorithm models can be deployed on the same, offline, local storage medium without the need to interact with the cloud server.

[0048] As a further solution, the natural language understanding model is deployed through a neural network model including a gated recurrent unit and an attention mechanism, including a BIGRU layer, a Dense layer, an Encoder layer, a LN layer, and an Embedding layer.

[0049] Specifically, the natural language understanding model accepts input text and inputs each word in the input text into the BIGRU layer to extract deep-level features of the text, obtains a vector output corresponding to the input text, and sends it to the Dense layer. The features in the vector output undergo nonlinear transformation in the Dense layer, extracts the associations between these features, and finally maps them to the output space; the Encoder layer has multiple encoders, each encoder completes a feature extraction of the input and obtains the intent classification of each word; the intent classification of each word is sent to the LN layer and the Embedding layer respectively; the LN layer performs regularization operation, the Embedding layer performs dimensionality reduction operation, and outputs the slot sequence corresponding to the input text.

[0050] Characteristics of the BN layer: Neurons in the same layer have different input means and variances, and all samples in the same batch have the same mean and variance.

[0051] Advantages of the LN layer: It is not affected by the batch size of the samples. It is suitable for RNN training but not for CNN training.

[0052] As a further solution, the Encoder layer includes an Attention layer, a Dropout1 layer, a LN+DENSE layer, and a Dropout2 layer.

[0053] Specifically, the Attention layer accepts input features and transmits the processed data to the Dropout1 layer and LN+DENSE layer for processing; the Dropout1 layer transmits the processed data to the LN+DENSE and Dropout2 layers for processing; the LN+DENSE layer transmits the processed data to the Dropout2 layer for processing, and the Dropout2 layer outputs the processed data.

[0054] As a further solution, the natural language understanding model is trained using training text data. The training text data is obtained through the following steps:

[0055] Step A1: collecting initial audio data through a silicon microphone or an electret microphone, wherein the initial audio data corresponds to prior real text data;

[0056] Step A2: performing speech recognition processing on the initial audio data to obtain initial speech recognition data;

[0057] Step A3: Compare the acquired initial speech recognition data with the real text data, and generate a corresponding error correction dictionary for the misrecognized data;

[0058] Step A4: Collect training audio data using a silicon microphone or an electret microphone;

[0059] Step A5: Perform speech recognition processing on the training audio data to obtain training text data.

[0060] It should be noted that current text preprocessing operations can be divided into two types: fixed-length and variable-length padding. Fixed-length text padding aims to fix the input text to a uniform length standard. Specifically, any insufficient length is padded with special markers, and any length exceeding the specified maximum is truncated. This section applies variable-length padding to the text corpus data. The specific implementation steps are as follows: First, the training corpus data is divided into batches according to a defined batch size. The maximum text length in each batch is calculated. Let the maximum text sequence length within the same batch be L, and the remaining data is padded with length L. Using variable-length padding to construct the training corpus data has two advantages: First, the model does not need to perform padding on the input text during inference, reducing the computational effort and hardware overhead. Second, during model training, due to the characteristics of recurrent neural networks, they cannot be trained in parallel. Applying equal-length padding to the input data within the same batch reduces the training length of the recurrent neural network portion of the model, thereby accelerating model training. The improvement in model training speed is particularly noticeable when the length of the training text data in the dataset varies greatly. It is important to note that if text preprocessing uses variable-length padding, then during the model construction process, the shape of the weight matrix and bias of each layer in the model should be independent of the input text length.

[0061] As a further solution, a length padding operation is performed on the training text data. The length padding operation is an indefinite length padding operation and is performed through the following steps:

[0062] Step B1: Read the training text data in batches;

[0063] Step B2: Find the maximum text length in each batch;

[0064] Step B3: Read the maximum text length in the same batch of training text data;

[0065] Step B4: Fill the training text data of the current batch with data according to the maximum text length of the same batch until the length is equal to the maximum text length of the same batch.

[0066] As a further solution, the vector output of the BIGRU layer at each moment is processed through three Dense layers to obtain the Query vector, Key vector, and Value vector. The importance of each word in the input text relative to other words in the sentence is calculated from the Query vector and Key vector, and the weight used is obtained by normalization through the Softmax function. The obtained normalized weight is weighted and summed with the Value vector to obtain the attention of the current input text. The attention formula is:

[0067]

[0068] in, Represent the vector representations corresponding to the Query vector, Key vector, and Value vector respectively.

[0069] It should be noted that the attention mechanism specifically passes the BiGRU's vector output at each moment through three fully connected layers to output three states, resulting in vector representations of the query vector (Query (Q), Key (K), and Value (V), respectively. The K and V vectors contain a semantic representation of the complete sequence information. The importance of each word relative to other words in the sentence is calculated using Q and K. The weights used are then normalized using the softmax function. The resulting normalized weights are then weighted and summed with V to form the current semantic representation.

[0070] As a further solution, the multi-head attention concatenates multiple attention output results to obtain the semantic representation of the current input text based on the multi-head attention. The semantic representation of the multi-head attention is:

[0071]

[0072] in, Represent the vector representations corresponding to the Query vector, Key vector, and Value vector respectively. Indicates multiple attentions, () represents the splicing function.

[0073] It should be noted that this paper uses a multi-head attention mechanism, which focuses on the semantic information in the text from multiple dimensions. Specifically, the output results of multiple attention layers are concatenated to obtain the semantic representation of the current text sequence based on multi-head attention.

[0074] To reduce model size without compromising performance, we use the same weight parameters for multiple encoders. Using the same weight parameters across multiple encoders requires ensuring the consistency of the input and output vectors. Specifically, the dimensions of the output vector of the fully connected layer within the encoder must be consistent with the dimensions of the vector input to the encoder. This multi-encoder architecture allows for more effective focus on important information within a text sequence, resulting in a better semantic representation of the text.

[0075] As a further solution, before the dimensionality reduction operation is performed on the Embedding layer, the intent classification results of each word Perform one-hot encoding and obtain the intent vector after the Embedding layer After the parameters are regularized in the LN layer of the neural network model, the obtained vector is transformed through a weight, and the obtained intention vector replaces the weight in the LN layer; the BIGRU layer outputs the vector matrix , and after passing through the LN layer, we get the vector , by adding the vector and Perform splicing to obtain the feature representation of the input text ,Depend on Perform part-of-speech tagging on the input text and vector Update; the specific calculation formula is:

[0076]

[0077]

[0078] in, is the intent classification result, is the intention vector, Output vector matrix for the BIGRU layer, for The vector obtained after the LN layer is is the feature of the input text, () represents the splicing function.

[0079] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A natural language understanding method based on parameter sharing, characterized in that: Establish a natural language understanding model with multiple encoders, and perform model training, model validation, and text data labeling on the natural language understanding model; wherein the multiple encoders use the same weight parameters and maintain the same dimensionality of the input vector and output vector; the multiple encoders have the same structure, and the parameters in the encoders are updated and shared in a preset order during backpropagation, thereby deepening the network without increasing network parameters; The natural language understanding model is deployed through a neural network model including a gated recurrent unit and an attention mechanism, including a BIGRU layer, a Dense layer, an Encoder layer, a LN layer, and an Embedding layer; The natural language understanding model accepts input text and feeds each word in the input text into the BIGRU layer to extract deep-level features of the text, obtaining a vector output corresponding to the input text. This vector output is then sent to the Dense layer, where the features in the vector output undergo nonlinear transformations, extracting the associations between these features and finally mapping them to the output space. The Encoder layer has multiple encoders, each of which extracts input features once and obtains the intent classification of each word. The intent classification of each word is sent to the LN layer and the Embedding layer respectively. The LN layer performs regularization, and the Embedding layer performs dimensionality reduction, and outputs a slot sequence corresponding to the input text. The Encoder layer includes the Attention layer, the Dropout1 layer, the LN+DENSE layer, and the Dropout2 layer; The Attention layer accepts the input features and transmits the processed data to the Dropout1 layer and LN+DENSE layer for processing; the Dropout1 layer transmits the processed data to the LN+DENSE and Dropout2 layers for processing; the LN+DENSE layer transmits the processed data to the Dropout2 layer for processing, and the Dropout2 layer outputs the processed data.

2. A natural language understanding method based on parameter sharing according to claim 1, characterized in that: The natural language understanding model is trained using training text data, and the training text data is obtained through the following steps: Step A1: collecting initial audio data through a silicon microphone or an electret microphone, wherein the initial audio data corresponds to prior real text data; Step A2: performing speech recognition processing on the initial audio data to obtain initial speech recognition data; Step A3: Compare the acquired initial speech recognition data with the real text data, and generate a corresponding error correction dictionary for the misrecognized data; Step A4: Collect training audio data using a silicon microphone or an electret microphone; Step A5: Perform speech recognition processing on the training audio data to obtain training text data.

3. A natural language understanding method based on parameter sharing according to claim 2, characterized in that: A length padding operation is also performed on the training text data. The length padding operation is an indefinite length padding operation and is performed through the following steps: Step B1: Read the training text data in batches; Step B2: Find the maximum text length in each batch; Step B3: Read the maximum text length in the same batch of training text data; Step B4: Fill the training text data of the current batch with data according to the maximum text length of the same batch until the length is equal to the maximum text length of the same batch.

4. The natural language understanding method based on parameter sharing according to claim 1, characterized in that: The vector output of the BIGRU layer at each moment is processed by three Dense layers to obtain the Query vector, Key vector and Value vector. The importance of each word in the input text relative to other words in the sentence is calculated from the Query vector and Key vector, and the weight used is obtained by normalization processing with the Softmax function. The obtained normalized weight is weighted and summed with the Value vector to obtain the attention of the current input text. The attention formula is: in, Represent the vector representations corresponding to the Query vector, Key vector, and Value vector respectively.

5. The natural language understanding method based on parameter sharing according to claim 4, characterized in that: Multi-head attention concatenates multiple attention outputs to obtain a semantic representation of the current input text based on multi-head attention. The semantic representation of the multi-head attention is: in, Represent the vector representations corresponding to the Query vector, Key vector, and Value vector respectively. Indicates multiple attentions, () represents the splicing function.

6. The natural language understanding method based on parameter sharing according to claim 1, characterized in that: Before the dimensionality reduction operation is performed on the Embedding layer, the intent classification results of each word Perform one-hot encoding and obtain the intent vector after the Embedding layer ,After regularizing the parameters in the LN layer of the neural network model, the obtained vector is transformed through a weight, and the obtained intention vector replaces the weight in the LN layer; The BIGRU layer outputs a matrix of vectors , and after passing through the LN layer, we get the vector , by adding the vector and Perform splicing to obtain the feature representation of the input text ,Depend on Perform part-of-speech tagging on the input text and vector Update; the specific calculation formula is: in, is the intent classification result, is the intention vector, Output vector matrix for the BIGRU layer, for The vector obtained after the LN layer is is the feature of the input text, () represents the splicing function.

Images