Oral language understanding method and device, electronic equipment, storage medium and product

Abstract

The invention provides a spoken language understanding method and device, electronic equipment, a storage medium and a product. The spoken language understanding method comprises the steps of obtaining a to-be-understood voice signal; inputting the voice signal into a trained spoken language understanding model to obtain semantic information of the voice signal; a spoken language understanding model comprising: an automatic speech recognition encoder for converting an acoustic feature of the speech signal into an acoustic representation; the automatic speech recognition decoder is used for performing text transcription on the acoustic representation to obtain a speech transcription text representation; the fusion module is used for fusing the acoustic representation and the voice transcription text representation to obtain a joint representation; the spoken language understanding decoder is used for performing semantic prediction on the joint representation to obtain a semantic representation; and the output layer is used for obtaining the semantic information of the voice signal based on the semantic representation, so that the semantic output can be directly predicted from the voice input through the end-to-end spoken language understanding model, the error propagation problem is reduced, and the accuracy of the semantic information of the voice signal output by the model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a spoken language understanding method, device, electronic device, storage medium and product. Background Art

[0002] Spoken Language Understanding (SLU) refers to extracting semantic information from user voice commands, such as domain, intent, and entity name. Current SLU systems usually adopt a cascade approach, that is, first converting speech into text through an Automatic Speech Recognition (ASR) system, and then extracting semantic information from the text through a Natural Language Understanding (NLU) system.

[0003] However, this approach has the problem of accumulation of ASR errors and neglect of acoustic information in speech, that is, the conversion errors in the ASR stage will be passed to the NLU stage, thereby reducing the overall performance of the entire system, and the ASR stage usually only focuses on the conversion of speech to text, which may lose acoustic features useful for NLU, such as intonation, intensity, etc. These features may be very important for understanding the user's emotional attitude or speech intention, resulting in a significant reduction in the accuracy of the spoken language understanding results. Summary of the invention

[0004] The present invention provides a spoken language understanding method, device, electronic device, storage medium and product, which are used to solve the defects of the prior art that ASR errors are accumulated and acoustic information in speech is ignored, and to achieve accurate output of spoken language semantic understanding results.

[0005] The present invention provides a spoken language comprehension method, comprising: Acquire a speech signal to be understood; The speech signal is input into a trained spoken language comprehension model to obtain semantic information of the speech signal; wherein the spoken language comprehension model comprises: an automatic speech recognition encoder, used to convert the acoustic features of the speech signal into an acoustic representation; an automatic speech recognition decoder, used to perform text transcription on the acoustic representation to obtain a speech transcription text representation; a fusion module, used to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation; a spoken language comprehension decoder, used to perform semantic prediction on the joint representation to obtain a semantic representation; and an output layer, used to obtain the semantic information of the speech signal based on the semantic representation.

[0006] According to a spoken language understanding method provided by the present invention, the fusion module comprises: A first cross-attention layer, used for performing cross-attention processing on the acoustic representation and the speech transcription text representation to obtain a first joint representation; A first feed-forward layer, configured to perform linear processing on the first joint representation to obtain a second joint representation; The first adapter layer is used to adjust the second joint representation to obtain a joint representation.

[0007] According to a spoken language understanding method provided by the present invention, the spoken language understanding decoder comprises: The self-attention layer is used to perform self-attention processing on the previous semantic token output by the spoken language understanding decoder to obtain the previous semantic representation; A second cross attention layer is used to perform cross attention processing on the previous semantic representation and the joint representation to obtain a first semantic representation; A second feed-forward layer, used for performing linear processing on the first semantic representation to obtain a second semantic representation; The second adapter layer is used to adjust the second semantic representation to obtain a semantic representation.

[0008] According to a spoken language comprehension method provided by the present invention, the spoken language comprehension model is trained in the following manner: Acquire a sample speech signal; The sample speech signal is input into the spoken language understanding model to be trained to obtain the training semantic information of the sample speech signal; wherein the spoken language understanding model to be trained comprises: an automatic speech recognition encoder, which is used to convert the acoustic features of the sample speech signal into a training acoustic representation; an automatic speech recognition decoder, which is used to perform text transcription on the training acoustic representation to obtain a training speech transcription text representation; a fusion module, which is used to perform feature fusion on the training acoustic representation and the training speech transcription text representation to obtain a training joint representation; a spoken language understanding decoder, which is used to perform semantic prediction on the training joint representation to obtain a training semantic representation; an output layer, which is used to obtain the training semantic information of the sample speech signal based on the training semantic representation; Calculating the overall multi-task learning loss based on the training semantic information, the training speech transcription text representation and the adjustment parameters; The model parameters are adjusted according to the overall multi-task learning loss to obtain a trained spoken language comprehension model.

[0009] According to a spoken language comprehension method provided by the present invention, the fusion module in the spoken language comprehension model to be trained includes: A first cross-attention layer, used for performing cross-attention processing on the training acoustic representation and the training speech transcription text representation to obtain a first training joint representation; A first Dropout layer is used to randomly discard the neural network units in the fusion module to obtain a second training joint representation output after the retained neural network units process the first training joint representation; a first feed-forward layer, configured to perform linear processing on the second training joint representation to obtain a third training joint representation; The first adapter layer is used to adjust the third training joint representation to obtain a training joint representation.

[0010] According to a spoken language understanding method provided by the present invention, the spoken language understanding decoder in the spoken language understanding model to be trained comprises: The self-attention layer is used to perform self-attention processing on the previous training semantic token output by the spoken language understanding decoder to obtain the previous training semantic representation; A second cross attention layer is used to perform cross attention processing on the previous training semantic representation and the training joint representation to obtain a first training semantic representation; A second Dropout layer is used to randomly discard the neural network units in the spoken language understanding decoder to obtain a second training semantic representation output by the retained neural network units after processing the first training semantic representation; a second feed-forward layer, configured to perform linear processing on the second training semantic representation to obtain a third training semantic representation; The second adapter layer is used to adjust the third training semantic representation to obtain a training semantic representation.

[0011] According to a spoken language understanding method provided by the present invention, the adapter layer is specifically used for: Based on the output result of the feedforward layer, the dimension reduction linear mapping matrix and the dimension reduction bias term, the dimension reduction feature representation is obtained; The output result of the adapter layer is obtained based on the output result of the feedforward layer, the features obtained by performing nonlinear processing on the reduced-dimensional feature representation through an activation function, the increased-dimensional linear mapping matrix and the increased-dimensional bias term.

[0012] The present invention also provides a spoken language understanding device, comprising: A speech signal acquisition module, used to acquire a speech signal to be understood; A semantic information output module is used to input the speech signal into a trained spoken language comprehension model to obtain the semantic information of the speech signal; wherein the spoken language comprehension model includes: an automatic speech recognition encoder, used to convert the acoustic features of the speech signal into an acoustic representation; an automatic speech recognition decoder, used to perform text transcription on the acoustic representation to obtain a speech transcription text representation; a fusion module, used to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation; a spoken language comprehension decoder, used to perform semantic prediction on the joint representation to obtain a semantic representation; and an output layer, used to obtain the semantic information of the speech signal based on the semantic representation.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned spoken language comprehension methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the spoken language comprehension method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned spoken language comprehension methods.

[0016] The spoken language understanding method, device, electronic device, storage medium and product provided by the present invention, after acquiring the speech signal to be understood, input the speech signal into a trained spoken language understanding model, use an automatic speech recognition encoder to convert the acoustic features of the speech signal into an acoustic representation, use an automatic speech recognition decoder to perform text transcription on the acoustic representation to obtain a speech transcription text representation, use a fusion module to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation, use a spoken language understanding decoder to perform semantic prediction on the joint representation to obtain a semantic representation, and use an output layer to obtain semantic information of the speech signal based on the semantic representation, so that through an end-to-end spoken language understanding model, semantic output can be directly predicted from speech input, effectively reducing the error propagation problem and improving the accuracy of the semantic information of the speech signal output by the spoken language understanding model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1It is a flowchart of a spoken language comprehension method provided by an embodiment of the present invention.

[0019] Figure 2 It is a schematic diagram of the architecture of an ASR encoder and an ASR decoder provided in an embodiment of the present invention.

[0020] Figure 3 It is a schematic diagram of the architecture of the spoken language comprehension model provided by an embodiment of the present invention.

[0021] Figure 4 This is one of the training flow diagrams of the spoken language comprehension model provided in an embodiment of the present invention.

[0022] Figure 5 This is the second schematic diagram of the training process of the spoken language comprehension model provided in the embodiment of the present invention.

[0023] Figure 6 It is a training diagram of the fusion module provided in an embodiment of the present invention.

[0024] Figure 7 It is a training diagram of the SLU decoder provided in an embodiment of the present invention.

[0025] Figure 8 It is a structural diagram of a spoken language understanding device provided by an embodiment of the present invention.

[0026] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Figure 1 Schematic diagram of the flow of the spoken language comprehension method provided by the embodiment of the present invention. Figure 1 The embodiment of the present invention provides a spoken language understanding method, which may specifically include the following steps: Step 110: Acquire a speech signal to be understood.

[0029] It should be noted that the execution subject of the spoken language understanding method provided in the embodiment of the present invention may be an electronic device, a component in an electronic device, an integrated circuit or a chip. The electronic device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present invention.

[0030] Specifically, the user's voice can be received through a voice acquisition device, the analog signal of the user's voice can be converted into a digital signal, and the digital signal can be de-noised to obtain a voice signal. The user's voice signal can be spoken understood, thereby extracting semantic information from the voice signal.

[0031] Step 120, inputting the speech signal into a trained spoken language comprehension model to obtain semantic information of the speech signal; wherein the spoken language comprehension model includes: an automatic speech recognition encoder, used to convert the acoustic features of the speech signal into an acoustic representation; an automatic speech recognition decoder, used to perform text transcription on the acoustic representation to obtain a speech transcription text representation; a fusion module, used to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation; a spoken language comprehension decoder, used to perform semantic prediction on the joint representation to obtain a semantic representation; an output layer, used to obtain the semantic information of the speech signal based on the semantic representation.

[0032] In an embodiment of the present invention, the ASR encoder and the ASR decoder can jointly form an Attention-based Encoder-Decoder (AED) architecture model for speech feature learning, and the Conformer network architecture is used to combine the Transformer's self-attention mechanism with the local receptive field of the Convolutional Neural Networks (CNN) to efficiently learn speech feature information. While the Transformer is able to capture long-distance dependencies, CNN is used to enhance the model's sensitivity to local features, thereby achieving a better performance balance.

[0033] Figure 2 Schematic diagram of the architecture of the ASR encoder and ASR decoder provided by the embodiment of the present invention. Figure 2 , the acoustic features of the speech signal can be input into the ASR encoder, and then encoded and feature extracted by the relative position encoding layer, convolution layer, feedforward layer, self-attention layer, convolution layer and feedforward layer in the ASR encoder, so as to convert the acoustic features into acoustic representations. The acoustic representation output by the feedforward layer and the previous text token output by the ASR decoder can be input into the ASR decoder, and then transcribed by the self-attention layer, cross-attention layer and feedforward layer in the ASR decoder, so as to obtain the speech transcription text representation, and then the speech transcription text representation and the acoustic representation can be input into the fusion module for fusion. In addition, the speech transcription text representation output by the ASR decoder can be input into the softmax output layer to obtain text information.

[0034] Traditional SLU systems usually adopt a cascade method, first performing ASR processing and then NLU processing, which causes ASR errors to be transmitted to the NLU stage, affecting the overall performance and resulting in low accuracy of spoken language understanding results. The embodiments of the present invention integrate automatic speech recognition and spoken language understanding functions through an end-to-end model, thereby being able to directly predict semantic output from speech input, effectively reducing the error propagation problem and facilitating improved accuracy of spoken language understanding results.

[0035] Figure 3 Schematic diagram of the structure of the spoken language comprehension model provided by the embodiment of the present invention. Figure 3 ,The spoken language understanding model may include an automatic speech recognition (ASR) encoder, an automatic speech recognition (ASR) decoder, a fusion module, a spoken language understanding (SLU) decoder, and an output layer.

[0036] Specifically, the feature extraction of the speech signal can be performed first to obtain the acoustic features of the speech signal, and the acoustic features of the speech signal can be input into the trained spoken language understanding model for processing. The ASR encoder can encode the acoustic features, thereby converting the acoustic features of the speech signal into an acoustic representation. The ASR decoder can perform text transcription based on the acoustic representation to obtain a speech transcription text representation. The fusion module can combine the acoustic representation from the ASR encoder and the speech transcription text representation of the ASR decoder, use the acoustic representation as a query, and use the speech transcription text representation as a key and value to generate a joint representation, and input the joint representation into the SLU decoder for semantic prediction. The SLU decoder can use the joint representation obtained from the fusion module to predict the intent and slot value to obtain a semantic representation. The softmax output layer can convert the semantic representation vector output by the SLU decoder into a probability distribution, select the word with the highest probability at each time step as the final semantic prediction result of the time step, and summarize the final semantic prediction results of each time step to obtain the semantic information of the speech signal.

[0037] After acquiring a speech signal to be understood, the embodiment of the present invention inputs the speech signal into a trained spoken language understanding model, uses an automatic speech recognition encoder to convert the acoustic features of the speech signal into an acoustic representation, uses an automatic speech recognition decoder to perform text transcription on the acoustic representation to obtain a speech transcription text representation, uses a fusion module to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation, uses a spoken language understanding decoder to perform semantic prediction on the joint representation to obtain a semantic representation, and uses an output layer to obtain semantic information of the speech signal based on the semantic representation, so that through an end-to-end spoken language understanding model, semantic output can be directly predicted from speech input, thereby effectively reducing the error propagation problem and improving the accuracy of the semantic information of the speech signal output by the spoken language understanding model.

[0038] Based on any of the above embodiments, the fusion module includes: a first cross-attention layer, used to perform cross-attention processing on the acoustic representation and the speech transcription text representation to obtain a first joint representation; a first feedforward layer, used to perform linear processing on the first joint representation to obtain a second joint representation; a first adapter layer, used to adjust the second joint representation to obtain a joint representation.

[0039] In an embodiment of the present invention, the fusion module may include a first cross-attention layer, a first feed-forward layer and a first adapter layer.

[0040] Specifically, the acoustic representation output by the ASR encoder and the speech-to-text representation output by the ASR decoder can be input into the first cross-attention layer in the fusion module. In the first cross-attention layer, the cross-attention mechanism can be used to perform cross-attention calculations between the acoustic representation and the text representation to generate a first joint representation that combines two different modal information, acoustic information and speech-to-text information.

[0041] The first feed-forward layer in the fusion module can perform a linear transformation on the first joint representation through at least one fully connected layer, and perform a nonlinear activation process through a nonlinear activation function, thereby obtaining a second joint representation obtained after the first joint representation is subjected to the linear transformation and the nonlinear activation process. The first adapter layer in the fusion module can adjust the second joint representation to obtain a joint representation.

[0042] The embodiments of the present invention can make better use of multimodal information and integrate the acoustic representation and text representation from ASR through a fusion module, so that the spoken language understanding model can more comprehensively utilize the information in the speech signal, which is beneficial for the spoken language understanding model to capture more semantic and contextual information and improve the accuracy and robustness of the output results of the spoken language understanding model.

[0043] By adding an adapter layer to the fusion module, the embodiment of the present invention can fine-tune the fusion module in the spoken language comprehension model to adapt to new tasks without retraining the entire model, which is conducive to further improving the accuracy of the output results of the spoken language comprehension model.

[0044] Based on any of the above embodiments, the spoken language comprehension decoder includes: a self-attention layer, which is used to perform self-attention processing on the previous semantic token output by the spoken language comprehension decoder to obtain the previous semantic representation; a second cross-attention layer, which is used to perform cross-attention processing on the previous semantic representation and the joint representation to obtain a first semantic representation; a second feedforward layer, which is used to perform linear processing on the first semantic representation to obtain a second semantic representation; and a second adapter layer, which is used to adjust the second semantic representation to obtain a semantic representation.

[0045] In an embodiment of the present invention, the SLU decoder may include a self-attention layer, a second cross-attention layer, a second feed-forward layer, and a second adapter layer.

[0046] Specifically, the semantic representation output by the SLU decoder at the previous time step and the joint representation output by the fusion module can be input to the SLU decoder for processing. In the self-attention layer, the self-attention mechanism can be used to process the input previous semantic representation and joint representation, so that the spoken language understanding model pays attention to other tokens in the sequence when processing each semantic token, thereby capturing global dependencies. In the second cross-attention layer, the cross-attention mechanism can be used to perform cross-attention calculations between the previous semantic representation and the joint representation, and the previous semantic information can be integrated into the current joint information to obtain the first semantic representation.

[0047] The second feed-forward layer in the SLU decoder can perform a linear transformation on the first semantic representation through at least one fully connected layer, and perform a nonlinear activation process through a nonlinear activation function, thereby obtaining a second semantic representation obtained after the first semantic representation is linearly transformed and nonlinearly activated. The second adapter layer in the SLU decoder can adjust the second semantic representation to obtain a semantic representation.

[0048] By adding an adapter layer to the SLU decoder, the embodiment of the present invention can fine-tune the SLU decoder in the spoken language comprehension model to adapt to new tasks without retraining the entire model, which is beneficial to further improve the accuracy of the output results of the spoken language comprehension model.

[0049] Figure 4 This is one of the training flow diagrams of the spoken language comprehension model provided by the embodiment of the present invention. Figure 4 Based on any of the above embodiments, the spoken language comprehension model can be trained by the following steps: Step 410: Acquire a sample speech signal.

[0050] In the embodiment of the present invention, after the sample speech signal is obtained, the sample speech signal can be converted into a Mel spectrum to extract the acoustic features of the sample speech signal, and then the acoustic features of the sample speech signal are input into the model for processing.

[0051] The model input representation can be as follows: X=(x1,x2,...,x t )(1); For time t, the acoustic features of the sample speech signal can be expressed as x t , the acoustic feature sequence of the sample speech signal can be expressed as X, x t =log(MelSpectrogram(s(t))). Wherein, s(t) may represent a speech signal frame at time t, MelSpectrogram may represent the conversion of the frame speech signal into a Mel spectrum, and log may represent taking the logarithm of the result.

[0052] Step 420, input the sample speech signal into the spoken language comprehension model to be trained to obtain the training semantic information of the sample speech signal; wherein the spoken language comprehension model to be trained includes: an automatic speech recognition encoder, used to convert the acoustic features of the sample speech signal into a training acoustic representation; an automatic speech recognition decoder, used to perform text transcription on the training acoustic representation to obtain a training speech transcription text representation; a fusion module, used to perform feature fusion on the training acoustic representation and the training speech transcription text representation to obtain a training joint representation; a spoken language comprehension decoder, used to perform semantic prediction on the training joint representation to obtain a training semantic representation; an output layer, used to obtain the training semantic information of the sample speech signal based on the training semantic representation.

[0053] Figure 5 This is the second schematic diagram of the training process of the spoken language comprehension model provided by the embodiment of the present invention. Figure 5 , the model output representation can be as follows: (2); Among them, Y represents the model output, Represents the output of the ASR encoder at each time step u by combining and the output of the SLU encoder at each time step v Get the model output.

[0054] The output of the ASR decoder is a text sequence of the training speech transcription: (3); in, represents the output of the ASR decoder from the 1st time step to the u-1th time step The output of the ASR decoder at the last time step u is obtained by combining the acoustic feature sequence X of the sample speech signal , It can represent the training speech transcription text representation output by the ASR decoder at time step u.

[0055] The output of the SLU decoder is an intent or a word slot: (4); in, represents the output of the combined SLU decoder from the 1st time step to the u-1th time step The output of the SLU decoder at the last time step u is obtained by combining the acoustic feature sequence X of the sample speech signal , It can represent the trained semantic representation output by the SLU decoder at time step u.

[0056] (5); According to the model architecture design, the ASR encoder is used to receive a series of acoustic feature inputs of the sample speech signal and generate corresponding training acoustic representations, which are then input into the ASR decoder for processing.

[0057] The output of the ASR encoder can be expressed as: (6); The ASR decoder is a Transformer-based autoregressive model that predicts the speech transcript symbol by symbol. The ASR decoder can use the predicted output of the previous time step Harmonic representation To predict the output of the next time step : (7); Therefore, the ASR decoder can predict the speech transcription text based on the output of the ASR encoder. This process can be divided into two steps: (1) Calculate the representation of the current state: The ASR decoder can use the previous prediction results (i.e., the part of the speech transcription text sequence that has been generated) and the acoustic representation output by the ASR encoder as input, focus on the relevant part of the input features through the attention mechanism, and calculate the output representation of the current position, thereby ensuring that the spoken language understanding model can generate the appropriate next word or character based on the previous text and current acoustic information.

[0058] (2) Generate prediction results: After obtaining the representation of the current state, the model can use a linear layer and a Softmax function to calculate the probability distribution of the next output symbol. Finally, the model can predict the next most likely word or phrase based on the current representation.

[0059] In an embodiment of the present invention, the fusion module can be used to combine the acoustic representation from the ASR encoder and the text representation from the ASR decoder, so as to effectively combine the acoustic and text information from the ASR system to provide a richer context for the SLU task. This combination utilizes the text information generated by the ASR system and the acoustic features of the original sound signal, which is conducive to improving the performance of the model in processing semantic understanding tasks, especially in complex situations such as noise or dialects in the speech signal, and can further improve the accuracy of spoken semantic understanding.

[0060] Compared with the problem that subtle semantic differences expressed only through acoustic signals may be lost in traditional text transcription methods, the embodiments of the present invention can not only use text transcription of speech to understand the user's intention, but also capture subtle semantic differences (such as intonation, pauses, etc.) that may be expressed only through acoustic signals, which is conducive to improving the accuracy of spoken semantic understanding.

[0061] In the embodiment of the present invention, the combination of the acoustic representation and the text representation can be expressed as: (8); in, can represent the trained acoustic representation obtained from the ASR encoder, It can represent the text representation of the training speech transcription generated by the ASR decoder. The Multi-Head Attention mechanism (MHA) can be used As a query, As key (key) and (value).

[0062] The joint representation obtained by the fusion module can be expressed as: (9); Joint Characterization It is a combination of acoustic and text representations integrated through linear layer processing, which can be used to input the SLU decoder at the next time step to predict the intent or slot value.

[0063] The SLU decoder can be used to directly predict semantic information, such as intent and slot value, from the speech signal input. The embodiment of the present invention can avoid the error propagation problem that occurs in the traditional two-stage method (ASR+NLU) through an end-to-end method. The SLU decoder can directly learn how to map to semantic labels from the joint representation that combines the acoustic information of the ASR encoder and the text information of the ASR decoder.

[0064] The output of the SLU decoder module can be expressed as: (10); in, It can be expressed in a given joint representation And the previous SLU decoding output label In this case, the decoder output.

[0065] Step 430, calculating the overall multi-task learning loss based on the training semantic information, the training speech transcription text representation and the adjustment parameters.

[0066] In an embodiment of the present invention, a combined loss function may be used to implement parameter optimization of a multi-task learning method. The loss function may be expressed as follows: (11); in, is the overall multi-task learning loss, can represent the ASR decoder output, can represent SLU decoded output, It is a tuning parameter that can be used to balance the loss weights of ASR and SLU tasks.

[0067] Step 440: Adjust model parameters according to the overall multi-task learning loss to obtain a trained spoken language comprehension model.

[0068] In the embodiment of the present invention, the model parameters can be adjusted according to the overall multi-task learning loss until the training end condition is met to obtain a trained spoken language comprehension model.

[0069] Based on any of the above embodiments, the fusion module in the oral comprehension model to be trained includes: a first cross-attention layer, used to perform cross-attention processing on the training acoustic representation and the training speech transcription text representation to obtain a first training joint representation; a first Dropout layer, used to randomly discard the neural network units in the fusion module to obtain a second training joint representation output after the retained neural network units process the first training joint representation; a first feedforward layer, used to perform linear processing on the second training joint representation to obtain a third training joint representation; and a first adapter layer, used to adjust the third training joint representation to obtain a training joint representation.

[0070] Figure 6 is a training diagram of the fusion module provided in an embodiment of the present invention. Figure 6 , the training acoustic representation output by the ASR encoder and the training speech transcription text representation output by the ASR decoder can be input into the first cross attention layer in the fusion module. In the first cross attention layer, the cross attention mechanism can be used to perform cross attention calculation between the acoustic representation and the text representation to generate the first training joint representation that combines the two different modal information of acoustic information and speech transcription text information.

[0071] The first Dropout layer in the fusion module can randomly discard the neural network units in the fusion module, process the first training joint representation through the retained neural network units, and output the second training joint representation.

[0072] The first feedforward layer in the fusion module can perform a linear transformation on the first training joint representation through at least one fully connected layer, and perform a nonlinear activation process through a nonlinear activation function, thereby obtaining a third training joint representation obtained after the second training joint representation is subjected to a linear transformation and a nonlinear activation process. The first adapter layer in the fusion module can adjust the third training joint representation to obtain a training joint representation.

[0073] By adding an adapter layer to the fusion module, the embodiment of the present invention can make minor adjustments by updating or adding only a small number of parameters without completely retraining the pre-trained model, thereby improving the adjustment efficiency and effect of the model on specific tasks or fields, thereby significantly reducing the required computing resources and training time while maintaining or improving model performance.

[0074] By introducing the Dropout layer in the fusion module, the embodiment of the present invention can randomly discard some features (i.e., set their values ​​to zero) during the training process, so that the model can learn on different feature subsets, thereby reducing overfitting, enhancing the generalization ability of the model, and making its performance on new data more robust.

[0075] Based on any of the above embodiments, the spoken language comprehension decoder in the spoken language comprehension model to be trained includes: a self-attention layer, used to perform self-attention processing on the previous training semantic token output by the spoken language comprehension decoder to obtain the previous training semantic representation; a second cross-attention layer, used to perform cross-attention processing on the previous training semantic representation and the training joint representation to obtain a first training semantic representation; a second Dropout layer, used to randomly discard the neural network units in the spoken language comprehension decoder to obtain a second training semantic representation output after the retained neural network units process the first training semantics; a second feedforward layer, used to perform linear processing on the second training semantic representation to obtain a third training semantic representation; a second adapter layer, used to adjust the third training semantic representation to obtain a training semantic representation.

[0076] Figure 7 Schematic diagram of the training of the SLU decoder provided by the embodiment of the present invention. Figure 7, the training semantic token output by the SLU decoder at the previous time step and the training joint representation output by the fusion module can be input to the SLU decoder for processing. In the self-attention layer, the self-attention mechanism can be used to process the input previous training semantic token and the training joint representation, so that the spoken language understanding model pays attention to other tokens in the sequence when processing each semantic token, thereby capturing global dependencies. In the second cross-attention layer, the cross-attention mechanism can be used to perform cross-attention calculations between the previous training semantic token and the training joint representation, and the previous semantic information can be integrated into the current joint information to obtain the first training semantic representation.

[0077] The second Dropout layer in the SLU decoder can randomly discard the neural network units in the SLU decoder, process the first training semantic representation through the retained neural network units, and output the second training semantic representation.

[0078] The second feedforward layer in the SLU decoder can perform a linear transformation on the second training semantic representation through at least one fully connected layer, and perform a nonlinear activation process through a nonlinear activation function, thereby obtaining a third training semantic representation obtained after the second training semantic representation is subjected to the linear transformation and nonlinear activation process. The second adapter layer in the SLU decoder can adjust the third training semantic representation to obtain a training semantic representation.

[0079] By adding an adapter layer in the SLU decoder, the embodiment of the present invention can make minor adjustments by updating or adding only a small number of parameters without completely retraining the pre-trained model, thereby improving the adjustment efficiency and effect of the model on specific tasks or fields, thereby significantly reducing the required computing resources and training time while maintaining or improving model performance.

[0080] By introducing a Dropout layer in the SLU decoder, the embodiment of the present invention can randomly discard some features (i.e., set their values ​​to zero) during the training process, so that the model can learn on different feature subsets, thereby reducing overfitting, enhancing the generalization ability of the model, and making its performance on new data more robust.

[0081] Based on any of the above embodiments, the adapter layer is specifically used to: obtain a reduced dimensionality feature representation based on the output result of the feedforward layer, the reduced dimensionality linear mapping matrix and the reduced dimensionality deviation term; obtain the output result of the adapter layer based on the output result of the feedforward layer, the features obtained by nonlinearly processing the reduced dimensionality feature representation through an activation function, the increased dimensionality linear mapping matrix and the increased dimensionality deviation term.

[0082] Specifically, the adapter layer can be expressed as: (12); in, is the output of the adapter layer module, is a process from the original dimension to a lower dimension The linear mapping matrix (i.e., the dimension-reduced linear mapping matrix), is the output of the previous layer (i.e., the output of the feedforward layer), is the corresponding bias term (i.e., dimensionality reduction bias term), ReLU is the activation function, is a lower dimension To original dimension The linear mapping matrix (i.e., the dimension-increasing linear mapping matrix), is the corresponding bias term (i.e., the dimension-increasing bias term).

[0083] The entire process is optimized under a multi-task learning framework, taking into account the performance of both ASR and SLU, enabling the model to effectively perform joint learning of semantics and transcription, thereby not only improving the accuracy of semantic understanding but also reducing the amount of data required for training and adapting to new tasks.

[0084] The embodiment of the present invention optimizes the performance of ASR and SLU simultaneously by adopting multi-task learning, and can take into account the needs of speech recognition and semantic understanding during the training process, thereby better balancing the performance between the two.

[0085] The spoken language understanding device provided by the present invention is described below. The spoken language understanding device described below and the spoken language understanding method described above can be referenced to each other.

[0086] Figure 8 Schematic diagram of the structure of the spoken language understanding device provided by the embodiment of the present invention. Figure 8 The embodiment of the present invention provides a spoken language understanding device, which may specifically include the following modules: A speech signal acquisition module 810, used to acquire a speech signal to be understood; The semantic information output module 820 is used to input the speech signal into a trained spoken language comprehension model to obtain the semantic information of the speech signal; wherein the spoken language comprehension model includes: an automatic speech recognition encoder, used to convert the acoustic features of the speech signal into an acoustic representation; an automatic speech recognition decoder, used to perform text transcription on the acoustic representation to obtain a speech transcription text representation; a fusion module, used to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation; a spoken language comprehension decoder, used to perform semantic prediction on the joint representation to obtain a semantic representation; an output layer, used to obtain the semantic information of the speech signal based on the semantic representation.

[0087] After acquiring a speech signal to be understood, the embodiment of the present invention inputs the speech signal into a trained spoken language understanding model, uses an automatic speech recognition encoder to convert the acoustic features of the speech signal into an acoustic representation, uses an automatic speech recognition decoder to perform text transcription on the acoustic representation to obtain a speech transcription text representation, uses a fusion module to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation, uses a spoken language understanding decoder to perform semantic prediction on the joint representation to obtain a semantic representation, and uses an output layer to obtain semantic information of the speech signal based on the semantic representation, so that through an end-to-end spoken language understanding model, semantic output can be directly predicted from speech input, thereby effectively reducing the error propagation problem and improving the accuracy of the semantic information of the speech signal output by the spoken language understanding model.

[0088] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the spoken language understanding method, which includes: obtaining a speech signal to be understood; inputting the speech signal into a trained spoken language understanding model to obtain semantic information of the speech signal; wherein the spoken language understanding model includes: an automatic speech recognition encoder, which is used to convert the acoustic features of the speech signal into an acoustic representation; an automatic speech recognition decoder, which is used to perform text transcription on the acoustic representation to obtain a speech transcription text representation; a fusion module, which is used to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation; a spoken language understanding decoder, which is used to perform semantic prediction on the joint representation to obtain a semantic representation; and an output layer, which is used to obtain the semantic information of the speech signal based on the semantic representation.

[0089] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the spoken language understanding method provided by the above-mentioned methods, which includes: obtaining a speech signal to be understood; inputting the speech signal into a trained spoken language understanding model to obtain semantic information of the speech signal; wherein the spoken language understanding model includes: an automatic speech recognition encoder, which is used to convert the acoustic features of the speech signal into an acoustic representation; an automatic speech recognition decoder, which is used to perform text transcription on the acoustic representation to obtain a speech transcription text representation; a fusion module, which is used to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation; a spoken language understanding decoder, which is used to perform semantic prediction on the joint representation to obtain a semantic representation; and an output layer, which is used to obtain the semantic information of the speech signal based on the semantic representation.

[0091] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the spoken language understanding method provided by the above-mentioned methods, the method comprising: obtaining a speech signal to be understood; inputting the speech signal into a trained spoken language understanding model to obtain semantic information of the speech signal; wherein the spoken language understanding model comprises: an automatic speech recognition encoder for converting the acoustic features of the speech signal into an acoustic representation; an automatic speech recognition decoder for performing text transcription on the acoustic representation to obtain a speech transcription text representation; a fusion module for performing fusion on the acoustic representation and the speech transcription text representation to obtain a joint representation; a spoken language understanding decoder for performing semantic prediction on the joint representation to obtain a semantic representation; and an output layer for obtaining the semantic information of the speech signal based on the semantic representation.

[0092] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for spoken language comprehension, characterized in that: include: Acquire a speech signal to be understood; The speech signal is input into a trained spoken language comprehension model to obtain semantic information of the speech signal; wherein the spoken language comprehension model comprises: an automatic speech recognition encoder, used to convert the acoustic features of the speech signal into an acoustic representation; an automatic speech recognition decoder, used to perform text transcription on the acoustic representation to obtain a speech transcription text representation; a fusion module, used to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation; a spoken language comprehension decoder, used to perform semantic prediction on the joint representation to obtain a semantic representation; and an output layer, used to obtain the semantic information of the speech signal based on the semantic representation.

2. The spoken language comprehension method according to claim 1, characterized in that: The fusion module comprises: A first cross-attention layer, used for performing cross-attention processing on the acoustic representation and the speech transcription text representation to obtain a first joint representation; A first feed-forward layer, configured to perform linear processing on the first joint representation to obtain a second joint representation; The first adapter layer is used to adjust the second joint representation to obtain a joint representation.

3. The spoken language comprehension method according to claim 1, characterized in that: The spoken language understanding decoder comprises: The self-attention layer is used to perform self-attention processing on the previous semantic token output by the spoken language understanding decoder to obtain the previous semantic representation; A second cross attention layer is used to perform cross attention processing on the previous semantic representation and the joint representation to obtain a first semantic representation; A second feed-forward layer, used for performing linear processing on the first semantic representation to obtain a second semantic representation; The second adapter layer is used to adjust the second semantic representation to obtain a semantic representation.

4. The spoken language comprehension method according to claim 1, characterized in that: The spoken language comprehension model is trained in the following way: Acquire a sample speech signal; The sample speech signal is input into the spoken language understanding model to be trained to obtain the training semantic information of the sample speech signal; wherein the spoken language understanding model to be trained comprises: an automatic speech recognition encoder, which is used to convert the acoustic features of the sample speech signal into a training acoustic representation; an automatic speech recognition decoder, which is used to perform text transcription on the training acoustic representation to obtain a training speech transcription text representation; a fusion module, which is used to perform feature fusion on the training acoustic representation and the training speech transcription text representation to obtain a training joint representation; a spoken language understanding decoder, which is used to perform semantic prediction on the training joint representation to obtain a training semantic representation; an output layer, which is used to obtain the training semantic information of the sample speech signal based on the training semantic representation; Calculating the overall multi-task learning loss based on the training semantic information, the training speech transcription text representation and the adjustment parameters; The model parameters are adjusted according to the overall multi-task learning loss to obtain a trained spoken language comprehension model.

5. The spoken language comprehension method according to claim 4, characterized in that: The fusion module in the spoken language comprehension model to be trained includes: A first cross-attention layer, used for performing cross-attention processing on the training acoustic representation and the training speech transcription text representation to obtain a first training joint representation; A first Dropout layer is used to randomly discard the neural network units in the fusion module to obtain a second training joint representation output after the retained neural network units process the first training joint representation; a first feed-forward layer, configured to perform linear processing on the second training joint representation to obtain a third training joint representation; The first adapter layer is used to adjust the third training joint representation to obtain a training joint representation.

6. The spoken language comprehension method according to claim 4, characterized in that: The spoken language comprehension decoder in the spoken language comprehension model to be trained comprises: The self-attention layer is used to perform self-attention processing on the previous training semantic token output by the spoken language understanding decoder to obtain the previous training semantic representation; A second cross attention layer is used to perform cross attention processing on the previous training semantic representation and the training joint representation to obtain a first training semantic representation; A second Dropout layer is used to randomly discard the neural network units in the spoken language understanding decoder to obtain a second training semantic representation output by the retained neural network units after processing the first training semantic representation; a second feed-forward layer, configured to perform linear processing on the second training semantic representation to obtain a third training semantic representation; The second adapter layer is used to adjust the third training semantic representation to obtain a training semantic representation.

7. The spoken language comprehension method according to any one of claims 1 to 6, characterized in that: The adapter layer is specifically used for: Based on the output result of the feedforward layer, the dimension reduction linear mapping matrix and the dimension reduction bias term, the dimension reduction feature representation is obtained; The output result of the adapter layer is obtained based on the output result of the feedforward layer, the features obtained by performing nonlinear processing on the reduced-dimensional feature representation through an activation function, the increased-dimensional linear mapping matrix and the increased-dimensional bias term.

8. A spoken language comprehension device, characterized in that: include: A speech signal acquisition module, used to acquire a speech signal to be understood; A semantic information output module is used to input the speech signal into a trained spoken language comprehension model to obtain the semantic information of the speech signal; wherein the spoken language comprehension model includes: an automatic speech recognition encoder, used to convert the acoustic features of the speech signal into an acoustic representation; an automatic speech recognition decoder, used to perform text transcription on the acoustic representation to obtain a speech transcription text representation; a fusion module, used to fuse the acoustic representation and the speech transcription text representation to obtain a joint representation; a spoken language comprehension decoder, used to perform semantic prediction on the joint representation to obtain a semantic representation; and an output layer, used to obtain the semantic information of the speech signal based on the semantic representation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the spoken language understanding method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the spoken language understanding method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the spoken language understanding method according to any one of claims 1 to 7 is implemented.

Images