Model training methods, speech intent recognition methods, devices, equipment and media

Abstract

This disclosure provides a training method, apparatus, electronic device, and medium for a speech intent recognition model, which can be applied to the field of artificial intelligence technology. The method includes: inputting audio samples into a speech intent recognition model to obtain audio features; inputting text samples into a text intent recognition model to obtain text features, wherein the text samples are obtained based on the audio samples, and the speech intent recognition model is obtained based on the text intent recognition model; obtaining a first distillation loss value and a second distillation loss value based on the audio features and text features, wherein the first distillation loss value is used to characterize the byte alignment gap between the audio features and text features, and the second distillation loss value is used to characterize the semantic alignment gap between the audio features and text features; and training the speech intent recognition model using the first distillation loss value and the second distillation loss value to obtain a trained speech intent recognition model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a method for training a model, a method for recognizing speech intent, an apparatus, an electronic device, a storage medium, and a program product. Background Technology

[0002] Currently, voice intelligence systems have been applied to various intelligent tasks, and many voice intelligence systems have begun to use voice for human-computer interaction. Voice intelligence systems directly identify user intent information from audio information through end-to-end voice intent recognition models.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problems in the related technology: the intent recognition results of the end-to-end voice intent recognition model are not accurate enough and cannot meet user needs. Summary of the Invention

[0004] In view of this, the present disclosure provides a method for training a model, a method for recognizing speech intent, an apparatus, an electronic device, a storage medium, and a program product.

[0005] One aspect of this disclosure provides a method for training a speech intent recognition model, comprising: inputting audio samples into the speech intent recognition model to obtain audio features; inputting text samples into the text intent recognition model to obtain text features, wherein the text samples are obtained based on the audio samples, and the speech intent recognition model is obtained based on the text intent recognition model; obtaining a first distillation loss value and a second distillation loss value based on the audio features and the text features, wherein the first distillation loss value is used to characterize the byte alignment gap between the audio features and the text features, and the second distillation loss value is used to characterize the semantic alignment gap between the audio features and the text features; and training the speech intent recognition model using the first distillation loss value and the second distillation loss value to obtain a trained speech intent recognition model.

[0006] According to an embodiment of this disclosure, obtaining the first distillation loss value based on the audio features and the text features includes: obtaining the first distillation loss value based on the audio features, the text features, and a first mutual attention matrix, wherein the audio features are the output results of the network layer of the speech intent recognition model, the text features are the output results of the network layer of the text intent recognition model, and the first mutual attention matrix is ​​an intermediate feature used to align the text features and the audio features byte by byte.

[0007] According to embodiments of this disclosure, when the network layer of the above-mentioned speech intent recognition model includes an attention mechanism, the training method of the above-mentioned speech intent recognition model further includes: inputting the above-mentioned audio samples into the above-mentioned speech intent recognition model to obtain audio intermediate features, wherein the above-mentioned audio intermediate features are the output results of the attention mechanism in the network layer of the above-mentioned speech intent recognition model; inputting the above-mentioned text samples into the above-mentioned text intent recognition model to obtain text intermediate features, wherein the above-mentioned text intermediate features are the output results of the attention mechanism in the network layer of the above-mentioned text intent recognition model; and obtaining a third distillation loss value based on the above-mentioned text intermediate features and the above-mentioned audio intermediate features, so as to train the above-mentioned speech intent recognition model using the above-mentioned first distillation loss value, the above-mentioned second distillation loss value and the above-mentioned third distillation loss value.

[0008] According to an embodiment of this disclosure, obtaining the third distillation loss value based on the above-mentioned text intermediate features and the above-mentioned audio intermediate features includes: obtaining the above-mentioned third distillation loss value based on the above-mentioned text intermediate features, the above-mentioned audio intermediate features, and a second mutual attention matrix, wherein the above-mentioned second mutual attention matrix is ​​an intermediate feature used to align the above-mentioned text intermediate features and the above-mentioned audio intermediate features by bytes.

[0009] According to embodiments of this disclosure, the audio feature data includes the aforementioned audio features or the aforementioned intermediate audio features, and the text feature data includes the aforementioned text features or the aforementioned intermediate text features. The audio feature data includes multiple components, and the text feature data includes multiple components. The training method for the aforementioned speech intent recognition model further includes: for each of the multiple audio feature data, determining target text feature data matching the audio feature data from the multiple text feature data based on a predetermined mapping relationship, wherein the predetermined mapping relationship is determined based on the mapping relationship between the network layers of the text intent recognition model and the network layers of the speech intent recognition model; and obtaining a mutual attention matrix matching the audio feature data based on the audio feature data and the target text feature data, wherein the mutual attention matrix includes the aforementioned first mutual attention matrix or the aforementioned second mutual attention matrix.

[0010] According to embodiments of this disclosure, obtaining a mutual attention matrix matching the audio feature data based on the audio feature data and the target text feature data includes: transposing the audio feature data to obtain transposed audio feature data; obtaining initial interaction feature data based on the target text feature data and the transposed audio feature data; convolving the initial interaction feature data to obtain convolved feature data; and activating the convolved feature data to obtain a mutual attention matrix matching the audio feature data.

[0011] According to embodiments of this disclosure, the aforementioned audio features include multiple features, the aforementioned text features include multiple features, and the aforementioned method of obtaining a first distillation loss value based on the aforementioned audio features, the aforementioned text features, and a first mutual attention matrix includes: for each of the multiple aforementioned audio features, obtaining an aligned audio feature that is aligned with a target text feature at the byte level based on the first mutual attention matrix that matches the aforementioned audio feature and the aforementioned audio feature, wherein the aforementioned target text feature is a text feature that has the aforementioned predetermined mapping relationship with the aforementioned audio feature; and obtaining the aforementioned first distillation loss value based on the multiple aforementioned target text features and the multiple aforementioned aligned audio features.

[0012] According to embodiments of this disclosure, the aforementioned audio intermediate features include multiple components, the aforementioned text intermediate features include multiple components, and the aforementioned method of obtaining the third distillation loss value based on the aforementioned text intermediate features, the aforementioned audio intermediate features, and the second mutual attention matrix includes: for each of the aforementioned audio intermediate features, determining a target text intermediate feature that matches the aforementioned audio intermediate feature from the aforementioned text intermediate features based on the aforementioned predetermined mapping relationship; obtaining an aligned audio intermediate feature that is aligned with the aforementioned target text intermediate feature at the byte level based on the aforementioned second mutual attention matrix, the transpose of the aforementioned second mutual attention matrix, and the aforementioned audio intermediate feature; and obtaining the aforementioned third distillation loss value based on the aforementioned target text intermediate features and the aforementioned aligned audio intermediate features.

[0013] According to embodiments of this disclosure, the audio samples include multiple samples, and the text samples include multiple samples. A first text sample obtained based on the audio samples is a positive sample pair with the audio samples, and a second text sample (excluding the first text sample) among the multiple text samples is a negative sample pair with the audio samples. Obtaining a second distillation loss value based on the audio features and the text features includes: for each audio sample among the multiple audio samples, for each audio feature among the audio samples, determining semantic audio features from the audio features; determining a first target text feature matching the audio features and a second target text feature matching the audio features, wherein the first target text feature is obtained based on a predetermined mapping relationship. The second target text feature obtained from the first text sample is based on the predetermined mapping relationship determined by the mapping relationship between the network layer of the text intent recognition model and the network layer of the speech intent recognition model. A first semantic target text feature is determined from the first target text feature, and a second semantic target text feature is determined from the second target text feature. A first semantic similarity is determined based on the semantic audio feature and the first semantic target text feature. A second semantic similarity is determined based on the semantic audio feature and the second semantic target text feature. Finally, a second distillation loss value is obtained based on multiple first semantic similarities and multiple second semantic similarities.

[0014] According to embodiments of this disclosure, the method further includes: inputting the audio sample into the speech intent recognition model to obtain an audio intent result; inputting the text sample into the text intent recognition model to obtain a text intent result; and obtaining a task loss value based on the audio intent result and the text intent result, so as to train the speech intent recognition model using the first distillation loss value, the second distillation loss value, and the task loss value.

[0015] Another aspect of this disclosure provides a voice intent recognition method, comprising: receiving voice information from a user; and inputting the voice information into a voice intent recognition model to obtain an intent recognition result, wherein the voice intent recognition model is obtained using the training method described above.

[0016] Another aspect of this disclosure provides a training apparatus for a speech intent recognition model, comprising: an audio input module for inputting audio samples into the speech intent recognition model to obtain audio features; a text input module for inputting text samples into a text intent recognition model to obtain text features, wherein the text samples are obtained based on the audio samples, and the speech intent recognition model is obtained based on the text intent recognition model; a loss determination module for obtaining a first distillation loss value and a second distillation loss value based on the audio features and the text features, wherein the first distillation loss value is used to characterize the byte alignment gap between the audio features and the text features, and the second distillation loss value is used to characterize the semantic alignment gap between the audio features and the text features; and a training module for training the speech intent recognition model using the first distillation loss value and the second distillation loss value to obtain a trained speech intent recognition model.

[0017] Another aspect of this disclosure provides a voice intent recognition device, comprising: a receiving module for receiving voice information from a user; and a result determining module for inputting the voice information into a voice intent recognition model to obtain an intent recognition result, wherein the voice intent recognition model is obtained using the training method described above.

[0018] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the methods described above.

[0019] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the method described above.

[0020] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, implement the method described above.

[0021] According to embodiments of this disclosure, the speech intent recognition model is obtained based on the text intent recognition model. It utilizes distillation technology to distill the parameters of the text intent recognition model onto the speech intent recognition model, making the training of the speech intent recognition model more efficient and accurate. In addition, by employing a technique that uses audio features and text features to obtain a first distillation loss value and a second distillation loss value, the audio features and text features are aligned at the semantic level and the byte level, respectively. Therefore, it at least partially overcomes the technical problem of poor alignment effect due to the inherent differences between text information and speech information, making the intent recognition result of the trained speech intent recognition model accurate. Attached Figure Description

[0022] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0023] Figure 1 This illustration schematically shows an exemplary system architecture of a training method and apparatus for applying a speech intent recognition model according to embodiments of the present disclosure;

[0024] Figure 2 A flowchart illustrating a method for training a speech intent recognition model according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 3 This illustration schematically shows a model training structure diagram of a speech intent recognition model according to an embodiment of the present disclosure;

[0026] Figure 4 A flowchart illustrating a speech intent recognition method according to an embodiment of the present disclosure is shown schematically.

[0027] Figure 5 A block diagram of a training apparatus for a speech intent recognition model according to an embodiment of the present disclosure is shown schematically.

[0028] Figure 6 A block diagram of a voice intent recognition device according to an embodiment of the present disclosure is shown schematically;

[0029] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a training method for a speech intent recognition model according to embodiments of the present disclosure. Detailed Implementation

[0030] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0032] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0033] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0034] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0035] Voice intelligence systems have been applied to various intelligent tasks, and many intelligent systems have begun to use voice for human-computer interaction. Among these systems, the pipeline structure for recognizing user intent in most systems is still user-ASR (Automatic Speech Recognition) system-NLP (Natural Language Processing) system.

[0036] However, this pipeline-style system has four limitations. First, the optimization of each system in the pipeline is carried out independently, which is time-consuming and labor-intensive. Second, due to environmental interference, the ASR recognition result is inaccurate, and the recognition error of the ASR system will be transmitted to the downstream system, causing error propagation and resulting in incorrect intent recognition, which in turn leads to inaccurate intent recognition results in NLP. Third, for the final intent recognition, not every word / word is equally important; if some keywords / words are processed correctly, the correct intent can be obtained. Fourth, some information is lost when the voice information passes through the ASR system, preventing the downstream from using richer voice information.

[0037] Therefore, related technologies have proposed an end-to-end intent recognition model that directly receives user audio information from the NLP system. This model directly identifies user intent information from the audio data, omitting the intermediate ASR system and eliminating error propagation. Furthermore, in pipeline architectures, the NLP receives only the recognized text information from the ASR system; much of the user's audio information, such as tone of voice and speech rate, is lost after passing through the ASR system. This information still provides some assistance in determining user intent. Therefore, an end-to-end intent recognition system can utilize more audio information to improve the accuracy of intent recognition.

[0038] Within the framework of end-to-end speech intent recognition, related technologies employ a two-stage model training. The first stage trains an ASR model, converting user audio information into representation vectors and then recognizing the user's spoken text. The second stage discards the objective of the first stage, retaining only the steps prior to converting to representation vectors, and changes the objective to recognizing the user's intent. Some methods also utilize triplet information, training a text intent recognition model using transcription-intent and a speech intent recognition model using speech-intent, finally aligning the predictions of both models using a loss function.

[0039] However, in two-stage model training, the gap between the task and the final goal during the first stage may result in the learned features not being significant enough for intent recognition. The second stage training cannot fully correct the model's direction towards intent recognition. Therefore, many current end-to-end speech intent recognition methods are based on speech-transcription-intent triples, using alignment or distillation methods to bring text intent recognition and speech intent recognition models closer together. Related techniques directly train two models and align their predictions using a loss function in the final prediction stage. However, simple distance loss only improves single-point pairs in the two representation spaces without utilizing the relationships between these pairs. Due to the inherent differences between text and speech information, alignment using simple distance loss is ineffective. Furthermore, many existing alignment methods ignore explicit alignment between audio frames and text tokens (words). Since understanding user intent often depends on certain keywords / words, this fine-grained cross-modal alignment (audio frames and text tokens) is crucial for end-to-end speech intent recognition models to perform intent recognition.

[0040] In view of the above, embodiments of this disclosure provide a training method for a speech intent recognition model, a training device for a speech intent recognition model, an electronic device, a readable storage medium, and a computer program product. The training method for the speech intent recognition model includes: inputting audio samples into a speech intent recognition model to obtain audio features; inputting text samples into a text intent recognition model to obtain text features, wherein the text samples are obtained based on the audio samples, and the speech intent recognition model is obtained based on the text intent recognition model; obtaining a first distillation loss value and a second distillation loss value based on the audio features and text features, wherein the first distillation loss value is used to characterize the byte alignment gap between the audio features and the text features, and the second distillation loss value is used to characterize the semantic alignment gap between the audio features and the text features; and training the speech intent recognition model using the first distillation loss value and the second distillation loss value to obtain a trained speech intent recognition model.

[0041] Figure 1 This illustration schematically depicts an exemplary system architecture for training methods and apparatus for voice intent recognition models according to embodiments of the present disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0042] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0043] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0044] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0045] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0046] It should be noted that the training method and voice intent recognition method of the voice intent recognition model provided in this embodiment can generally be executed by server 105. Correspondingly, the training device and voice intent recognition device of the voice intent recognition model provided in this embodiment can generally be located in server 105. The training method and voice intent recognition method of the voice intent recognition model provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the training device and voice intent recognition device of the voice intent recognition model provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Alternatively, the training method and voice intent recognition method of the voice intent recognition model provided in this embodiment can also be executed by terminal devices 101, 102, or 103, or by other terminal devices different from terminal devices 101, 102, or 103. Accordingly, the training device and the voice intent recognition device of the voice intent recognition model provided in this embodiment of the present disclosure may also be set in terminal device 101, 102 or 103, or in other terminal devices different from terminal device 101, 102 or 103.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0048] Figure 2 A flowchart illustrating a training method for a speech intent recognition model according to an embodiment of the present disclosure is shown.

[0049] like Figure 2 As shown, the method includes operations S201 to S204.

[0050] In operation S201, audio samples are input into the speech intent recognition model to obtain audio features.

[0051] In operation S202, the text sample is input into the text intent recognition model to obtain text features. The text sample is obtained based on the audio sample, and the speech intent recognition model is obtained based on the text intent recognition model.

[0052] In operation S203, based on audio features and text features, a first distillation loss value and a second distillation loss value are obtained. The first distillation loss value is used to characterize the byte alignment gap between audio features and text features, and the second distillation loss value is used to characterize the semantic alignment gap between audio features and text features.

[0053] In operation S204, the speech intent recognition model is trained using the first distillation loss value and the second distillation loss value to obtain the trained speech intent recognition model.

[0054] According to embodiments of this disclosure, the audio sample is plain audio input by the user, for example, obtaining the audio information of "Please call Xiao A". The text sample is correct plain text, generated by manually listening to and transcribing the audio sample. The text intent recognition model is trained using text samples and corresponding intent labels. Using distillation learning techniques, the parameters of the text intent recognition model are distilled onto the speech intent recognition model to train and optimize the speech intent recognition model.

[0055] According to embodiments of this disclosure, both the text intent recognition model and the speech intent recognition model can be based on the BERT (Bidirectional Encoder Representation from Transformers) model, but are not limited to it; other deep learning models can also be used. The text intent recognition model has higher complexity, potentially a 12-layer BERT. The speech intent recognition model has lower complexity, potentially a 4-layer BERT. During distillation, the third layer of the text intent recognition model can be mapped to the first layer of the speech intent recognition model, the sixth layer to the second layer, and so on. Furthermore, the text intent recognition model is not optimized during distillation; only the speech intent recognition model is optimized.

[0056] According to embodiments of this disclosure, audio samples are input into a speech intent recognition model to obtain audio features. Text samples are input into a text intent recognition model to obtain text features. A first distillation loss value is obtained based on the audio features and text features. The first distillation loss value characterizes the byte alignment gap between the audio features and text features. The byte alignment gap is the difference between the audio frame and the text token. A second distillation loss value is obtained based on the audio features and text features. The second distillation loss value characterizes the semantic alignment gap between the audio features and text features. The semantic alignment gap is the difference between the audio features and text features in the overall representation dimension.

[0057] According to embodiments of this disclosure, during the training process of the model, based on the architecture of distilling a text intent recognition model into a speech intent recognition model, the speech intent recognition model is trained using a first distillation loss value and a second distillation loss value, so that the speech intent recognition model can learn important character / word features and overall semantic features, thereby obtaining a more accurate speech intent recognition model.

[0058] According to embodiments of this disclosure, the speech intent recognition model is obtained based on the text intent recognition model. It utilizes distillation technology to distill the parameters of the text intent recognition model onto the speech intent recognition model, making the training of the speech intent recognition model more efficient and accurate. In addition, by employing a technique that uses audio features and text features to obtain a first distillation loss value and a second distillation loss value, the audio features and text features are aligned at the semantic level and the byte level, respectively. Therefore, it at least partially overcomes the technical problem of poor alignment effect due to the inherent differences between text information and speech information, making the intent recognition result of the trained speech intent recognition model accurate.

[0059] According to embodiments of this disclosure, obtaining a first distillation loss value based on audio features and text features may include the following operations:

[0060] Based on audio features, text features, and a first mutual attention matrix, a first distillation loss value is obtained, where audio features are the output of the network layer of the speech intent recognition model, text features are the output of the network layer of the text intent recognition model, and the first mutual attention matrix is ​​an intermediate feature used to align text features and audio features byte by byte.

[0061] According to embodiments of this disclosure, there are inherent differences between the representations of text and audio information, resulting in the audio representation being longer than the text representation. For example, the audio of a sentence is 4 seconds long, while the text is 10 characters long. Text information uses individual characters as tokens, so its representation length is 10. Audio information uses audio frames as tokens; if a frame is calculated as 20ms, the audio representation length is 200. Directly applying a loss function between these two representations with such large length differences is not feasible, and simply padding the text tokens into the same length of speech is not optimal, as it causes the audio frame representations to align with meaningless text representations. Therefore, a first mutual attention matrix is ​​introduced to capture fine-grained interactions between text tokens and audio frames, enabling the speech intent recognition model to focus on the more important audio frames.

[0062] According to embodiments of this disclosure, a first distillation loss value is obtained based on audio features, text features, and a first mutual attention matrix. By training the speech intent recognition model based on the first distillation loss value, the byte alignment gap between audio features and text features is effectively reduced, making the speech intent recognition model more accurate in its predictions.

[0063] According to embodiments of this disclosure, when an attention mechanism is included in the network layer of the speech intent recognition model, the training method for the speech intent recognition model may further include the following steps:

[0064] Audio samples are input into the speech intent recognition model to obtain audio intermediate features, which are the output of the attention mechanism in the network layer of the speech intent recognition model. Text samples are input into the text intent recognition model to obtain text intermediate features, which are the output of the attention mechanism in the network layer of the text intent recognition model. Based on the text intermediate features and audio intermediate features, a third distillation loss value is obtained so that the speech intent recognition model can be trained using the first distillation loss value, the second distillation loss value, and the third distillation loss value.

[0065] According to embodiments of this disclosure, a third distillation loss value is obtained based on text intermediate features and audio intermediate features. This third distillation loss value represents the loss between the outputs of the attention mechanisms in the network layers of the speech intent recognition model and the text intent recognition model. The first distillation loss value represents the loss value corresponding to each network layer in the speech intent recognition model and the text intent recognition model. Both the first and third distillation loss values ​​are used to characterize the byte alignment gap between audio features and text features. Training the speech intent recognition model using both the first and third distillation loss values ​​simultaneously results in better byte alignment and higher model training accuracy compared to training using only one of them.

[0066] According to embodiments of this disclosure, obtaining a third distillation loss value based on text intermediate features and audio intermediate features may include the following operations:

[0067] The third distillation loss value is obtained based on the text intermediate features, audio intermediate features, and the second mutual attention matrix, where the second mutual attention matrix is ​​used to align the text intermediate features and audio intermediate features byte by byte.

[0068] According to embodiments of this disclosure, a second mutual attention matrix is ​​determined based on text intermediate features and audio intermediate features. A third distillation loss value is obtained based on the audio intermediate features, text intermediate features, and the second mutual attention matrix. Training the speech intent recognition model using this third distillation loss value effectively reduces the byte alignment gap between audio and text features, resulting in more accurate predictions from the speech intent recognition model.

[0069] According to embodiments of this disclosure, audio feature data includes audio features or audio intermediate features, text feature data includes text features or text intermediate features, audio feature data includes multiple features, and text feature data includes multiple features.

[0070] The training method for a speech intent recognition model may also include the following operations:

[0071] For each audio feature data in multiple audio feature data, a target text feature data matching the audio feature data is determined from multiple text feature data based on a predetermined mapping relationship, wherein the predetermined mapping relationship is determined based on the mapping relationship between the network layers of the text intent recognition model and the network layers of the speech intent recognition model; and based on the audio feature data and the target text feature data, a mutual attention matrix matching the audio feature data is obtained, wherein the mutual attention matrix includes a first mutual attention matrix or a second mutual attention matrix.

[0072] According to embodiments of this disclosure, target text feature data matching audio feature data is determined from multiple text feature data based on a predetermined mapping relationship. Specifically, the predetermined mapping relationship between the text intent recognition model and the speech intent recognition model can be determined as follows: the 3nth network layer of the text intent recognition model is mapped to the nth network layer of the speech intent recognition model, where n is an integer greater than 0. For example, there may be a mapping between the 3rd network layer of the text intent recognition model and the 1st network layer of the speech intent recognition model, or a mapping between the 6th network layer of the text intent recognition model and the 2nd network layer of the speech intent recognition model. However, this is not a limitation. Any predetermined mapping relationship set based on the correlation between the network layers of the text intent recognition model and the speech intent recognition model during the distillation process is acceptable.

[0073] According to embodiments of this disclosure, obtaining a mutual attention matrix that matches the audio feature data based on audio feature data and target text feature data may include the following operations:

[0074] The audio feature data is transposed to obtain transposed audio feature data; based on the target text feature data and the transposed audio feature data, initial interaction feature data is obtained; the initial interaction feature data is convolved to obtain convolved feature data; and the convolved feature data is activated to obtain a mutual attention matrix that matches the audio feature data.

[0075] According to embodiments of this disclosure, both audio feature data and target text feature data are represented in vector form. The audio feature data includes audio features or intermediate audio features. The target text feature data includes target text features or intermediate target text features. The transposed audio feature data and target text feature data are multiplied together to obtain initial interaction feature data. A convolution kernel is then used to convolve the interaction feature data once to obtain convolved feature data. Finally, the convolved feature data is activated to obtain a mutual attention matrix that matches the audio feature data. The mutual attention matrix includes a first mutual attention matrix and a second mutual attention matrix.

[0076] According to an embodiment of this disclosure, in one implementation, initial interaction feature data is determined using audio features from audio feature data and target text features from target text feature data, and then a first mutual attention matrix is ​​calculated. The specific calculation formula is as follows:

[0077]

[0078] Among them, A ca Let represent the first mutual attention matrix, i represent the number of layers in the speech intent recognition model, and g(i) represent the number of layers in the text intent recognition model that have a mapping relationship with the number of layers in the speech intent recognition model. This represents the target text features in the target text feature data. This represents the audio features in the audio feature data. express The transpose of , where K represents the convolution kernel.

[0079] According to an embodiment of this disclosure, in another implementation, initial interaction feature data is determined using intermediate audio features from the audio feature data and intermediate target text features from the target text feature data, and then a second mutual attention matrix is ​​calculated. The specific calculation formula is as follows:

[0080]

[0081] Among them, A c Let represent the second mutual attention matrix, and i, n represent the nth attention head in the i-th layer of the speech intent recognition model. Let g(i) represent the intermediate features of the target text in the target text feature data, and g(i) and n represent the corresponding layer numbers in the text intent recognition model that have a mapping relationship with i and n. This represents the intermediate audio features in the audio feature data. express The transpose of .

[0082] According to embodiments of this disclosure, the audio features include multiple features, and the text features include multiple features. Based on the audio features, text features, and a first mutual attention matrix, a first distillation loss value is obtained, which may include the following operations:

[0083] For each of the multiple audio features, an aligned audio feature is obtained based on a first mutual attention matrix that matches the audio feature and the audio feature itself, which is aligned with the target text feature at the byte level, wherein the target text feature is a text feature that has a predetermined mapping relationship with the audio feature; and a first distillation loss value is obtained based on the multiple target text features and the multiple aligned audio features.

[0084] According to embodiments of this disclosure, for each of a plurality of audio features, an aligned audio feature is obtained that is aligned with the target text feature at the byte level, based on a first mutual attention matrix matching the audio feature and the audio feature itself. A first distillation loss value is obtained based on the plurality of target text features and the plurality of aligned audio features. The first distillation loss value is calculated as follows:

[0085]

[0086] Among them, L hid L represents the first distillation loss value. s f represents the number of layers in the speech intent recognition model. h This represents the loss value of the current network layer in the speech intent recognition model.

[0087] The loss value of the current network layer of the speech intent recognition model is calculated based on the mean squared error, and the calculation method is as follows:

[0088]

[0089] According to embodiments of this disclosure, the audio intermediate features include multiple features, and the text intermediate features include multiple features. A third distillation loss value is obtained based on the text intermediate features, the audio intermediate features, and the second mutual attention matrix, which may include the following operations:

[0090] For each of the multiple audio intermediate features, a target text intermediate feature matching the audio intermediate feature is determined from the multiple text intermediate features based on a predetermined mapping relationship; an aligned audio intermediate feature that is aligned with the target text intermediate feature at the byte level is obtained based on the second mutual attention matrix, the transpose of the second mutual attention matrix and the audio intermediate feature; and a third distillation loss value is obtained based on the multiple target text intermediate features and the multiple aligned audio intermediate features.

[0091] According to embodiments of this disclosure, a third distillation loss value is obtained based on multiple target text intermediate features and multiple aligned audio intermediate features. The calculation method for the third distillation loss value is as follows:

[0092]

[0093] Among them, L att denoted by , f represents the loss value of the third distillation, f represents the loss value of the network layer containing the attention matrix in the speech intent recognition model, and N represents the total number of attention heads in the speech intent recognition model.

[0094] In one implementation, aligned audio intermediate features, aligned byte-level with the target text intermediate features, can be obtained based on the second mutual attention matrix, its transpose, and the audio intermediate features. Specifically, the loss value corresponding to the network layer containing the attention matrix in the speech intent recognition model is calculated based on the mean squared error, as follows:

[0095]

[0096] Among them, f a Let g(i) represent the network layer containing the attention matrix in the speech intent recognition model, and g(i) and n represent the corresponding layer numbers in the text intent recognition model that have a mapping relationship with i and n. Represents the second mutual attention matrix A c The transpose of .

[0097] In another alternative embodiment, aligned audio intermediate features, aligned byte-level with the target text intermediate features, can be obtained based on the first mutual attention matrix, the transpose of the first mutual attention matrix, and the audio intermediate features. The loss value corresponding to the network layer containing the attention matrix in the speech intent recognition model is calculated based on the mean squared error, as follows:

[0098]

[0099] Among them, f aLet g(i) represent the network layer containing the attention matrix in the speech intent recognition model, and g(i) and n represent the corresponding layer numbers in the text intent recognition model that have a mapping relationship with i and n. Let A represent the first mutual attention matrix. ca The transpose of .

[0100] According to embodiments of this disclosure, there are multiple audio samples and multiple text samples. A first text sample obtained based on the audio samples is a positive sample pair with the audio sample, and a second text sample other than the first text sample among the multiple text samples is a negative sample pair with the audio sample.

[0101] The second distillation loss value is obtained based on audio and text features, and may include the following operations:

[0102] For each audio sample among multiple audio samples, for each audio feature in the audio sample, semantic audio features are determined from the audio features; a first target text feature matching the audio features and a second target text feature matching the audio features are determined, wherein the first target text feature is obtained from the first text sample based on a predetermined mapping relationship, and the second target text feature is obtained from the second text sample based on a predetermined mapping relationship, which is determined based on the mapping relationship between the network layers of the text intent recognition model and the network layers of the speech intent recognition model;

[0103] First semantic target text features are determined from first target text features, and second semantic target text features are determined from second target text features; first semantic similarity is determined based on semantic audio features and first semantic target text features; second semantic similarity is determined based on semantic audio features and second semantic target text features; and second distillation loss value is obtained based on multiple first semantic similarities and multiple second semantic similarities.

[0104] According to embodiments of this disclosure, audio samples include multiple audio samples, such as voice messages like "Please call A," "Please call B," and "Please play music." Text samples also include multiple text samples, such as text messages like "Please call A," "Please call B," and "Please play music." A first text sample obtained based on the audio sample forms a positive sample pair with the audio sample, and a second text sample (excluding the first text sample) among the multiple text samples forms a negative sample pair with the audio sample. For example, based on the voice message "Please call A," the first text sample obtained is the text message "Please call A," forming a positive sample pair. Based on the voice message "Please call A," the second text samples obtained are the text messages "Please call B" and "Please play music," with the audio sample forming a negative sample pair with each sample in the second sample. And so on, each audio sample corresponds to one positive sample pair and multiple negative sample pairs.

[0105] According to embodiments of this disclosure, an audio sample includes multiple audio features, each audio feature representing each frame in the audio sample. A text sample includes multiple text features, each text audio feature representing each word in the text sample. For each audio feature in the audio sample, a semantic audio feature is determined from the audio features, wherein the semantic audio feature is a feature that can represent the overall semantics of the audio sample.

[0106] According to embodiments of this disclosure, a first target text feature matching the audio feature and a second target text feature matching the audio feature are determined. For example, assuming the network layer containing the audio feature is the first layer of the speech intent recognition model, then based on a predetermined mapping relationship, the first target text feature and the second target text feature matching the audio feature are determined in the third layer of the text intent recognition model.

[0107] According to embodiments of this disclosure, a first semantic target text feature is determined from a first target text feature, wherein the first semantic target feature is a feature that can represent the overall semantics of the first target text feature. A second semantic target text feature is determined from a second target text feature, wherein the second semantic target feature is a feature that can represent the overall semantics of the second target text feature.

[0108] According to embodiments of this disclosure, a first semantic similarity is determined based on semantic audio features and a first semantic target text feature; a second semantic similarity is determined based on semantic audio features and a second semantic target text feature; and a second distillation loss value is obtained based on multiple first semantic similarities and multiple second semantic similarities. The calculation method for the second distillation loss value is as follows:

[0109]

[0110] Among them, L cl This indicates the second distillation loss value. This represents the semantic audio features of the i-th layer of the speech intent recognition model for the j-th sample. This represents the semantic audio features of the i-th layer of the speech intent recognition model for the k-th sample. Let g(j) represent the first and second semantic target text features of the corresponding text intent recognition model at layer g(j), τ represent hyperparameters, sim represent similarity calculation, and B represent the number of samples.

[0111] According to embodiments of this disclosure, the method may further include the following operations:

[0112] Audio samples are input into the speech intent recognition model to obtain audio intent results; text samples are input into the text intent recognition model to obtain text intent results; and based on the audio intent results and text intent results, task loss values ​​are obtained so as to train the speech intent recognition model using the first distillation loss value, the second distillation loss value, and the task loss value.

[0113] According to embodiments of this disclosure, both audio intent results and text intent results can intuitively represent the user's intent, such as the intent to make a phone call. Based on the audio intent results and text intent results, a task loss value is obtained. The formula for calculating the task loss value is as follows:

[0114]

[0115] Among them, L pred Indicates the task loss value. Indicates the intended result of the text. This indicates the intended audio result.

[0116] According to embodiments of this disclosure, the final loss value can be calculated using the first distillation loss value, the second distillation loss value, and the task loss value. Alternatively, the final loss value can be calculated using the first distillation loss value, the second distillation loss value, the third distillation loss value, and the task loss value, and the speech intent recognition model can be trained based on the final loss value. The formula for calculating the final loss value is as follows:

[0117] L=αL hid +βL att +γL cl +μL pred (10)

[0118] Where α, β, γ, and μ represent weight hyperparameters.

[0119] Figure 3 The diagram illustrates a model training structure of a speech intent recognition model according to an embodiment of the present disclosure.

[0120] like Figure 3 As shown, the training method for the speech intent recognition model includes the following steps.

[0121] Step 1: Input the text sample into the text intent recognition model 301 to obtain text features 302. The text intent recognition model 301 can also simultaneously output the text intent recognition result (Logits) corresponding to the text sample. t The audio samples are input into the speech intent recognition model 303 to obtain audio features 304. The speech intent recognition model 303 can also simultaneously output the speech intent recognition results (Logits) corresponding to the audio samples. s The text intent recognition model 301 includes multiple network layers. Nt The speech intent recognition model 303 also includes multiple network layers. Ns ).

[0122] For example: Inputting the text sample "please make coffee" into the text intent recognition model 301 yields text features 302. The audio data corresponding to the user-inputted text sample is obtained and input into the speech encoder, which outputs an audio sample. This audio sample is then input into the speech intent recognition model 303 to obtain audio features 304.

[0123] Step 2: Transpose the audio feature 304 to obtain the transposed audio feature. Based on the text feature 302 and the transposed audio feature, obtain the initial interaction feature 305.

[0124] Step 3: Convolve the initial interaction feature 305 to obtain the convolved feature, activate the convolved feature to obtain the mutual attention matrix that matches the audio feature 304.

[0125] Step 4: Based on the mutual attention matrix, the aligned audio features 306 of the text features 302 are obtained from the byte level alignment.

[0126] Steps 2, 3, and 4 represent aligning audio and text features at the byte level by introducing a cross attention alignment mechanism.

[0127] Step 5: For each audio feature in the audio sample, determine semantic audio features 307 from the audio features.

[0128] Step 6: Determine the first target text feature and the second target text feature that match the audio feature 304; determine the first semantic target text feature 308 from the first target text feature; and determine the second semantic target text feature 309 from the second target text feature.

[0129] Step 7: Based on semantic audio features 307 and first semantic target text features 308, determine the first semantic similarity; based on semantic audio features and second semantic target text features 309, determine the second semantic similarity, and obtain the aligned audio features with the text features aligned at the semantic level.

[0130] Steps 5, 6, and 7 represent aligning audio and text features at the semantic level by introducing contrastive learning alignment.

[0131] Step 8: Train the speech intent recognition model based on the aligned audio features 306 obtained from byte-level alignment of text features and the aligned audio features from semantic level alignment. Specifically, during model training, the first distillation loss value (L) obtained from byte-level alignment of audio features and text features can be used. hid ) and third distillation loss (L att The model is then trained. The specific calculation process for the first distillation loss value can be found in formula (3), and the specific calculation formula for the third distillation loss value can be found in formula (5).

[0132] Figure 4 A flowchart illustrating a speech intent recognition method according to an embodiment of the present disclosure is shown schematically.

[0133] like Figure 4 As shown, the method includes operations S401 and S402.

[0134] When operating S401, it receives voice information from the user.

[0135] In operation S402, voice information is input into the voice intent recognition model to obtain the intent recognition result. The voice intent recognition model is obtained by using the training method of the voice intent recognition model.

[0136] According to embodiments of this disclosure, a voice intent recognition model receives voice information input by a user, performs end-to-end recognition of the voice information to obtain an intent recognition result, and invokes a process matching the voice information based on the intent recognition result. For example, upon receiving the voice information "Please call Xiao A," the voice intent recognition model recognizes it, finds Xiao A's phone number based on the intent recognition result, and makes the call. Accurately recognizing user intent based on voice information and redirecting to the relevant process effectively improves the efficiency of human-computer interaction.

[0137] Figure 5 A block diagram of a training apparatus for a speech intent recognition model according to an embodiment of the present disclosure is shown schematically.

[0138] like Figure 5 As shown, the training device 500 for the speech intent recognition model includes an audio input module 510, a text input module 520, a loss determination module 530, and a training module 540.

[0139] The first audio input module 510 is used to input audio samples into the speech intent recognition model to obtain audio features.

[0140] The first input module 520 is used to input text samples into the text intent recognition model to obtain text features. The text samples are obtained based on audio samples, and the speech intent recognition model is obtained based on the text intent recognition model.

[0141] The first loss determination module 530 is used to obtain a first distillation loss value and a second distillation loss value based on audio features and text features. The first distillation loss value is used to characterize the byte alignment gap between audio features and text features, and the second distillation loss value is used to characterize the semantic alignment gap between audio features and text features.

[0142] Training module 540 is used to train the speech intent recognition model using the first distillation loss value and the second distillation loss value to obtain the trained speech intent recognition model.

[0143] According to embodiments of this disclosure, the speech intent recognition model is obtained based on the text intent recognition model. It utilizes distillation technology to distill the parameters of the text intent recognition model onto the speech intent recognition model, making the training of the speech intent recognition model more efficient and accurate. In addition, by employing a technique that uses audio features and text features to obtain a first distillation loss value and a second distillation loss value, the audio features and text features are aligned at the semantic level and the byte level, respectively. Therefore, it at least partially overcomes the technical problem of poor alignment effect due to the inherent differences between text information and speech information, making the intent recognition result of the trained speech intent recognition model accurate.

[0144] According to embodiments of this disclosure, the first loss determination module 530 includes a first loss determination submodule.

[0145] The first loss determination submodule is used to obtain a first distillation loss value based on audio features, text features, and a first mutual attention matrix. The audio features are the output of the network layer of the speech intent recognition model, the text features are the output of the network layer of the text intent recognition model, and the first mutual attention matrix is ​​an intermediate feature used to align the text features and audio features byte by byte.

[0146] According to embodiments of the present disclosure, the training apparatus 500 for the speech intent recognition model further includes a second audio input module, a second text input module, and a second loss determination module.

[0147] The second audio input module is used to input audio samples into the speech intent recognition model to obtain audio intermediate features, wherein the audio intermediate features are the output results of the attention mechanism in the network layer of the speech intent recognition model.

[0148] The second text input module is used to input text samples into the text intent recognition model to obtain text intermediate features, where the text intermediate features are the output of the attention mechanism in the network layer of the text intent recognition model.

[0149] The second loss determination module is used to obtain the third distillation loss value based on the text intermediate features and audio intermediate features, so as to train the speech intent recognition model using the first distillation loss value, the second distillation loss value and the third distillation loss value.

[0150] According to embodiments of this disclosure, the second loss determination module includes a second loss determination submodule.

[0151] The second loss determination submodule is used to obtain the third distillation loss value based on the text intermediate features, the audio intermediate features, and the second mutual attention matrix, wherein the second mutual attention matrix is ​​used to align the text intermediate features and the audio intermediate features by bytes.

[0152] According to embodiments of the present disclosure, the training apparatus 500 for the speech intent recognition model further includes a text feature determination module and a matrix determination module.

[0153] The text feature determination module is used to determine the target text feature data that matches the audio feature data from multiple text feature data for each audio feature data in multiple audio feature data, based on a predetermined mapping relationship. The predetermined mapping relationship is determined based on the mapping relationship between the network layers of the text intent recognition model and the network layers of the speech intent recognition model.

[0154] The matrix determination module is used to obtain a mutual attention matrix that matches the audio feature data based on the audio feature data and the target text feature data, wherein the mutual attention matrix includes a first mutual attention matrix or a second mutual attention matrix.

[0155] According to embodiments of this disclosure, the matrix determination module includes a transpose submodule, an interaction feature determination submodule, a convolution submodule, and an activation submodule.

[0156] The transpose submodule is used to transpose the audio feature data to obtain the transposed audio feature data.

[0157] The interaction feature determination submodule is used to obtain initial interaction feature data based on the target text feature data and the transposed audio feature data.

[0158] The convolution submodule is used to convolve the initial interactive feature data to obtain convolutional feature data.

[0159] The activation submodule is used to activate the convolutional feature data to obtain a mutual attention matrix that matches the audio feature data.

[0160] According to embodiments of this disclosure, the first loss determination submodule includes a first alignment feature determination unit and a first loss determination unit.

[0161] The first alignment feature determination unit is used to obtain, for each of the multiple audio features, an aligned audio feature that is aligned with the target text feature at the byte level, based on a first mutual attention matrix that matches the audio feature and the audio feature, wherein the target text feature is a text feature that has a predetermined mapping relationship with the audio feature.

[0162] The first loss determination unit is used to obtain the first distillation loss value based on multiple target text features and multiple aligned audio features.

[0163] According to embodiments of this disclosure, the second loss determination submodule includes a matching feature determination unit, a second alignment feature determination unit, and a second loss determination unit.

[0164] The matching feature determination unit is used to determine, based on a predetermined mapping relationship, a target text intermediate feature that matches the audio intermediate feature from multiple text intermediate features for each audio intermediate feature.

[0165] The second alignment feature determination unit is used to obtain aligned audio intermediate features that are aligned with the target text intermediate features at the byte level, based on the second mutual attention matrix, the transpose of the second mutual attention matrix, and the audio intermediate features.

[0166] The second loss determination unit is used to obtain the third distillation loss value based on multiple target text intermediate features and multiple aligned audio intermediate features.

[0167] According to embodiments of this disclosure, the first loss determination module 530 includes a semantic feature determination submodule, a first target feature determination submodule, a second target feature determination submodule, a first similarity determination submodule, a second similarity determination submodule, and a third loss determination submodule.

[0168] The semantic feature determination submodule is used to determine semantic audio features from audio features for each audio feature in an audio sample.

[0169] The first target feature determination submodule is used to determine a first target text feature that matches the audio feature and a second target text feature that matches the audio feature. The first target text feature is obtained from a first text sample based on a predetermined mapping relationship, and the second target text feature is obtained from a second text sample based on a predetermined mapping relationship. The predetermined mapping relationship is determined based on the mapping relationship between the network layer of the text intent recognition model and the network layer of the speech intent recognition model.

[0170] The second target feature determination submodule is used to determine a first semantic target text feature from the first target text feature and a second semantic target text feature from the second target text feature.

[0171] The first similarity determination submodule is used to determine the first semantic similarity based on semantic audio features and first semantic target text features.

[0172] The second similarity determination submodule is used to determine the second semantic similarity based on semantic audio features and second semantic target text features.

[0173] The third loss determination submodule is used to obtain the second distillation loss value based on multiple first semantic similarities and multiple second semantic similarities.

[0174] According to embodiments of this disclosure, the training apparatus 500 for the speech intent recognition model further includes an audio intent determination module, a text intent determination module, and a task loss determination module.

[0175] The audio intent determination module is used to input audio samples into the speech intent recognition model to obtain audio intent results.

[0176] The text intent determination module is used to input text samples into the text intent recognition model to obtain text intent results.

[0177] The task loss determination module is used to obtain task loss values ​​based on audio intent results and text intent results, so as to train the speech intent recognition model using the first distillation loss value, the second distillation loss value, and the task loss value.

[0178] Figure 6 A block diagram of a voice intent recognition device according to an embodiment of the present disclosure is shown schematically.

[0179] like Figure 6 As shown, the voice intent recognition device 600 includes a receiving module 610 and a result determination module 620.

[0180] The receiving module 610 is used to receive voice information from the user.

[0181] The result determination module 620 is used to input speech information into the speech intent recognition model to obtain the intent recognition result. The speech intent recognition model is obtained by using the training method of the speech intent recognition model.

[0182] Any one or more of the modules, submodules, and units according to embodiments of this disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, and units according to embodiments of this disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, and units according to embodiments of this disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, and units according to embodiments of this disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0183] For example, any plurality of the audio input module 510, text input module 520, loss determination module 530, and training module 540 may be combined into one module / unit / subunit, or any one of these modules / units / subunits may be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits may be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of the present disclosure, at least one of the audio input module 510, text input module 520, loss determination module 530, and training module 540 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the audio input module 510, text input module 520, loss determination module 530, and training module 540 may be implemented at least partially as a computer program module that can perform corresponding functions when the computer program module is run.

[0184] It should be noted that the training device part of the speech intent recognition model in the embodiments of this disclosure corresponds to the training method part of the speech intent recognition model in the embodiments of this disclosure. For a detailed description of the training device part of the speech intent recognition model, please refer to the training method part of the speech intent recognition model, which will not be repeated here.

[0185] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a training method for a speech intent recognition model according to embodiments of the present disclosure. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0186] like Figure 7As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0187] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0188] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The system 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0189] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0190] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0191] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0192] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0193] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the training method for the voice intent recognition model provided in the embodiments of this disclosure.

[0194] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0195] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0196] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0198] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for training a speech intent recognition model, comprising: Audio samples are input into the speech intent recognition model to obtain audio features; Text samples are input into a text intent recognition model to obtain text features, wherein the text samples are obtained based on the audio samples, and the speech intent recognition model is obtained based on the text intent recognition model; Based on the audio features, the text features, and the first mutual attention matrix, a first distillation loss value is obtained, wherein the first distillation loss value is used to characterize the byte alignment difference between the audio features and the text features; Based on the audio features and the text features, a second distillation loss value is obtained, wherein the second distillation loss value is used to characterize the semantic alignment gap between the audio features and the text features; and The speech intent recognition model is trained using the first distillation loss value and the second distillation loss value to obtain the trained speech intent recognition model; The first mutual attention matrix is ​​determined in the following manner: Based on a predetermined mapping relationship, target text feature data matching audio feature data is determined from multiple text feature data sets. The predetermined mapping relationship is determined based on the mapping relationship between the network layers of the text intent recognition model and the network layers of the speech intent recognition model. The audio feature data includes the audio features output by the network layers of the speech intent recognition model, and the text feature data includes the text features output by the network layers of the text intent recognition model. Based on the audio feature data and the target text feature data, a mutual attention matrix matching the audio feature data is obtained, wherein the mutual attention matrix includes the first mutual attention matrix.

2. The method according to claim 1, wherein: When the network layer of the speech intent recognition model includes an attention mechanism, The training method for the speech intent recognition model further includes: The audio sample is input into the speech intent recognition model to obtain audio intermediate features, wherein the audio intermediate features are the output of the attention mechanism in the network layer of the speech intent recognition model; The text sample is input into the text intent recognition model to obtain intermediate text features, wherein the intermediate text features are the output of the attention mechanism in the network layer of the text intent recognition model; and Based on the text intermediate features and the audio intermediate features, a third distillation loss value is obtained, so as to train the speech intent recognition model using the first distillation loss value, the second distillation loss value and the third distillation loss value.

3. The method according to claim 2, wherein: The process of obtaining the third distillation loss value based on the text intermediate features and the audio intermediate features includes: The third distillation loss value is obtained based on the text intermediate features, the audio intermediate features, and the second mutual attention matrix, wherein the second mutual attention matrix is ​​an intermediate feature used to align the text intermediate features and the audio intermediate features byte by byte.

4. The method according to claim 3, wherein: The audio feature data also includes the audio intermediate features, and the text feature data also includes the text intermediate features; The mutual attention matrix also includes the second mutual attention matrix.

5. The method according to claim 1 or 4, wherein: The step of obtaining a mutual attention matrix that matches the audio feature data based on the audio feature data and the target text feature data includes: The audio feature data is transposed to obtain the transposed audio feature data; Based on the target text feature data and the transposed audio feature data, initial interaction feature data is obtained; The initial interaction feature data is convolved to obtain convolved feature data; and The convolutional feature data is activated to obtain a mutual attention matrix that matches the audio feature data.

6. The method according to claim 1, wherein: The audio features include multiple components, and the text features include multiple components. The process of obtaining the first distillation loss value based on the audio features, the text features, and the first mutual attention matrix includes: For each of the plurality of audio features, based on a first mutual attention matrix matching the audio feature and the audio feature, an aligned audio feature is obtained that is byte-level aligned with a target text feature, wherein the target text feature is a text feature that has the predetermined mapping relationship with the audio feature; and The first distillation loss value is obtained based on multiple target text features and multiple aligned audio features.

7. The method according to claim 3 or 4, wherein: The audio intermediate features include multiple features, and the text intermediate features include multiple features. The process of obtaining the third distillation loss value based on the text intermediate features, the audio intermediate features, and the second mutual attention matrix includes: For each of the plurality of audio intermediate features, a target text intermediate feature that matches the audio intermediate feature is determined from the plurality of text intermediate features based on the predetermined mapping relationship; Based on the second mutual attention matrix, the transpose of the second mutual attention matrix, and the audio intermediate features, aligned audio intermediate features are obtained that are aligned with the target text intermediate features at the byte level; and The third distillation loss value is obtained based on multiple target text intermediate features and multiple aligned audio intermediate features.

8. The method according to any one of claims 1 to 4, wherein: The audio samples include multiple ones, and the text samples include multiple ones. A first text sample obtained based on the audio samples is a positive sample pair with the audio samples, and a second text sample among the multiple text samples other than the first text sample is a negative sample pair with the audio samples. The process of obtaining the second distillation loss value based on the audio features and the text features includes: For each of the plurality of audio samples, For each audio feature in the audio sample, a semantic audio feature is determined from the audio feature; A first target text feature matching the audio feature and a second target text feature matching the audio feature are determined, wherein the first target text feature is obtained from the first text sample based on a predetermined mapping relationship, and the second target text feature is obtained from the second text sample based on the predetermined mapping relationship, wherein the predetermined mapping relationship is determined based on the mapping relationship between the network layer of the text intent recognition model and the network layer of the speech intent recognition model; Determine a first semantic target text feature from the first target text feature, and determine a second semantic target text feature from the second target text feature; Based on the semantic audio features and the first semantic target text features, a first semantic similarity is determined; Based on the semantic audio features and the second semantic target text features, a second semantic similarity is determined; and The second distillation loss value is obtained based on multiple first semantic similarities and multiple second semantic similarities.

9. The method according to claim 1, further comprising: The audio sample is input into the speech intent recognition model to obtain the audio intent result; The text sample is input into the text intent recognition model to obtain the text intent result; and Based on the audio intent result and the text intent result, a task loss value is obtained so as to train the speech intent recognition model using the first distillation loss value, the second distillation loss value and the task loss value.

10. A method for recognizing speech intent, comprising: Receive voice messages from users; as well as The voice information is input into the voice intent recognition model to obtain the intent recognition result. The speech intent recognition model is obtained using the training method described in any one of claims 1 to 9.

11. A training device for a speech intent recognition model, comprising: The audio input module is used to input audio samples into the speech intent recognition model to obtain audio features; A text input module is used to input text samples into a text intent recognition model to obtain text features, wherein the text samples are obtained based on the audio samples, and the speech intent recognition model is obtained based on the text intent recognition model; A first loss determination module is configured to obtain a first distillation loss value and a second distillation loss value based on the audio features and the text features, wherein the first distillation loss value is used to characterize the byte alignment gap between the audio features and the text features, and the second distillation loss value is used to characterize the semantic alignment gap between the audio features and the text features; and The training module is used to train the speech intent recognition model using the first distillation loss value and the second distillation loss value to obtain the trained speech intent recognition model. The first loss determination module includes: The first loss determination submodule is used to obtain the first distillation loss value based on the audio features, the text features, and the first mutual attention matrix; The training device for the speech intent recognition model also includes: A text feature determination module is used to determine target text feature data that matches audio feature data from multiple text feature data based on a predetermined mapping relationship. The predetermined mapping relationship is determined based on the mapping relationship between the network layers of the text intent recognition model and the network layers of the speech intent recognition model. The audio feature data includes the audio features output by the network layers of the speech intent recognition model, and the text feature data includes the text features output by the network layers of the text intent recognition model. The matrix determination module is used to obtain a mutual attention matrix that matches the audio feature data based on the audio feature data and the target text feature data, wherein the mutual attention matrix includes the first mutual attention matrix.

12. A voice intent recognition device, comprising: The receiving module is used to receive voice information from the user; The result determination module is used to input the voice information into the voice intent recognition model to obtain the intent recognition result, wherein the voice intent recognition model is obtained using the training method as described in any one of claims 1 to 9.

13. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 10.

14. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 10.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images