Training method, device and electronic equipment for speech intention recognition model

Abstract

The present disclosure relates to a training method, device and electronic device for a speech intention recognition model, and relates to the field of speech recognition technology. In the present disclosure, a first speech sample with an intent label and a corresponding first text sample are obtained. A trained first multimodal extraction network obtained by training a second speech sample and a corresponding second text sample is obtained. Based on the first speech sample with an intent label and the corresponding first text sample, the speech intention recognition model is trained to obtain a trained speech intention recognition model. The second multimodal extraction network in the speech intention recognition model has the same network structure as the first multimodal extraction network and inherits the network parameters of the trained first multimodal extraction network. The intent recognition network in the speech intention recognition model obtains an estimated speech intent based on the various modal features extracted by the second multimodal extraction network. In this way, the cost of obtaining training samples can be reduced, and the difficulty of implementing model training can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of speech recognition technology, and in particular to a training method, device, and electronic device for a speech intent recognition model. Background Art

[0002] At present, the application of voice intent recognition is becoming more and more widespread. For example, by performing voice intent recognition on the voice generated during live broadcast, the current link in the live broadcast process can be determined.

[0003] In related technologies, a speech intent recognition model is often trained directly based on speech samples with intent labels, and then intent recognition is performed based on this speech intent recognition model. This approach requires a large number of speech samples with intent labels for training, and these intent-labeled speech samples require manual labeling, which is costly to obtain. Therefore, model training is relatively difficult to implement. Summary of the Invention

[0004] The present disclosure provides a method, device, and electronic device for training a speech intent recognition model to at least address the difficulty of implementing model training in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for training a speech intent recognition model is provided, comprising:

[0006] Obtaining a first training sample set, where the first training sample set includes a first speech sample with an intent label and a corresponding first text sample;

[0007] Obtaining a trained first multimodal extraction network, the first multimodal extraction network comprising at least a speech extraction subnetwork for extracting speech semantic features and a text extraction subnetwork for extracting text semantic features, the first multimodal extraction network being trained using a second training sample set consisting of a second speech sample and a corresponding second text sample;

[0008] Training a speech intent recognition model based on the first speech sample with the intent label and the corresponding first text sample to obtain a trained speech intent recognition model;

[0009] Among them, the speech intention recognition model includes a second multimodal extraction network and an intention recognition network. The second multimodal extraction network has the same network structure as the first multimodal extraction network and inherits the network parameters of the trained first multimodal extraction network. The intention recognition network obtains an estimated speech intention based on the modal features extracted by the second multimodal extraction network.

[0010] Optionally, obtaining a trained first multimodal extraction network includes:

[0011] Inputting N second speech samples into the speech extraction subnetwork for semantic feature extraction to obtain N speech semantic features, and inputting N second text samples corresponding to the N second speech samples into the text extraction subnetwork for semantic feature extraction to obtain N text semantic features; wherein N is a positive integer greater than or equal to 2;

[0012] Calculating the semantic similarity between each of the N speech semantic features and each of the N text semantic features;

[0013] Based on the semantic similarity, network parameters of the speech extraction subnetwork and the text extraction subnetwork are adjusted to obtain the trained first multimodal extraction network.

[0014] Optionally, adjusting the network parameters of the speech extraction subnetwork and the text extraction subnetwork based on the semantic similarity includes:

[0015] Based on the semantic similarity, construct a similarity matrix; wherein the number of rows and columns of the similarity matrix are both N, an element in the similarity matrix represents the similarity between a first semantic feature corresponding to the row where the element is located and a second semantic feature corresponding to the column where the element is located, and one of the first semantic feature and the second semantic feature is selected as the speech semantic feature and the other is the text semantic feature;

[0016] Calculating a first target loss value according to each row element and each column element in the similarity matrix;

[0017] The network parameters of the speech extraction subnetwork and the text extraction subnetwork are adjusted according to the first target loss value.

[0018] Optionally, calculating the first target loss value according to each row element and each column element in the similarity matrix includes:

[0019] For any row in the similarity matrix, calculate a first loss value corresponding to the row based on the elements on the diagonal of the row and the elements on the off-diagonal of the row; the first loss value is negatively correlated with the elements on the diagonal of the row, and the first loss value is positively correlated with the elements on the off-diagonal of the row;

[0020] For any column in the similarity matrix, calculate a second loss value corresponding to the column based on the elements on the diagonal of the column and the elements on the off-diagonal of the column; the second loss value is negatively correlated with the elements on the diagonal of the column, and the second loss value is positively correlated with the elements on the off-diagonal of the column;

[0021] The first target loss value is determined based on the first loss value corresponding to each of the rows and the second loss value corresponding to each of the columns.

[0022] Optionally, before calculating the semantic similarity between each of the N speech semantic features and each of the N text semantic features, the method further includes:

[0023] For any second speech sample among the N second speech samples, performing pooling processing on the speech semantic features of the second speech sample in the time dimension to obtain a speech representation vector of the speech semantic features;

[0024] The calculating of the semantic similarity between each of the N speech semantic features and each of the N text semantic features comprises:

[0025] The semantic similarity between the speech representation vector of each of the speech semantic features and the text representation vector of each of the text semantic features is calculated.

[0026] Optionally, the training of a speech intent recognition model based on the first speech sample with the intent label and the corresponding first text sample to obtain a trained speech intent recognition model includes:

[0027] Inputting the first speech sample and the first text sample into the speech extraction subnetwork and the text extraction subnetwork in the second multimodal extraction network respectively to extract semantic features, thereby obtaining target speech semantic features and target text semantic features;

[0028] Inputting the target speech semantic features and the target text semantic features into the intention recognition network to obtain the estimated speech intention;

[0029] Determining a second target loss value according to the estimated speech intent and the intent label;

[0030] Based on the second target loss value, adjust the network parameters of the intent recognition network, or adjust the network parameters of the intent recognition network, the speech extraction subnetwork and the text extraction subnetwork in the second multimodal extraction network to obtain the trained speech intent recognition model.

[0031] According to a second aspect of an embodiment of the present disclosure, a training device for a speech intent recognition model is provided, comprising:

[0032] A first acquisition module is configured to acquire a first training sample set, where the first training sample set includes a first speech sample with an intent label and a corresponding first text sample;

[0033] A second acquisition module is configured to execute acquisition of a trained first multimodal extraction network, wherein the first multimodal extraction network includes at least a speech extraction subnetwork for extracting speech semantic features and a text extraction subnetwork for extracting text semantic features, and the first multimodal extraction network is trained using a second training sample set consisting of a second speech sample and a corresponding second text sample;

[0034] A training module is configured to train a speech intent recognition model based on the first speech sample with the intent label and the corresponding first text sample to obtain a trained speech intent recognition model;

[0035] Among them, the speech intention recognition model includes a second multimodal extraction network and an intention recognition network. The second multimodal extraction network has the same network structure as the first multimodal extraction network and inherits the network parameters of the trained first multimodal extraction network. The intention recognition network obtains an estimated speech intention based on the modal features extracted by the second multimodal extraction network.

[0036] Optionally, the second acquisition module is specifically configured to execute:

[0037] Inputting N second speech samples into the speech extraction subnetwork for semantic feature extraction to obtain N speech semantic features, and inputting N second text samples corresponding to the N second speech samples into the text extraction subnetwork for semantic feature extraction to obtain N text semantic features; wherein N is a positive integer greater than or equal to 2;

[0038] Calculating the semantic similarity between each of the N speech semantic features and each of the N text semantic features;

[0039] Based on the semantic similarity, network parameters of the speech extraction subnetwork and the text extraction subnetwork are adjusted to obtain the trained first multimodal extraction network.

[0040] Optionally, the second acquisition module is further configured to execute:

[0041] Based on the semantic similarity, construct a similarity matrix; wherein the number of rows and columns of the similarity matrix are both N, an element in the similarity matrix represents the similarity between a first semantic feature corresponding to the row where the element is located and a second semantic feature corresponding to the column where the element is located, and one of the first semantic feature and the second semantic feature is selected as the speech semantic feature and the other is the text semantic feature;

[0042] Calculating a first target loss value according to each row element and each column element in the similarity matrix;

[0043] The network parameters of the speech extraction subnetwork and the text extraction subnetwork are adjusted according to the first target loss value.

[0044] Optionally, the second acquisition module is further configured to execute:

[0045] For any row in the similarity matrix, calculate a first loss value corresponding to the row based on the elements on the diagonal of the row and the elements on the off-diagonal of the row; the first loss value is negatively correlated with the elements on the diagonal of the row, and the first loss value is positively correlated with the elements on the off-diagonal of the row;

[0046] For any column in the similarity matrix, calculate a second loss value corresponding to the column based on the elements on the diagonal of the column and the elements on the off-diagonal of the column; the second loss value is negatively correlated with the elements on the diagonal of the column, and the second loss value is positively correlated with the elements on the off-diagonal of the column;

[0047] The first target loss value is determined based on the first loss value corresponding to each of the rows and the second loss value corresponding to each of the columns.

[0048] Optionally, the device further includes:

[0049] A pooling module is configured to, before the second acquisition module 202 calculates the semantic similarity between each of the N speech semantic features and each of the N text semantic features, perform pooling processing on the speech semantic features of any of the N second speech samples in the time dimension to obtain a speech representation vector of the speech semantic features;

[0050] The second acquisition module is further configured to execute: calculating the semantic similarity between the speech representation vector of each of the speech semantic features and the text representation vector of each of the text semantic features.

[0051] Optionally, the training module is specifically configured to execute:

[0052] Inputting the first speech sample and the first text sample into the speech extraction subnetwork and the text extraction subnetwork in the second multimodal extraction network respectively to extract semantic features, thereby obtaining target speech semantic features and target text semantic features;

[0053] Inputting the target speech semantic features and the target text semantic features into the intention recognition network to obtain the estimated speech intention;

[0054] Determining a second target loss value according to the estimated speech intent and the intent label;

[0055] Based on the second target loss value, adjust the network parameters of the intent recognition network, or adjust the network parameters of the intent recognition network, the speech extraction subnetwork and the text extraction subnetwork in the second multimodal extraction network to obtain the trained speech intent recognition model.

[0056] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0057] a processor; a memory for storing instructions executable by the processor;

[0058] The processor is configured to execute the instructions to implement the method as described in any one of the first aspects.

[0059] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device executes the method as described in any one of the first aspects.

[0060] According to a fifth aspect of an embodiment of the present disclosure, a computing device program product is provided, which includes readable program instructions. When the readable program instructions are executed by a processor of an electronic device, the electronic device executes the method as described in any one of the first aspects.

[0061] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: obtaining a first training sample set, the first training sample set including a first speech sample with an intent label and a corresponding first text sample. Obtaining a trained first multimodal extraction network, the first multimodal extraction network at least including a speech extraction subnetwork for extracting speech semantic features, and a text extraction subnetwork for extracting text semantic features, the first multimodal extraction network is trained using a second training sample set consisting of a second speech sample and a corresponding second text sample. Based on the first speech sample with an intent label and the corresponding first text sample, the speech intent recognition model is trained to obtain a trained speech intent recognition model. The speech intent recognition model includes a second multimodal extraction network and an intent recognition network. The second multimodal extraction network has the same network structure as the first multimodal extraction network and inherits the network parameters of the trained first multimodal extraction network. The intent recognition network obtains an estimated speech intent based on the modal features extracted by the second multimodal extraction network. Because the first multimodal extraction network was first trained using the readily available second training sample set, the second multimodal extraction network in the speech intent recognition model to be trained inherits the network parameters of the first multimodal extraction network trained using the readily available training data in the first training phase. This is equivalent to the second multimodal extraction network having already been pre-trained using the readily available second training sample set, with only the intent recognition network remaining untrained. Therefore, the speech intent recognition model can subsequently be trained using a small number of intent-labeled samples, which can reduce the amount of intent-labeled training data required, thereby lowering the cost of acquiring training samples and reducing the difficulty of model training.

[0062] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0064] Figure 1 is a flowchart of a method for training a speech intent recognition model according to an exemplary embodiment;

[0065] Figure 2 is a schematic diagram of a similarity matrix according to an exemplary embodiment;

[0066] Figure 3 is a schematic diagram of a training process according to an exemplary embodiment;

[0067] Figure 4 is another schematic diagram of a training process according to an exemplary embodiment;

[0068] Figure 5 is a block diagram of a training device for a speech intent recognition model according to an exemplary embodiment;

[0069] Figure 6 is a block diagram of a device for training a speech intent recognition model according to an exemplary embodiment;

[0070] Figure 7 It is a block diagram of another apparatus for training a speech intent recognition model according to an exemplary embodiment. DETAILED DESCRIPTION

[0071] In order to enable ordinary people in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0072] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0073] Figure 1 is a flowchart of a method for training a speech intent recognition model according to an exemplary embodiment. Figure 1 As shown, the method may include:

[0074] Step 101: Obtain a first training sample set, where the first training sample set includes a first speech sample with an intent label and a corresponding first text sample.

[0075] Step 102: Obtain a trained first multimodal extraction network, wherein the first multimodal extraction network includes at least a speech extraction subnetwork for extracting speech semantic features and a text extraction subnetwork for extracting text semantic features, and the first multimodal extraction network is trained using a second training sample set consisting of a second speech sample and a corresponding second text sample.

[0076] In the embodiments of the present disclosure, the speech sample can be speech generated in the field to which the speech intent recognition model is applied. For example, it can be speech data generated in historical live broadcasts, thereby improving the subsequent application effect of the speech intent recognition model to a certain extent. Of course, it can also be speech data generated in other fields, and this disclosure does not limit this.

[0077] The first text sample may be obtained by performing automatic speech recognition (ASR) on the first voice sample, and the first text sample corresponds to the content of the first voice sample. The second text sample may be obtained by performing ASR recognition on the second voice sample, and the second text sample corresponds to the content of the second voice sample.

[0078] It should be noted that the embodiments of the present disclosure can be applied to a training device, which can be a terminal device such as a smart phone, a tablet computer, a personal computer (PC), or a server.

[0079] In an embodiment of the present disclosure, the speech extraction subnetwork and the text extraction subnetwork can be pre-trained with the training goal of matching the speech semantic features of at least two second speech samples extracted by the speech extraction subnetwork with the text semantic features of at least two second text samples extracted by the text extraction subnetwork, thereby obtaining a trained first multimodal extraction network.

[0080] In the embodiment of the present disclosure, the speech extraction subnetwork can extract at least two speech semantic features from at least two second speech samples, and the text extraction subnetwork can extract at least two text semantic features from the corresponding at least two second text samples. By determining which speech semantic feature matches which text semantic feature, the speech extraction subnetwork and the text extraction subnetwork are pre-trained, so that the speech extraction subnetwork and the text extraction subnetwork can learn rich semantic information and learn the ability to generate accurate and corresponding matching semantic features for paired speech and text, thereby ensuring the recognition accuracy of the subsequent intent recognition network. Among them, pre-training can be regarded as the first training stage in the training process of the speech intent recognition model. The training data in the first training stage is the <speech-text> pair. Since the text corresponding to the speech can be obtained directly based on the ASR algorithm, the difficulty of obtaining the training data in the first training stage is relatively low. In the first training stage, the speech extraction subnetwork and the text extraction subnetwork can be trained based on the easily accessible second training sample set.

[0081] Step 103: Based on the first speech sample with the intent label and the corresponding first text sample, a speech intent recognition model is trained to obtain a trained speech intent recognition model; the speech intent recognition model includes a second multimodal extraction network and an intent recognition network, the second multimodal extraction network has the same network structure as the first multimodal extraction network, and inherits the network parameters of the trained first multimodal extraction network, and the intent recognition network obtains an estimated speech intent based on the modal features extracted by the second multimodal extraction network.

[0082] Among them, the training in this step can be regarded as the second training stage in the training process of the speech intent recognition model. The training data in the second training stage are <speech, text-intent label> pairs. Although the training data in the second training stage needs to be manually labeled, the acquisition difficulty and cost are relatively high. However, since the second multimodal extraction network in the speech intent recognition model to be trained inherits the network parameters of the first multimodal extraction network obtained by training in the first training stage using easily accessible training data, it is equivalent to that the second multimodal extraction network in the speech intent recognition model to be trained has been pre-trained using the easily accessible second training sample set, and only the intent recognition network remains untrained. Therefore, compared with the prior art method of setting only one training stage and using a large number of samples with intent labels to directly train the speech intent recognition model, the training of the speech intent recognition model can be achieved in the second training stage using a small amount of the first training set with intent labels, which can reduce the amount of training data with intent labels required to a certain extent, thereby reducing the sample acquisition cost and the difficulty of implementing model training to a certain extent.

[0083] In summary, the training method of the speech intention recognition model provided by the embodiment of the present disclosure obtains a first training sample set, and the first training sample set includes a first speech sample with an intent label and a corresponding first text sample. A trained first multimodal extraction network is obtained, and the first multimodal extraction network at least includes a speech extraction subnetwork for extracting speech semantic features, and a text extraction subnetwork for extracting text semantic features. The first multimodal extraction network is trained using a second training sample set consisting of a second speech sample and a corresponding second text sample. Based on the first speech sample with an intent label and the corresponding first text sample, the speech intention recognition model is trained to obtain a trained speech intention recognition model. Among them, the speech intention recognition model includes a second multimodal extraction network and an intention recognition network. The second multimodal extraction network has the same network structure as the first multimodal extraction network and inherits the network parameters of the trained first multimodal extraction network. The intention recognition network obtains an estimated speech intention based on the modal features extracted by the second multimodal extraction network. Because the first multimodal extraction network was first trained using the readily available second training sample set, the second multimodal extraction network in the speech intent recognition model to be trained inherits the network parameters of the first multimodal extraction network trained using the readily available training data in the first training phase. This is equivalent to the second multimodal extraction network having already been pre-trained using the readily available second training sample set, with only the intent recognition network remaining untrained. Therefore, the speech intent recognition model can subsequently be trained using a small number of intent-labeled samples, which can reduce the amount of intent-labeled training data required, thereby lowering the cost of acquiring training samples and reducing the difficulty of model training.

[0084] Optionally, in the embodiment of the present disclosure, the step of obtaining the trained first multimodal extraction network may specifically include:

[0085] Step S21: input N second speech samples into the speech extraction subnetwork for semantic feature extraction to obtain N speech semantic features, and input N second text samples corresponding to the N second speech samples into the text extraction subnetwork for semantic feature extraction to obtain N text semantic features; N is a positive integer greater than or equal to 2.

[0086] Among them, the specific value of N can be set according to actual needs, as long as N is not less than 2 and is a positive integer. Exemplarily, the value of N can be equal to the number of a batch of data defined in the training process, for example, N can be equal to 16. The speech extraction subnetwork can be a deep neural network. For example, the speech extraction subnetwork can be a deep self-attention network (Conformer) with convolution. Among them, Conformer is a neural network with good effect in the field of speech. Therefore, using Conformer as the speech extraction subnetwork can improve the speech intention recognition effect of the model to a certain extent. And compared with the method of using a recurrent neural network, the effect on long sequence speech related tasks can be ensured. Of course, other networks can also be used, and the embodiments of the present disclosure are not limited to this. The text extraction subnetwork can be a masked language model. Specifically, it can be a masked language model that has been trained using massive unsupervised text data, such as a deep bidirectional self-attention network (Bidirectional Encoder Representation from Transformers, BERT), or it can be a robust deep bidirectional self-attention network (Robustly Optimized BERT Pretraining Approach, RoBERTa) or a masked corrected deep bidirectional self-attention network (MLM as correction BERT, MacBERT), which has equivalent effects.

[0087] A batch of paired second speech samples and second text samples can be input into the speech extraction subnetwork and the text extraction subnetwork respectively. The speech extraction subnetwork can first extract the speech features of the second speech sample. The specific extraction method can refer to the prior art, for example, one-hot encoding, which will not be repeated here. Processing is performed based on the speech features to obtain speech semantic features. When defining the speech extraction subnetwork, the network that extracts speech features can be regarded as part of the speech extraction subnetwork, or the two can be regarded as two independent networks, that is, the input of the speech extraction subnetwork can be a speech sample or the speech features of the speech sample. The two are just different in the division of the network for extracting speech features, and are essentially the same.

[0088] The text extraction subnetwork can also first extract text features from the text sample. For example, based on a preset text table, text features can be converted and extracted through a text lookup. Processing is then performed based on the text features to obtain semantic features. The network for extracting text features is similar to the network for extracting speech features and will not be further elaborated here.

[0089] Step S22: Calculate the semantic similarity between each of the N speech semantic features and each of the N text semantic features.

[0090] Optionally, in an embodiment of the present disclosure, before calculating the semantic similarity between each of the N speech semantic features and each of the N text semantic features, the speech semantic features can be processed as follows: Step S31, for any second speech sample among the N second speech samples, the speech semantic features of the second speech sample are pooled in the time dimension to obtain a speech representation vector of the speech semantic features.

[0091] Specifically, for the second speech sample, the speech semantic features output by the speech extraction subnetwork can be in matrix form. For each time unit (e.g., an audio frame), a P-dimensional feature vector is extracted. The feature vectors of all time units of the second speech sample are aggregated to form a feature matrix. A row of the feature matrix corresponds to a time unit, and a column corresponds to a one-dimensional feature vector. Assuming the second speech sample consists of Q audio frames, a Q*P feature matrix can be obtained.

[0092] Speech semantic features can be pooled in the time dimension, such as average pooling or max pooling. Specifically, for each column in the feature matrix, the maximum element value is extracted to ultimately obtain a 1*P speech representation vector. This speech representation vector is the speech representation vector of the speech semantic feature.

[0093] Accordingly, the semantic similarity between the speech representation vector of each of the speech semantic features and the text representation vector of each of the text semantic features can be calculated. In this step, by determining the semantic similarity between the speech semantic features and the text semantic features, a clear reference standard can be provided for training. Specifically, the similarity calculation can be performed using the speech vector representation of each of the N speech semantic features, as well as the text vector representation of each of the N text semantic features. For example, the semantic similarity can be calculated by calculating the cosine similarity between each speech vector representation and each text vector representation. By converting the speech semantic features and the text semantic features into sentence-level speech representation vectors and text representation vectors, respectively, the similarity between the two vectors can be directly calculated to obtain the semantic similarity. This can reduce the amount of computation and simplify the similarity calculation. It should be noted that the sentence level is mentioned here because speech data and text data are usually preprocessed and segmented according to sentences. Therefore, a speech sample or a text sample is often the content of a sentence, and the speech representation vector of the speech sample or the text representation vector of the text sample is a sentence-level representation vector.

[0094] In the embodiment of the present disclosure, before using the speech semantic features to calculate the similarity, the speech semantic features of the second speech sample are first pooled in the time dimension to obtain a speech representation vector of the speech semantic features. The speech representation vector based on the speech semantic features participates in the similarity calculation, which can, to a certain extent, make the feature representation effect of subsequent similarity calculation better, thereby ensuring the overall processing effect of the model.

[0095] It should be noted that the text extraction subnetwork can add a character at the beginning of the sentence when outputting text semantic features, or add characters at the beginning and end of the sentence respectively, to represent the text representation of this sentence. For example, when the text speech extraction subnetwork is BERT, RoBERTa or MacBERT, this type of network can add a character at the beginning of the sentence. In the embodiment of the present disclosure, the first character of the text semantic feature can be directly taken as the text representation vector at the sentence level. Of course, for the network that does not output this first character of the sentence, the text semantic feature can also be subjected to the aforementioned pooling processing on the time dimension or other transformation processing to obtain a text representation vector. For example, maximum pooling is performed according to characters. For example, the second text sample includes 10 characters and symbols added at the beginning and end of the sentence, for a total of 12 characters. One character corresponds to an R-dimensional feature vector, and the feature vectors of all characters of the second text sample are combined to form a feature matrix. Among them, one row of the feature matrix corresponds to one character, and one column corresponds to a one-dimensional feature vector, so a 12*R feature matrix can be obtained. Furthermore, for each column in the feature matrix, the maximum element value can be extracted to finally obtain a 1*R text representation vector. This text representation vector is the text representation vector of the text semantic feature. In the embodiment of the present disclosure, before using the text semantic feature to calculate the similarity, the text semantic feature of the second text sample is first pooled in the character dimension to obtain the speech representation vector of the text semantic feature. The text representation vector based on the text semantic feature participates in the similarity calculation, which can make the feature representation effect of the subsequent similarity calculation better to a certain extent, thereby ensuring the overall processing effect of the model.

[0096] The specific values ​​of P and Q can be set according to actual conditions. Assuming that P and Q are both 768, the batch data of the second voice sample and the second text sample are respectively converted into two vectors of 16 times 768 dimensions.

[0097] Step S23: Based on the semantic similarity, adjust the network parameters of the speech extraction subnetwork and the text extraction subnetwork to obtain the trained first multimodal extraction network.

[0098] In this step, the current loss value can be determined based on the multiple semantic similarities obtained by calculation, and the network parameters of the speech extraction subnetwork and the text extraction subnetwork can be optimized and updated based on the current loss value. For example, the parameter update method can be the back propagation method commonly used in deep neural networks. And continue to repeat the above training process after the update until the loss value meets the preset requirements, for example, it is less than the preset loss value threshold. After continuous updating and iteration, the pre-training of the speech extraction subnetwork and the text extraction subnetwork can be completed. Among them, when the updated speech extraction subnetwork and the text extraction subnetwork are trained again, the second speech sample and the corresponding second text sample in the batch that have not been used can be used.

[0099] In the embodiment of the present disclosure, N second speech samples are input into the speech extraction subnetwork for semantic feature extraction to obtain N speech semantic features, and N second text samples corresponding to the N second speech samples are input into the text extraction subnetwork for semantic feature extraction to obtain N text semantic features; N is a positive integer greater than or equal to 2. The semantic similarity between each speech semantic feature in the N speech semantic features and each text semantic feature in the N text semantic features is calculated. Based on the semantic similarity, the network parameters of the speech extraction subnetwork and the text extraction subnetwork are adjusted to obtain the trained first multimodal extraction network. In this way, to a certain extent, the speech extraction subnetwork and the text extraction subnetwork can learn rich semantic information and learn the ability to generate accurate and corresponding semantic features, thereby ensuring the training effect of pre-training.

[0100] Optionally, the step of adjusting the network parameters of the speech extraction subnetwork and the text extraction subnetwork based on the semantic similarity may specifically include:

[0101] Step S41: construct a similarity matrix based on the semantic similarity; wherein the number of rows and columns of the similarity matrix are both N, and an element in the similarity matrix represents the similarity between the first semantic feature corresponding to the row where the element is located and the second semantic feature corresponding to the column where the element is located, and among the first semantic feature and the second semantic feature, one is selected as the speech semantic feature and the other is the text semantic feature.

[0102] Specifically, when the first semantic feature is a speech semantic feature, the second semantic feature is a text semantic feature; when the first semantic feature is a text semantic feature, the second semantic feature is a speech semantic feature.

[0103] In the disclosed embodiments, a row in the similarity matrix may correspond to a text semantic feature, a column may correspond to a speech semantic feature, and an element is the similarity between the text semantic feature corresponding to the row in which the element resides and the speech semantic feature corresponding to the column in which the element resides. Alternatively, a row in the similarity matrix may correspond to a speech semantic feature, and a column may correspond to a text semantic feature.

[0104] For example, Figure 2 This is a schematic diagram of a similarity matrix according to an exemplary embodiment. Assume that a row corresponds to a text semantic feature and a column corresponds to a speech semantic feature. i Represents the semantic features of the text, S j Represents the speech semantic features, i, j∈1,…,N. Then the similarity matrix can be Figure 2 As shown. i S j It can represent the semantic similarity between the i-th text semantic feature and the j-th speech semantic feature. The semantic similarity matrix A can be = cosine similarity (S_N, T_N), where A i,j Represents the similarity between the text semantic features of the i-th second text sample and the speech semantic features of the j-th second speech sample, that is, the element in the i-th row and j-th column.

[0105] Step S42: Calculate a first target loss value according to each row element and each column element in the similarity matrix.

[0106] In this step, when there are multiple semantic similarities, the calculation efficiency of the first target loss value can be improved to a certain extent by dividing the rows and columns.

[0107] Step S43: Adjust the network parameters of the speech extraction subnetwork and the text extraction subnetwork according to the first target loss value.

[0108] In the embodiment of the present disclosure, by constructing a similarity matrix and calculating the first target loss value by row and column for the similarity matrix, the calculation efficiency of the first target loss value can be improved to a certain extent, thereby improving the subsequent parameter adjustment efficiency.

[0109] Optionally, in an embodiment of the present invention, the elements on the diagonal of the similarity matrix are used to represent the similarity between the first target speech sample and the first target text sample, and the content of the first target speech sample matches the content of the first target text sample; the elements on the off-diagonal of the similarity matrix are used to represent the similarity between the second target speech sample and the second target text sample, and the content of the second target speech sample does not match the content of the second target text sample; the first target speech sample and the second target speech sample belong to N second speech samples, and the first target text sample and the second target text sample belong to N second text samples. The operation of calculating the first target loss value based on each row element and each column element in the similarity matrix may specifically include:

[0110] Step S51: For any row in the similarity matrix, calculate a first loss value corresponding to the row based on the elements on the diagonal of the row and the elements on the off-diagonal of the row; the first loss value is negatively correlated with the elements on the diagonal of the row, and the first loss value is positively correlated with the elements on the off-diagonal of the row.

[0111] Step S52: For any column in the similarity matrix, calculate a second loss value corresponding to the column based on the elements on the diagonal of the column and the elements on the off-diagonal of the column; the second loss value is negatively correlated with the elements on the diagonal of the column, and the second loss value is positively correlated with the elements on the off-diagonal of the column.

[0112] Specifically, the first loss value may be negatively correlated with the first proportion, which is used to characterize the ratio of the elements on the diagonal of the row to the sum of the elements on the non-diagonal of the row, and the second loss value may be negatively correlated with the second proportion, which is used to characterize the ratio of the elements on the diagonal of the column to the sum of the elements on the non-diagonal of the column. In this step, for any row, the ratio of the elements on the diagonal of the matrix in the row to the sum of all elements in the row can be calculated, or the ratio of the elements on the diagonal of the matrix in the row to the sum of the elements not on the diagonal of the matrix in the row can be calculated as the first proportion; the first loss value is calculated based on the first proportion, for example, the negative first proportion is directly used as the first loss value. The larger the first proportion, the smaller the first loss value can be. For any column, the ratio of the elements on the matrix diagonal in the column to the sum of all elements in the column, or the ratio of the elements on the matrix diagonal in the column to the sum of the elements not on the matrix diagonal in the column can be calculated as the second proportion; the second loss value is calculated based on the second proportion, for example, the negative second proportion is directly used as the second loss value. The larger the second proportion, the smaller the second loss value can be. Exemplarily, the cross entropy can be calculated separately by row and column based on a preset cross entropy loss function. The cross entropy corresponding to each row is the first loss value corresponding to each row, and the cross entropy corresponding to each column is the second loss value corresponding to each column. Of course, other functions can also be used to calculate the first loss value and the second loss value, and the embodiment of the present invention does not limit this.

[0113] Step S53: Determine the first target loss value based on the first loss value corresponding to each row and the second loss value corresponding to each column.

[0114] In this step, the first target loss value can be negatively correlated with the elements on the diagonal and positively correlated with the elements on the off-diagonal. The sum of all first loss values ​​and the sum of all second loss values ​​can be calculated to obtain a first sum and a second sum, and the ratio of the sum of the first sum and the second sum to 2N can be calculated. 2N can represent the total number of all first loss values ​​and all second loss values. In other words, the mean of the first loss values ​​and the second loss values ​​can be calculated as the first target loss value.

[0115] For example, in one implementation, the calculation method of the first target loss value can be expressed by the following formula:

[0116]

[0117] in, It can represent the first loss value, It can represent the second loss value.

[0118] In the embodiment of the present disclosure, since the elements on the diagonal of the similarity matrix represent the similarity between a pair of matching second speech samples and second text samples, that is, the elements on the diagonal can represent a correct pairing relationship, and the elements on the off-diagonal can represent an incorrect pairing relationship. Therefore, in the embodiment of the present disclosure, the first target loss value is calculated in such a way that the first loss value is negatively correlated with the elements on the diagonal of the similarity matrix in the row and positively correlated with the elements on the off-diagonal of the similarity matrix in the row, and the second loss value is negatively correlated with the elements on the diagonal of the similarity matrix in the column and positively correlated with the elements on the off-diagonal of the similarity matrix in the column. In this way, to a certain extent, in the process of determining the loss value, the similarity of the paired data on the diagonal can be maximized, and the similarity of the paired data on the non-diagonal can be minimized. When the first target loss value is large, that is, when matching semantic features cannot be generated for paired speech texts, the model parameters can be adjusted and optimized to a greater extent, thereby improving the model's ability to match at least two speech semantic features with at least two text semantic features respectively, and further improving the ability to generate matching semantic features for paired speech texts, thereby ensuring the model's subsequent intention recognition effect.

[0119] Figure 3 FIG. 1 is a schematic diagram of a training process according to an exemplary embodiment. Figure 3 As shown, for the paired second voice samples and second text samples, segmentation is performed, that is, batch segmentation can be performed first, that is, N second voice samples and N second text samples corresponding to the N second voice samples are divided. Then, the second voice samples and the second text samples can be input respectively to extract voice features and extract text features. The extracted voice features and text features are input into the voice extraction subnetwork and the text extraction subnetwork. Among them, the voice extraction subnetwork and the text extraction subnetwork can be the backbone network of the voice encoder and the backbone network of the text encoder respectively. For the obtained voice representation vector and text representation vector, the semantic similarity matrix is ​​calculated. Finally, the first target loss value is calculated, and the network parameters of the voice extraction subnetwork and the text extraction subnetwork are adjusted based on the first target loss value. In this process, the voice extraction subnetwork and the text extraction subnetwork both participate in the training. After training with a large amount of voice-text pair data, two encoders that can fully represent voice and text information are obtained.

[0120] In an embodiment of the present disclosure, in a constructed batch of data, the model can learn to accurately determine which speech and text data are paired. For example, the batch data has two pairs of audio-text pairs, namely audio A, audio B, text A, and text B. The similarity between audio A and text A, the similarity between audio A and text B, the similarity between audio B and text A, and the similarity between audio B and text B are calculated in pairs, respectively, to obtain four pairing relationships. Taking the label between audio and text with a correct pairing relationship as 1 and the label between audio and text with an incorrect pairing relationship as 0 as an example, the labels of audio A and text A and audio B and text B are 1, and the others are all 0. By training the speech extraction subnetwork and the text extraction subnetwork, the model can determine which paired labels are 1 and which paired labels are 0. That is, the parameter optimization in the first training stage can correspond to a classification problem. Cosine similarity is used as a way to associate two modal information. By using the cross-entropy loss function, the model learns to match the correct pairing relationship from a batch of data, thereby improving the ability to generate matching semantic features for paired speech and text, thereby improving processing efficiency in the batch training method.

[0121] Optionally, the step of training the speech intent recognition model based on the first speech sample with the intent label and the corresponding first text sample to obtain a trained speech intent recognition model may specifically include:

[0122] Step S61: Input the first speech sample and the first text sample into the speech extraction subnetwork and the text extraction subnetwork in the second multimodal extraction network respectively to extract semantic features, thereby obtaining target speech semantic features and target text semantic features.

[0123] Specifically, the implementation method of extracting the target speech semantic features and target text semantic features by the speech extraction subnetwork and the text extraction subnetwork can refer to the relevant description of extracting speech semantic features and text semantic features mentioned above, which will not be repeated here.

[0124] Step S62: Input the target speech semantic features and the target text semantic features into the intention recognition network to obtain the estimated speech intention.

[0125] Specifically, the target speech semantic features and the target text semantic features are spliced, and the spliced ​​features are input into the intent recognition network. After processing by the intent recognition network, the estimated speech intention can be obtained. Among them, the intent recognition network can be a multimodal speech intent recognition model. The input target speech semantic features and target text semantic features can be the speech representation vector and text representation vector obtained after processing. Specifically, the method of obtaining the speech representation vector and the text representation vector can refer to the above-mentioned related description and will not be repeated here.

[0126] Step S63: Determine a second target loss value based on the estimated speech intent and the intent label.

[0127] Step S64: Based on the second target loss value, adjust the network parameters of the intent recognition network, or adjust the network parameters of the intent recognition network, the speech extraction subnetwork, and the text extraction subnetwork in the second multimodal extraction network to obtain the trained speech intent recognition model.

[0128] Specifically, the method for calculating the second target loss value can be selected according to actual needs, and this disclosure does not impose any restrictions. For example, the similarity between the estimated speech intent and the intent label can be calculated, and the second target loss value can be determined based on the similarity, and the second target loss value is negatively correlated with the similarity. The specific parameter adjustment method can be selected as needed, and this disclosure does not impose any restrictions on this.

[0129] Since the speech extraction subnetwork and the text extraction subnetwork have a certain accuracy rate after pre-training, in the embodiment of the present disclosure, it is possible to choose to adjust only the network parameters of the intent recognition network as needed, thereby greatly reducing the amount of parameters to be adjusted and improving training efficiency. Alternatively, the network parameters of the speech extraction subnetwork, the text extraction subnetwork, and the intent recognition network obtained by pre-training can be adjusted simultaneously. In this way, although the amount of parameters to be adjusted is relatively large compared to the former, since the speech extraction subnetwork and the text extraction subnetwork have been pre-trained, the entire parameter adjustment calculation will be easier to converge, that is, the training time can be shortened compared to the case without pre-training, which also helps to improve training efficiency. As an example, when the determined second target loss value is not large (for example, less than a specified threshold), only the parameters of the intent recognition network can be adjusted. When the determined second target loss value is large (for example, not less than a specified threshold), the network parameters of the speech extraction subnetwork, the text extraction subnetwork, and the intent recognition network obtained by pre-training are adjusted simultaneously. Of course, other judgment conditions can also be configured, or the network to be adjusted parameters in the speech extraction subnetwork and the text extraction subnetwork can be selected as needed. This disclosure is not limited to this.

[0130] In the embodiment of the present disclosure, the speech extraction subnetwork and the text extraction subnetwork obtained in the first training phase can be loaded and imported, and after the import, the paired training data of the first speech sample with the intent label and the corresponding first text sample can be used to continue training. Specifically, the parameter values ​​of the network parameters in the trained first multimodal extraction network can be obtained. The values ​​of the network parameters in the second multimodal extraction network are set based on the parameter values ​​of the network parameters to achieve the import. Exemplarily, the parameter values ​​of the various network parameters in the trained first multimodal extraction network can be used to initialize the values ​​of the various network parameters in the second multimodal extraction network, so that the second multimodal extraction network inherits the network parameters of the trained first multimodal extraction network. In this way, by setting the values ​​of the network parameters in the second multimodal extraction network based on the parameter values ​​of the network parameters in the trained first multimodal extraction network, the second multimodal extraction network in the speech intent recognition model can achieve the effect of pre-training, thereby improving the overall training efficiency of the speech intent recognition model to a certain extent.

[0131] Figure 4 is another schematic diagram of a training process according to an exemplary embodiment, such as Figure 4 As shown, a first speech sample and a first text sample with an intent label can be input. Then, speech features and text features are extracted. Then, the imported pre-trained speech extraction subnetwork and text extraction subnetwork are input to obtain target speech semantic features and target text semantic features, respectively. Next, the target speech semantic features and target text semantic features are input into the intent recognition network to obtain an estimated speech intent. Based on the estimated speech intent and the intent label, a second target loss value is determined. The network parameters are adjusted based on the second target loss value.

[0132] It should be noted that in the embodiment of the present disclosure, since the pre-training of the speech extraction subnetwork and the text extraction subnetwork has been completed, when obtaining the first speech sample with the intent label and the corresponding first text sample, only a small amount of manually annotated data can be obtained, thereby reducing the data preparation cost.

[0133] An application scenario involved in the embodiments of the present disclosure is described below. At present, traditional voice intent recognition is mostly used in the field of human-computer interaction, for example, smart homes and mobile phone voice assistants. In these scenarios, the voice often contains simple instructional content, and the voice content itself is basically equivalent to the intention, and the specific expression changes relatively little. For example, the intention is "open the curtains", and the corresponding voice content can be "open the curtains", "open the curtains", and "open the curtains". In this scenario, traditional voice intent recognition methods often use a two-stage model. The first step is to convert the voice into text (ASR results) through the Automatic Speech Recognition (ASR) model. In the second step, the ASR results are input into the Natural Language Understanding (NLU) model to output predefined intent categories.

[0134] In live broadcast scenarios or intelligent customer service scenarios, the voices that need to be recognized (such as the host's explanation voice) are often long and declarative. The explanation content itself is not equivalent to the intent, and the same intent can have a large number of different specific expressions. For example, during a live broadcast, the content of the host's explanation can be divided into intentions such as creating an atmosphere, introducing products, and answering audience questions. Different intentions often indicate the current link. Live broadcast platforms in different links have different traffic distribution strategies for live broadcast rooms. Therefore, it is necessary to identify the host's explanation intentions. In one implementation method, an end-to-end model for speech content understanding (for example, an E2E-SLU model) is constructed directly based on an end-to-end approach. In this method, the speech signal is directly input and the intent category is output.

[0135] In the embodiment of the present disclosure, a two-stage multimodal method is adopted. The speech is first converted into text, and both the speech and text are input into the speech intent recognition model, so that the speech intent recognition model outputs the intent category based on the speech and text. In this way, the two modal information of speech and text can be fully utilized, thereby improving the recognition effect to a certain extent. In the embodiment of the present disclosure, a two-stage training method is adopted. In the first stage, speech-text data with lower acquisition difficulty is used to pre-train the speech extraction subnetwork and the text extraction subnetwork. Since the speech extraction subnetwork and the text extraction subnetwork have been trained in the first stage, the speech intent recognition model can be trained in the second stage using a small amount of intent-speech and text data with higher acquisition difficulty. Compared with the method of directly using training data with intent labels for training, it can reduce the dependence on manually annotated data to a certain extent, reduce the amount of training data with intent labels required, and thus reduce the data cost required for training to a certain extent, reduce the difficulty of implementing model training, and improve the feasibility of model training.

[0136] By reducing the reliance on manually labeled data, it can, to a certain extent, solve the problem that the accuracy of intent recognition is heavily dependent on the amount of manually labeled training data, while the cost of manually labeled training data is high, making it difficult to guarantee the training effect.

[0137] In an embodiment of the present disclosure, a method for speech intent recognition is also provided. The method can be executed after completing the above-mentioned training process. The method can be implemented in a terminal device such as a smartphone, a tablet computer, or a personal computer (PC), or in a device such as a server.

[0138] Specifically, in this method, a speech to be recognized can be obtained. The speech to be recognized is the speech whose intention needs to be recognized, and each speech to be recognized can be the speech of a sentence. The speech to be recognized and the text corresponding to the speech to be recognized are input into the speech intention recognition model to obtain an estimated speech intention. Among them, the speech intention recognition model is trained using the above-mentioned training method of the speech intention recognition model, so the speech intention recognition method has all the beneficial effects of the training method of the speech intention recognition model of the exemplary embodiment of the present invention, which will not be repeated here. It should be understood that when the speech intention recognition model performs calculations, the speech extraction subnetwork will first extract the speech semantic features of the speech to be recognized, and the text extraction subnetwork will extract the text semantic features of the text corresponding to the speech to be recognized, and then the intention recognition network will calculate the estimated intention.

[0139] Figure 5 FIG. 1 is a block diagram of a training device for a speech intent recognition model according to an exemplary embodiment. Figure 5 As shown, the device 20 may include:

[0140] A first acquisition module 201 is configured to acquire a first training sample set, where the first training sample set includes a first speech sample with an intent label and a corresponding first text sample;

[0141] A second acquisition module 202 is configured to acquire a trained first multimodal extraction network, wherein the first multimodal extraction network includes at least a speech extraction subnetwork for extracting speech semantic features and a text extraction subnetwork for extracting text semantic features, and the first multimodal extraction network is trained using a second training sample set consisting of a second speech sample and a corresponding second text sample;

[0142] The training module 203 is configured to train the speech intent recognition model based on the first speech sample with the intent label and the corresponding first text sample to obtain a trained speech intent recognition model;

[0143] Among them, the speech intention recognition model includes a second multimodal extraction network and an intention recognition network. The second multimodal extraction network has the same network structure as the first multimodal extraction network and inherits the network parameters of the trained first multimodal extraction network. The intention recognition network obtains an estimated speech intention based on the modal features extracted by the second multimodal extraction network.

[0144] In an optional embodiment, the second acquisition module 202 is specifically configured to execute:

[0145] Inputting N second speech samples into the speech extraction subnetwork for semantic feature extraction to obtain N speech semantic features, and inputting N second text samples corresponding to the N second speech samples into the text extraction subnetwork for semantic feature extraction to obtain N text semantic features; wherein N is a positive integer greater than or equal to 2;

[0146] Calculating the semantic similarity between each of the N speech semantic features and each of the N text semantic features;

[0147] Based on the semantic similarity, network parameters of the speech extraction subnetwork and the text extraction subnetwork are adjusted to obtain the trained first multimodal extraction network.

[0148] In an optional embodiment, the second acquisition module 202 is further configured to execute:

[0149] Based on the semantic similarity, construct a similarity matrix; wherein the number of rows and columns of the similarity matrix are both N, an element in the similarity matrix represents the similarity between a first semantic feature corresponding to the row where the element is located and a second semantic feature corresponding to the column where the element is located, and one of the first semantic feature and the second semantic feature is selected as the speech semantic feature and the other is the text semantic feature;

[0150] Calculating a first target loss value according to each row element and each column element in the similarity matrix;

[0151] The network parameters of the speech extraction subnetwork and the text extraction subnetwork are adjusted according to the first target loss value.

[0152] In an optional embodiment, the second acquisition module 202 is further configured to execute:

[0153] For any row in the similarity matrix, calculate a first loss value corresponding to the row based on the elements on the diagonal of the row and the elements on the off-diagonal of the row; the first loss value is negatively correlated with the elements on the diagonal of the row, and the first loss value is positively correlated with the elements on the off-diagonal of the row;

[0154] For any column in the similarity matrix, calculate a second loss value corresponding to the column based on the elements on the diagonal of the column and the elements on the off-diagonal of the column; the second loss value is negatively correlated with the elements on the diagonal of the column, and the second loss value is positively correlated with the elements on the off-diagonal of the column;

[0155] The first target loss value is determined based on the first loss value corresponding to each of the rows and the second loss value corresponding to each of the columns.

[0156] In an optional embodiment, the device 20 further includes:

[0157] A pooling module is configured to, before the second acquisition module 202 calculates the semantic similarity between each of the N speech semantic features and each of the N text semantic features, perform pooling processing on the speech semantic features of any of the N second speech samples in the time dimension to obtain a speech representation vector of the speech semantic features;

[0158] The second acquisition module 202 is further configured to execute: calculating the semantic similarity between the speech representation vector of each speech semantic feature and the text representation vector of each text semantic feature.

[0159] In an optional embodiment, the training module 203 is specifically configured to execute:

[0160] Inputting the first speech sample and the first text sample into the speech extraction subnetwork and the text extraction subnetwork in the second multimodal extraction network respectively to extract semantic features, thereby obtaining target speech semantic features and target text semantic features;

[0161] Inputting the target speech semantic features and the target text semantic features into the intention recognition network to obtain the estimated speech intention;

[0162] Determining a second target loss value according to the estimated speech intent and the intent label;

[0163] Based on the second target loss value, adjust the network parameters of the intent recognition network, or adjust the network parameters of the intent recognition network, the speech extraction subnetwork and the text extraction subnetwork in the second multimodal extraction network to obtain the trained speech intent recognition model.

[0164] Optionally, the device 20 also includes: a third acquisition module, configured to execute training of the speech intent recognition model based on the first speech sample with the intent label and the corresponding first text sample in the training module 203, and before obtaining the trained speech intent recognition model, execute acquiring the parameter values ​​of the network parameters in the trained first multimodal extraction network; a setting module, configured to execute setting the values ​​of the network parameters in the second multimodal extraction network based on the parameter values ​​of the network parameters.

[0165] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0166] According to one embodiment of the present disclosure, an electronic device is provided, comprising: a processor and a memory for storing processor-executable instructions, wherein the processor is configured to implement the steps in the training method of the speech intent recognition model in any of the above embodiments when executed.

[0167] According to one embodiment of the present disclosure, a computer-readable storage medium is also provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can perform the steps in the training method of the speech intent recognition model in any of the above embodiments.

[0168] According to one embodiment of the present disclosure, a computing device program product is also provided, which includes readable program instructions. When the readable program instructions are executed by a processor of an electronic device, the electronic device is able to perform the steps in the training method of the speech intent recognition model in any of the above embodiments.

[0169] Figure 67 is a block diagram of an apparatus for training a speech intent recognition model according to an exemplary embodiment. The apparatus 700 may include a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 710, an input / output interface 712, a sensor component 714, a communication component 716, and a processor 720. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method for training the speech intent recognition model. In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 704 including instructions, and the above-mentioned instructions can be executed by the processor 720 of the apparatus 700 to complete the above-mentioned method. Optionally, the storage medium may be a non-temporary computing device readable storage medium, for example, the non-temporary computing device readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0170] Figure 7 8 is a block diagram of another apparatus for training a speech intent recognition model according to an exemplary embodiment. The apparatus 800 may include a processing component 822, a memory 832, an input / output interface 858, a network interface 850, and a power supply component 826. The apparatus 800 may be provided as a server. The application program stored in the memory 832 may include one or more modules, each of which corresponds to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the above-mentioned speech intent recognition model training method.

[0171] The user information (including but not limited to the user's device information, user personal information, etc.) and related data involved in this disclosure are all information authorized by the user or authorized by all parties.

[0172] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0173] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a speech intent recognition model, characterized in that: The method comprises: Obtaining a first training sample set, where the first training sample set includes a first speech sample with an intent label and a corresponding first text sample; Obtaining a trained first multimodal extraction network, the first multimodal extraction network comprising at least a speech extraction subnetwork for extracting speech semantic features and a text extraction subnetwork for extracting text semantic features, the first multimodal extraction network being trained using a second training sample set consisting of a second speech sample and a corresponding second text sample; Training a speech intent recognition model based on the first speech sample with the intent label and the corresponding first text sample to obtain a trained speech intent recognition model; Among them, the speech intention recognition model includes a second multimodal extraction network and an intention recognition network. The second multimodal extraction network has the same network structure as the first multimodal extraction network and inherits the network parameters of the trained first multimodal extraction network. The intention recognition network obtains an estimated speech intention based on the modal features extracted by the second multimodal extraction network.

2. The method according to claim 1, characterized in that The obtaining of a trained first multimodal extraction network includes: Inputting N second speech samples into the speech extraction subnetwork for semantic feature extraction to obtain N speech semantic features, and inputting N second text samples corresponding to the N second speech samples into the text extraction subnetwork for semantic feature extraction to obtain N text semantic features; wherein N is a positive integer greater than or equal to 2; Calculating the semantic similarity between each of the N speech semantic features and each of the N text semantic features; Based on the semantic similarity, network parameters of the speech extraction subnetwork and the text extraction subnetwork are adjusted to obtain the trained first multimodal extraction network.

3. The method according to claim 2, characterized in that The adjusting the network parameters of the speech extraction subnetwork and the text extraction subnetwork based on the semantic similarity includes: Based on the semantic similarity, construct a similarity matrix; wherein the number of rows and columns of the similarity matrix are both N, an element in the similarity matrix represents the similarity between a first semantic feature corresponding to the row where the element is located and a second semantic feature corresponding to the column where the element is located, and one of the first semantic feature and the second semantic feature is selected as the speech semantic feature and the other is the text semantic feature; Calculating a first target loss value according to each row element and each column element in the similarity matrix; The network parameters of the speech extraction subnetwork and the text extraction subnetwork are adjusted according to the first target loss value.

4. The method according to claim 3, characterized in that The calculating the first target loss value according to each row element and each column element in the similarity matrix includes: For any row in the similarity matrix, calculate a first loss value corresponding to the row based on the elements on the diagonal of the row and the elements on the off-diagonal of the row; the first loss value is negatively correlated with the elements on the diagonal of the row, and the first loss value is positively correlated with the elements on the off-diagonal of the row; For any column in the similarity matrix, calculate a second loss value corresponding to the column based on the elements on the diagonal of the column and the elements on the off-diagonal of the column; the second loss value is negatively correlated with the elements on the diagonal of the column, and the second loss value is positively correlated with the elements on the off-diagonal of the column; The first target loss value is determined based on the first loss value corresponding to each of the rows and the second loss value corresponding to each of the columns.

5. The method according to any one of claims 2 to 4, characterized in that: Before calculating the semantic similarity between each of the N speech semantic features and each of the N text semantic features, the method further includes: For any second speech sample among the N second speech samples, performing pooling processing on the speech semantic features of the second speech sample in the time dimension to obtain a speech representation vector of the speech semantic features; The calculating of the semantic similarity between each of the N speech semantic features and each of the N text semantic features comprises: The semantic similarity between the speech representation vector of each of the speech semantic features and the text representation vector of each of the text semantic features is calculated.

6. The method according to claim 1, characterized in that The method of training a speech intent recognition model based on the first speech sample with the intent label and the corresponding first text sample to obtain a trained speech intent recognition model includes: Inputting the first speech sample and the first text sample into the speech extraction subnetwork and the text extraction subnetwork in the second multimodal extraction network respectively to extract semantic features, thereby obtaining target speech semantic features and target text semantic features; Inputting the target speech semantic features and the target text semantic features into the intention recognition network to obtain the estimated speech intention; Determining a second target loss value according to the estimated speech intent and the intent label; Based on the second target loss value, adjust the network parameters of the intent recognition network, or adjust the network parameters of the intent recognition network, the speech extraction subnetwork and the text extraction subnetwork in the second multimodal extraction network to obtain the trained speech intent recognition model.

7. A training device for a speech intention recognition model, characterized in that: The device comprises: A first acquisition module is configured to acquire a first training sample set, where the first training sample set includes a first speech sample with an intent label and a corresponding first text sample; A second acquisition module is configured to execute acquisition of a trained first multimodal extraction network, wherein the first multimodal extraction network includes at least a speech extraction subnetwork for extracting speech semantic features and a text extraction subnetwork for extracting text semantic features, and the first multimodal extraction network is trained using a second training sample set consisting of a second speech sample and a corresponding second text sample; A training module is configured to train a speech intent recognition model based on the first speech sample with the intent label and the corresponding first text sample to obtain a trained speech intent recognition model; Among them, the speech intention recognition model includes a second multimodal extraction network and an intention recognition network. The second multimodal extraction network has the same network structure as the first multimodal extraction network and inherits the network parameters of the trained first multimodal extraction network. The intention recognition network obtains an estimated speech intention based on the modal features extracted by the second multimodal extraction network.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 6.

10. A computing device program product, characterized in that The computing device program product comprises readable program instructions, which, when executed by a processor of an electronic device, enable the electronic device to perform the method according to any one of claims 1 to 6.

Images