Speech text alignment model training method and device, electronic equipment and storage medium

Abstract

The application provides a speech text alignment model training method and device, electronic equipment and a storage medium, and relates to the field of large models. The method comprises the following steps: obtaining a pre-training model and text data, and performing time information prediction on each word in the text data based on the pre-training model to obtain a time prediction result corresponding to each word; obtaining speech data, and performing text reconstruction on the speech data based on the time prediction result to obtain reconstructed text data; performing optimal alignment path prediction on the reconstructed text data and the speech data based on a dynamic programming algorithm to obtain an optimal alignment path; and adjusting parameters of the pre-training model based on the optimal alignment path to obtain a speech text alignment model. The application solves the problem of low applicability of a speech text alignment model in related technologies.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model, in particular, the present application relates to a speech text alignment model training method and device, electronic equipment and storage medium. BACKGROUND

[0002] In recent years, multi-modal human-computer dialogue has attracted widespread social attention, especially speech text multi-modal dialogue. Through a speech text alignment model, specific speech text multi-modal downstream task design can be achieved, such as speech to text translation task and speech understanding task.

[0003] However, traditional word-level alignment technology usually converts text into phoneme sequences and aligns based on these phonemes. This method largely ignores the semantic information in the text, resulting in an inability to achieve ideal results in speech text multi-modal dialogue tasks. Existing word-level alignment models such as mask modeling and cross-modal contrastive learning are mostly designed for specific downstream tasks, making it difficult to maintain the same performance in a wider range of speech text tasks. For example, in specific tasks such as speech to text translation or speech understanding, existing technologies may perform well, but they cannot maintain the same level of performance in other speech text tasks, which limits the model's versatility and applicability. And in the pre-training process of the current word-level alignment model, it relies heavily on word-level annotation data, which requires high-cost and time-consuming word-level alignment annotation, and it is extremely difficult to obtain enough word-level annotation data, which cannot fully utilize a large amount of available unlabeled data, limiting its application in a wider range of scenarios.

[0004] From the above, how to improve the applicability of the speech text alignment model training method remains to be solved. SUMMARY

[0005] The present application provides a speech text alignment model training method, device, electronic equipment and storage medium, which can solve the problem of low applicability of the speech text alignment model training method in related technologies. The technical solution is as follows:

[0006] According to one aspect of the present application, a speech text alignment model training method is provided, characterized in that it comprises:

[0007] Obtaining a pre-training model and text data, and predicting time information for each word in the text data based on the pre-training model to obtain time prediction results corresponding to each word;

[0008] Obtaining speech data, and reconstructing text data based on the time prediction results to obtain reconstructed text data;

[0009] The optimal alignment path is obtained by predicting an optimal alignment path of the reconstructed text data and the speech data based on a dynamic programming algorithm.

[0010] The speech text alignment model is obtained by adjusting parameters of the pre-training model based on the optimal alignment path.

[0011] According to an aspect of the present application, a speech text alignment model training device comprises:

[0012] An information prediction module is configured to obtain a pre-training model and text data, and predict time information of each word in the text data based on the pre-training model to obtain a time prediction result corresponding to each word;

[0013] A reconstruction module is configured to obtain speech data, and reconstruct text based on the time prediction result to obtain reconstructed text data;

[0014] A path prediction module is configured to predict an optimal alignment path of the reconstructed text data and the speech data based on a dynamic programming algorithm to obtain the optimal alignment path;

[0015] A parameter adjustment module is configured to adjust parameters of the pre-training model based on the optimal alignment path to obtain a speech text alignment model.

[0016] In an exemplary embodiment, the pre-training model comprises a text encoder and a fully connected layer.

[0017] The information prediction module comprises:

[0018] A text feature vector unit is configured to input the text data into the text encoder to obtain a text feature vector corresponding to each word in the text data;

[0019] A duration prediction unit is configured to input the text feature vector into the fully connected layer to predict a length proportion of each word relative to the entire text data;

[0020] A time prediction result unit is configured to determine a start time and an end time of each word based on the length proportion corresponding to each word to generate a time prediction result corresponding to each word.

[0021] In an exemplary embodiment, the text data comprises a text embedding and an absolute position embedding.

[0022] The text feature vector unit comprises:

[0023] A text paragraph embedding subunit is configured to perform sequence division processing on the text data to obtain a text paragraph embedding.

[0024] a text feature vector unit, configured to perform summation processing on the text embedding, the absolute position embedding, and the text passage embedding, and input the text encoder, to obtain a text feature vector corresponding to each word.

[0025] In an example embodiment, the pre-training model comprises a speech encoder and a fully connected layer.

[0026] The reconstruction module comprises:

[0027] a speech feature vector unit, configured to input the speech data into the speech encoder, to obtain a speech feature vector corresponding to each speech segment in the speech data;

[0028] a word prediction unit, configured to perform word prediction on each speech feature vector based on the fully connected layer, to obtain a word corresponding to each speech feature vector, and generate a word prediction result;

[0029] a reconstructed text data unit, configured to perform passage division on the word prediction result based on the time prediction result, to obtain reconstructed text data.

[0030] In an example embodiment, the speech data comprises a speech embedding.

[0031] The speech feature vector unit comprises:

[0032] a speech passage embedding sub-unit, configured to perform sequence division processing on the speech data, to obtain a speech passage embedding;

[0033] a speech feature vector sub-unit, configured to perform summation processing on the speech passage embedding and the speech embedding, and input a self-attention layer in the speech encoder, to obtain a speech feature vector corresponding to each speech segment in the speech data.

[0034] In an example embodiment, the path prediction module comprises:

[0035] a probability distribution unit, configured to perform normalization processing on the reconstructed text data, to obtain a probability distribution of each word corresponding to each speech feature vector;

[0036] a total probability unit, configured to calculate a total probability of the speech data for each alignment path based on a time sequence relationship of each speech feature vector and the probability distribution, in a dynamic programming algorithm, and determine an alignment path with the highest total probability as an optimal alignment path.

[0037] In an example embodiment, the parameter adjustment module comprises:

[0038] a path backtracking unit, configured to perform path backtracking on the optimal alignment path, to obtain a length proportion corresponding to each word in the reconstructed text data.

[0039] a consistency loss unit configured to perform divergence calculation based on a length proportion of each word in the reconstructed text data relative to the entire reconstructed text data and a length proportion of each word in the text data relative to the entire text data, to obtain a consistency loss;

[0040] a parameter adjustment unit configured to adjust model parameters of the pre-training model based on the consistency loss, and repeat the training process until the pre-training model meets a preset pre-training target, to obtain a speech-text alignment model.

[0041] According to an aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory has stored thereon computer readable instructions; the computer readable instructions are executed by one or more processors, so that the electronic device implements the speech-text alignment model training method as described above.

[0042] According to an aspect of the present application, a storage medium has stored thereon computer readable instructions, which are executed by one or more processors to implement the speech-text alignment model training method as described above.

[0043] According to an aspect of the present application, a computer program product includes computer readable instructions stored in a storage medium, and one or more processors of an electronic device read the computer readable instructions from the storage medium, load and execute the computer readable instructions, so that the electronic device implements the speech-text alignment model training method as described above.

[0044] The technical scheme provided by the present application has the beneficial effects that:

[0045] In the above technical scheme, the time information of each word in the text data is predicted, and the text data is reconstructed from the speech data, and the optimal alignment path of the reconstructed text data and the speech data is found by using a dynamic programming algorithm, and finally a speech-text alignment model is obtained. The time alignment prediction of the speech-text alignment model is gradually optimized by using the dynamic programming algorithm, which not only further improves the prediction accuracy of the model, but also enhances the fine-grained understanding of the model to the input data. At the same time, a large amount of unlabeled text data and speech data are used for training, which breaks the dependence on labeled data in the pre-training process, and significantly improves the accuracy and applicability of the model in the speech-text downstream task.

[0046] Thus, the problem of low applicability of the speech-text alignment model training method in the related art can be effectively solved. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0048] Figure 1 is a schematic diagram according to the implementation environment involved in the present application;

[0049] Figure 2 is a flowchart of a speech text alignment model training method according to an exemplary embodiment;

[0050] Figure 3 is Figure 2 Step 210 in the corresponding embodiment is a flowchart of one embodiment;

[0051] Figure 4a is Figure 3 Step 211 in the corresponding embodiment is a flowchart of one embodiment;

[0052] Figure 4b is a schematic diagram of a text encoder according to an exemplary embodiment;

[0053] Figure 5 is Figure 2 Step 230 in the corresponding embodiment is a flowchart of one embodiment;

[0054] Figure 6a is Figure 5 Step 231 in the corresponding embodiment is a flowchart of one embodiment;

[0055] Figure 6b is a schematic diagram of a speech encoder according to an exemplary embodiment;

[0056] Figure 7 is Figure 2 Step 250 in the corresponding embodiment is a flowchart of one embodiment;

[0057] Figure 8 is Figure 2 Step 270 in the corresponding embodiment is a flowchart of one embodiment;

[0058] Figure 9 is a specific implementation schematic diagram of a speech text alignment model training method in an application scenario;

[0059] Figure 10 is a structural block diagram of a speech text alignment model training method device according to an exemplary embodiment;

[0060] Figure 11 is a hardware structure diagram of a server according to an exemplary embodiment;

[0061] Figure 12 is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] Embodiments of the present application are described in detail below with reference to examples illustrated in the accompanying drawings, in which the same or similar components or components having the same or similar functions are denoted by the same or similar reference numerals throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are for the purpose of explanation only, and cannot be interpreted as limiting the present application.

[0063] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an", "said" and "the" used herein also include the plural forms. It should be further understood that the use of the phrase "comprising" in the specification of the present disclosure means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0064] As described previously, in the existing speech text alignment model training method, the difference between the data sets is detected, the calculation is complex in the leakage detection process, and the detection effect is poor when there is out-of-order problem in the test data, so the accuracy is low.

[0065] As can be seen from the above, the related art still has the defect that the applicability of the speech text alignment model training method is not high.

[0066] Therefore, the speech text alignment model training method provided by the present application can effectively improve the accuracy of the speech text alignment model training method. Accordingly, the speech text alignment model training method is applicable to a speech text alignment model training method device, which can be deployed on an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, for example, the computer device includes a desktop computer, a notebook computer, a server, etc. The electronic device can also be an electronic device with a central control function, for example, the electronic device includes a gateway, etc. The electronic device can also be a portable mobile electronic device, for example, the electronic device includes a smart phone, a tablet computer, etc.

[0067] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0068] Figure 1 A schematic diagram of an implementation environment involved in an image processing method. It should be noted that this implementation environment is only an example adapted to the present application and should not be considered as providing any limitation on the scope of use of the present application.

[0069] The implementation environment includes a collection end 110 and a service end 130.

[0070] Specifically, the collection end 110 can be considered as an information collection device, including but not limited to a camera, a positioning instrument and other electronic devices with information collection functions, and can also be considered as an integrated device of an information collection and speech text alignment model training method, including but not limited to a notebook computer and a desktop computer and other electronic devices with both information collection functions and speech text alignment model training method functions. That is, in some embodiments, the collection end 110 can collect text data and speech data.

[0071] The service end 130 can also be considered as a speech text alignment model training device, including but not limited to a smart phone, a desktop computer, a notebook computer, a server and other electronic devices with speech text alignment model training method functions, and can also be a computer cluster composed of multiple servers, or even a cloud computing center composed of multiple servers. Among them, the service end 130 is used to provide background services, for example, the background services include but are not limited to speech text alignment model training services and the like.

[0072] The service end 130 and the collection end 110 pre-establish a network communication connection through wired or wireless and the like, and realize data transmission between the service end 130 and the collection end 110 through the network communication connection. The transmitted data includes but is not limited to text data and speech data and the like.

[0073] In an application scenario, through the interaction between the collection end 110 and the service end 130, the collection end 110 collects text data and speech data, and uploads the text data and the speech data to the service end 130 to request the service end 130 to provide speech text alignment model training services.

[0074] At this time, for the service end 130, after receiving the text data and the speech data uploaded by the collection end 110, the speech text alignment model training service is called to train the speech text alignment model according to the text data and the speech data, solving the problem of low applicability of the speech text alignment model training method in the related art.

[0075] Please refer to Figure 2The embodiment of the present application provides a speech text alignment model training method, which is suitable for an electronic device Figure 1 The server 130 in the shown implementation environment can also be a desktop computer, a notebook computer, a server, etc.

[0076] In the following method embodiment, for the convenience of description, the execution subject of each step of the method is taken as an example for description, but this does not constitute a specific limitation.

[0077] As shown in the method, the method can include the following steps: Figure 2

[0078] In step 210, a pre-training model and text data are acquired, and time information prediction is performed on each word in the text data based on the pre-training model, to obtain a time prediction result corresponding to each word.

[0079] The text data includes text information that needs to be aligned with speech, and text embedding and absolute position embedding generated from the text information. The pre-training model is a preset model that needs to be trained for text-speech alignment capability. The text alignment model can be obtained by training the pre-training model.

[0080] Specifically, the pre-training model includes a text encoder, a speech encoder, a full connection layer and a modal fusion layer. The text encoder is used to encode the text data in the input data to obtain a text feature vector. The speech encoder is used to encode the speech data in the input data to obtain a speech modal feature vector. The modal fusion layer is used to cross-modal information fusion of the text modal feature vector and the speech modal feature vector to obtain a mixed modal feature vector used for pre-training task or downstream prediction task in dialogue understanding. The full connection layer is used for text and speech prediction of the input data.

[0081] In a possible implementation, the pre-training model is a SPECTRA model.

[0082] In step 230, speech data is acquired, and text reconstruction is performed on the speech data based on the time prediction result, to obtain reconstructed text data.

[0083] The speech data includes unannotated speech information corresponding to the text data, which needs to be aligned with speech, and speech embedding generated from the speech information. Through text reconstruction, the speech data can be reconstructed into text data, and then the text content corresponding to the speech data can be aligned and predicted.

[0084] In a possible implementation, the speech data is a speech waveform segment collected in a multi-turn dialogue, and the text data is a corresponding text sentence. Each word in the text sentence can be decomposed into a sub-word sequence.​

[0085] Step 250, based on the dynamic programming algorithm, the optimal alignment path of the reconstructed text data and the speech data is predicted, and the optimal alignment path is obtained.

[0086] Wherein, the optimal alignment path is calculated according to the dynamic programming algorithm, and the text reconstruction loss of the speech data is minimized.

[0087] Step 270, based on the optimal alignment path, the parameters of the pre-trained model are adjusted to obtain a speech-text alignment model.

[0088] Wherein, the pre-trained model is continuously adjusted by the optimal alignment path to realize the optimization of the pre-trained model, so that the pre-trained model is trained in the direction of improving the quality of the speech-text alignment result generation, and finally a speech-text generation model is obtained.

[0089] Through the above process, the time alignment prediction of the speech-text alignment model is gradually optimized by the dynamic programming algorithm, which not only further improves the prediction accuracy of the model, but also enhances the fine-grained understanding of the input data. At the same time, a large amount of unlabeled text data and speech data are used for training, which can realize word-level alignment through word-level alignment annotated parallel corpus, greatly reducing the cost and time of data preparation, breaking the dependence on annotated data in the pre-training process, and significantly improving the accuracy and applicability of the model in the speech-text alignment downstream task.

[0090] In an example embodiment, as shown in Figure 3 Step 210 can include the following steps:

[0091] Step 211, input the text data into the text encoder to obtain the text feature vector corresponding to each word in the text data.

[0092] In an example embodiment, as shown in Figure 4a Step 211 can include the following steps:

[0093] Step 2111, sequence division processing is performed on the text data to obtain text paragraph embedding.

[0094] Specifically, the text data is divided into sequences according to the interval of the dialogue in the text data, and the text paragraph embedding is added to the text embedding in the text data to complete the sequence division processing of the text data.

[0095] In a possible implementation, special markers are used to divide the sequences in the text data and segment the dialogue. For example, markers <s>Indicates the start of a sequence, set flag< / s> are set in the text data to represent the end of each round of dialogue. At this time, the learnable paragraph embedding e A and eB , will e B Add to and e The text paragraph embedding is obtained above, and the dialogue intervals in the text data can be distinguished through the text paragraph embedding. It is t i And the text embedding of the last tag.

[0096] Step 2113: Summate the text embedding, absolute position embedding, and text paragraph embedding and input them into the text encoder to obtain the text feature vectors of each word.

[0097] Specifically, the text embedding, absolute position embedding, and paragraph embedding are summed and then input into the text encoder to obtain the text feature vector, which is used to indicate the hidden state of the text data in the hidden space.

[0098] In one possible implementation, such as Figure 4b As shown, the text encoder is a RoBERTa model. The output hidden state of the RoBERTa model is obtained by inputting text embeddings, absolute position embeddings, and paragraph embeddings into the RoBERTa model. in Represents the output hidden state of RoBERTa, where n t It is the text input length, and d h It is the hidden state dimension.

[0099] Through the above process, the text data is distinguished by sequence segmentation, which improves the accuracy and efficiency of speech-text alignment.

[0100] Step 213: Input the text feature vector into the fully connected layer to predict the duration of each word in the text data and obtain the length ratio of each word relative to the overall text data.

[0101] In one possible implementation, a matrix consisting of the text hidden representations of the first sub-word of each word is used as a text feature vector to predict the duration of each word in the text data.

[0102] The formula is as follows:

[0103]

[0104] in, This indicates the proportion of each word in the total length of the text data.

[0105] It is a matrix composed of the text-hidden representations of the first subword of each word.

[0106] Step 215, based on the length proportion of each word, the start time and the end time of each word are determined, and the time prediction result corresponding to each word is generated.

[0107] It should be noted that by obtaining the start time and the end time of each word in the text data, each word is aligned one by one, for example, the first word of the speech data starts from 0.0 seconds, and ends with the last word. Each word is closely connected to the next word without overlap or gap, and each word lasts at least 0.1 seconds. By the time sequence of each word in the text data, after determining the alignment of the first word and the speech data, the corresponding relationship between the subsequent words and the speech data is determined one by one and aligned.

[0108] In one possible implementation, after obtaining the time proportion of each word, the length of each word is obtained by multiplying the proportion of each word in the total length of the text data by the speech duration of the text information, and then the start time and the end time of each word when arranged in time sequence are calculated.

[0109] Wherein, the start time and the end time of each word are calculated as follows:

[0110]

[0111] Wherein, w ij is a word, s ij is the start time, e ij is the end time.

[0112] Through the above process, the full connection layer is used to predict the duration of each word, so as to ensure that the time alignment prediction of the speech text alignment model maintains monotonicity, and improve the alignment accuracy and efficiency of the speech text model.

[0113] In an exemplary embodiment, as shown in Figure 5 Step 230 can include the following steps:

[0114] Step 231, inputting the speech data into the speech encoder to obtain the speech feature vector corresponding to each speech segment in the speech data.

[0115] In one possible implementation, the speech encoder is a WavLM structure including an additional convolutional layer.

[0116] In an exemplary embodiment, as shown in Figure 6a Step 231 can include the following steps:

[0117] Step 2311, performing sequence division processing on the speech data to obtain a speech paragraph embedding.

[0118] The sequence division processing is performed on the speech data, different speech sequences in the speech data are distinguished, and the speech sequence division processing is completed by adding the speech paragraph embedding to the speech embedding in the speech data.

[0119] In a possible implementation, the sequence division processing is performed by using [CLS] and [SEP] marks to mark the start and end of the speech sequence embedding.

[0120] In step 2313, the speech paragraph embedding and the speech embedding are summed and input into a self-attention layer in the speech encoder to obtain speech feature vectors corresponding to each speech segment in the speech data.

[0121] Specifically, the speech encoder is as shown in the figure. Figure 6b The additional convolutional layer is a convolutional layer added on the basis of WavLM, Conv represents a convolutional layer, Projection represents a projection layer, and Self-Attention represents a self-attention layer. The speech input waveform is input into the speech encoder. The output of the projection layer is connected with the speech data after the sequence division processing, and the speech paragraph embedding (e A , e B ) is added to the summed speech embedding. Then, the self-attention layer is input to generate , wherein

[0122] represents the output hidden state of the last self-attention layer, n s is the length of the speech embedding.

[0123] Through the above process, the corresponding text paragraph embedding and speech paragraph embedding are added to the output of the text encoder and the speech encoder, so that the text data and the speech data can be distinguished. Finally, the two modalities are effectively integrated by using the self-attention layer, and the speech text alignment model is universal in multi-modal.

[0124] In step 233, word prediction is performed on each speech feature vector based on a full connection layer to obtain words corresponding to each speech feature vector, and a word prediction result is generated.

[0125] It should be noted that when the speech feature vector is predicted, the text information needs to be isolated, and therefore the text information needs to be masked by adding marks, and the speech feature vector is input into the full connection layer for prediction.

[0126] In a possible implementation, the prediction formula is as follows:

[0127]

[0128] , wherein

[0129] wherein U i is the predicted value of the first subword of the text corresponding to each speech feature vector V is the vocabulary parameter, d h is the size of the speech feature vector.

[0130] At step 235, the word prediction results are paragraph-divided based on the time prediction results to obtain reconstructed text data.

[0131] Specifically, the text data present in the word prediction results is divided by using the predicted start time and end time of the word, so as to align the prediction results of the word with the corresponding word.

[0132] It should be noted that after the text reconstruction, the model parameters in the pre-trained model except the parameters of the full connection layer are fixed, and the cross-entropy loss is used to perform the reverse gradient update on the parameters of the full connection layer, so as to revise the word prediction results.

[0133] Through the above process, the text data is independently predicted only by relying on the speech data, so as to ensure the alignment capability of the speech text alignment model, and through the paragraph division of the word prediction results, a closer alignment is achieved, and the speech text alignment effect is improved.

[0134] In an exemplary embodiment, as shown in FIG. 2, step 250 can include the following steps: Figure 7

[0135] At step 251, the reconstructed text data is normalized to obtain the probability distribution of each word corresponding to each speech feature vector.

[0136] The probability distribution is obtained by normalization processing through a softmax function.

[0137] Specifically, by extracting the subword sequence corresponding to each word in the reconstructed text data from the word prediction results, the probability of each speech feature vector corresponding to the initial subword in the subword sequence in the reconstructed text data is determined, and the probability distribution of each speech feature vector is generated.

[0138] The probability distribution formula of each speech feature vector is as follows:

[0139]

[0140] wherein F(x, y) represents the highest probability that the first x speech feature vectors correspond to the first y text words of the sentence. i

[0141] ​​Step 253, based on the time sequence relationship and probability distribution of each speech feature vector, the total probability of the speech data for each alignment path is calculated by a dynamic programming algorithm, and the alignment path with the highest total probability is determined as the optimal alignment path.

[0142] Wherein, since each speech feature vector has a time sequence, after determining the word corresponding to the current speech feature vector, the word corresponding to the next speech feature vector can be determined based on the time sequence of the speech feature vector, which is a word consecutive to the previous word sequence. The total probability of each speech data generating the alignment path of the reconstructed text is calculated by a dynamic programming algorithm, and the alignment path with the highest total probability is determined as the optimal alignment path.

[0143] Through the above process, the pre-training model is iteratively trained by a dynamic programming algorithm, and the optimal alignment is performed in the hidden space, and the cross-modal feature representation of the pre-training model is gradually optimized. It can perform excellently in specific tasks, and can also maintain consistent high performance in a wide range of speech text tasks, thereby improving the applicability of the model, avoiding the loss of semantic information in the text when converting text to speech, and thus better applying to speech text alignment pre-training that emphasizes understanding tasks.

[0144] In an example embodiment, as shown in Figure 8 Step 270 can include the following steps:

[0145] Step 271, path backtracking is performed on the optimal alignment path to obtain the length proportion of each word in the reconstructed text data.

[0146] Specifically, by backtracking the selection of each node in the alignment path, the feature vector interval between each node corresponding word is derived, and then the time length of each word is determined by the length of the feature vector interval.

[0147] In a possible implementation, the path backtracking algorithm is as follows:

[0148]

[0149] Step 273, based on the length proportion of each word in the reconstructed text data relative to the entire reconstructed text data and the length proportion of each word in the text data relative to the entire text data, a divergence calculation is performed to obtain a consistency loss.

[0150] In a possible implementation, the consistency loss is the KL divergence of the duration of each word in the reconstructed text data and the duration of each word in the text data.

[0151] Specifically, the consistency loss function formula is as follows:

[0152]

[0153] Among them, y i To reconstruct each word and o in the text data i These are the words in the text data.

[0154] In one possible implementation, when the word t i In the middle of the text sequence of reconstructed text data, i.e., word t i When it is neither the first nor the last sentence, the result can be predicted by the time of its context: {t i-k , ..., t i-1 , t i} and {t i-k+1 , ..., t i , t i+1 The consistency loss is calculated using the following formula:

[0155]

[0156] in, It is a text sequence {t i-k , ..., t i-1 , t i The word t in} i Time prediction results It is to reconstruct the text sequence {t} i-k+1 , ..., t i , t i+1 The time prediction results for}.

[0157] Step 275: Adjust the model parameters of the pre-trained model based on the consistency loss, repeat the training process until the pre-trained model meets the preset pre-training objective, and obtain the speech-text alignment model.

[0158] The pre-training objectives of the pre-trained model include preset cross-modal response selection tasks, text masking modeling tasks, dialogue consistency tasks, temporal alignment prediction tasks, modal reconstruction tasks, and speech masking modeling tasks.

[0159] The six pre-training objectives are combined to form a single pre-training objective for speech-text pre-training. The formula for the pre-training objective is as follows:

[0160]

[0161] Wherein, CRS represents the cross-modal response selection task, MLM represents the text mask modeling task, CON represents the dialogue consistency task, TAP represents the temporal alignment prediction task, RE represents the modal reconstruction task, and MAM represents the speech mask modeling task.

[0162] Through the above process, the training and optimization of the speech text large model unsupervised word-level alignment are realized by constructing a consistency loss function, and the information in different turns of the same dialogue is aligned, which helps the pre-training model to understand the text speech information more finely, and the consistency between the context is enhanced by the consistency loss, so as to further constrain the model to produce correct alignment and improve the alignment performance of the speech text alignment model.

[0163] Figure 9 is a specific implementation schematic diagram of a speech text alignment model training method in an application scenario. In the figure, Text encoder represents a text encoder, Speech Encoder represents a speech encoder, Modality Fusion Module represents a modality fusion module, CRS represents a cross-modal response selection task, MLM represents a text mask modeling task, CON represents a dialogue consistency task, TAP represents a time alignment prediction task, DP represents a dynamic programming algorithm, RE represents a modality reconstruction task, and MAM represents a speech mask modeling task. In the application scenario, D = {T1, T2,..., Tn} is used to represent a dialogue containing n dialogue turns as text data and speech data, where T1, T2,..., Tn represent n dialogue turns, and each dialogue turn T1, T2,..., Tn is composed of text data and speech data. n} and speech data. Among them, T i is composed of the original speech waveform segment s i and its corresponding text data . Among them, w ij represents the jth word of the sentence t i , and m i represents the length of the sentence t i . s ij / e ij represents the start / end time of w ij . Each word w ij can be decomposed into a subword sequence, denoted as

[0164] (q represents the length of the subword sequence). For each dialogue turn T i (where i > 1), k+1 (k≥1) text dialogue turns and 2 speech dialogue turns are used to construct the text data input X i , which is represented as {t i-k , …, t i-1 , t i , s i-1 , s i}.

[0165] Since the length of the speech representation usually exceeds the length of its corresponding text representation, in order to improve the efficiency in the pre-training process, the number of speech turns is limited to two.

[0166] Using RoBERTa as a text encoder, learnable paragraph embeddings are introduced. A and

[0167] e B e B Add to and e Above, among which It is t i And the text embedding of the last tag.

[0168] Then, the text embedding, absolute position embedding, and paragraph embedding are summed and input into the text encoder to obtain the RoBERTa output hidden state.

[0169] An additional convolutional layer is added to the WavLM structure to generate the speech encoder, so that the output label of each speech feature represents approximately 200 milliseconds of speech, with a stride of 100 milliseconds. [CLS] and [SEP] labels are used to mark the start and end of the first speech sequence embedding. The output of the projection layer is then concatenated with the embeddings labeled [CLS] and [SEP]. The speech segment embeddings (e...) are then... A e B The speech segment embedding and the sum of the speech embeddings are added to the output after concatenation. The speech segment embedding and the sum of the speech embeddings are fed into the self-attention layer to produce the output hidden state of the last self-attention layer. in,

[0170]

[0171] The pre-trained model takes unmasked text and speech as input and uses a time prediction head to generate time predictions. It employs fully connected layers to predict the start and end times of each word in the text data, generating the time prediction results.

[0172] All text input was masked with [MASK] tags. Speech features were then analyzed. Through the fully connected layer W st The process involves predicting the first subword of the text corresponding to each speech feature vector, and generating reconstructed text data.

[0173] We utilize cross-entropy loss to perform a backward gradient update on the fully connected layer while freezing other parameters of the model. After the backward update, we obtain the revised matrix U. i From U i Extracting sub-word lists The corresponding row. The elements in the list represent sentence t. i The first subword of each word in the text. We apply the softmax function to the extracted matrix to obtain the probability of each speech feature vector corresponding to the initial subword of each text word.

[0174] The optimal alignment path is obtained by determining the alignment path that maximizes the total probability using a dynamic programming algorithm. The alignment path can be traced back and the selection of each node is determined, thereby deducing the interval between the feature vectors of each word. By measuring the length of these intervals, the length of each word is established . By dividing by the speech length L i , normalization is performed, and the consistency loss is calculated by calculating the KL divergence between y i and o i .

[0175] In pre-training, when optimizing the time prediction, the model is also optimized for Masked Language Modeling (MLM), Masked Acoustic Modeling (MAM), and Conversation Response Selection (CRS). We combine the six pre-training objectives to form a pre-training objective for speech-text pre-training.

[0176] Based on the consistency loss and the pre-training objectives described above, the training and optimization of the speech-text large model in an unsupervised word-level alignment are carried out.

[0177] In this application scenario, three innovative pre-training objectives (Masked Language Modeling (MLM), Masked Acoustic Modeling (MAM), and Conversation Response Selection (CRS)) are proposed for the pre-training model, enabling it to effectively train on data lacking word-level annotations. Using the proposed dynamic programming-based speech-text unannotated word-level alignment method, two size models USDP-base and USDP-large are trained to verify the effectiveness and advancement of the method proposed in this patent.

[0178] For different downstream tasks, we test the performance of the pre-trained USDP model in different downstream tasks.

[0179] Among them, the MSA task aims to predict the sentiment label from multiple forms. This patent conducts experiments on two multi-modal data sets MOSI and MOSEI to evaluate the effectiveness of the speech-text alignment model for the MSA task. The positive / negative emotion classification accuracy (denoted as Acc2) and the seven-class classification task (denoted as Acc7) are used as evaluation indicators to compare the speech-text alignment model and the baseline method. The USDP-large model improves the Acc7 on the MOSI dataset by 7.03 percentage points, exceeding the most advanced MIB method. On the MOSEI dataset, the result is improved by 5.95 percentage points, exceeding the SOTA method BBFN.

[0180] The ERC task requires a model to predict the emotion class of an utterance from a speech clip with transcription and dialogue history. Experiments are conducted on the speech-text alignment model on the ERC task, where the USDP-large model outperforms the SOTA method M2FNET by 1.72 and 2.77 percentage points on the MELD and IEMOCAP datasets, respectively, with significantly improved accuracy.

[0181] The SLU task aims to predict the user's intention from spoken utterances and text transcriptions. The MIntRec dataset is used as the experimental dataset for SLU, and the 20-class and binary classification accuracy (denoted as Acc20 and Acc2, respectively) are used as evaluation metrics. As can be seen from Table TODO, the USDP-large model improves over the previous SOTA method MAG-BERT by 4.27 and 4.52 percentage points on the 20-class results, respectively.

[0182] In the dialogue state tracking task, the SpokenWoZ dataset is used to evaluate the USDP. The results show that the speech-text alignment model USDP-large outperforms the previous SOTA method SPACE+WavLM+TripPy by 2.13 percentage points in joint goal accuracy (JGA).

[0183] In all downstream tasks, the speech-text alignment model is compared with other baseline models, and the comparison results are shown in the following table:

[0184]

[0185] Please refer to Figure 10 In the embodiments of the present application, a speech-text alignment model training device 900 is provided, which includes but is not limited to: an information prediction module 910, a reconstruction module 930, a path prediction module 950, and a parameter adjustment module 970.

[0186] The information prediction module 910 is configured to obtain a pre-trained model and text data, and perform time information prediction on each word in the text data based on the pre-trained model, to obtain time prediction results corresponding to each word.

[0187] The reconstruction module 930 is configured to obtain speech data, and perform text reconstruction on the speech data based on the time prediction results, to obtain reconstructed text data.

[0188] The path prediction module 950 is configured to perform optimal alignment path prediction on the reconstructed text data and the speech data based on a dynamic programming algorithm, to obtain an optimal alignment path.

[0189] The parameter adjustment module 970 is configured to perform parameter adjustment on the pre-trained model based on the optimal alignment path, to obtain a speech-text alignment model.

[0190] It should be noted that the speech-text alignment model training method and device provided in the above embodiments are only used as examples for the division of the above functional modules. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the speech-text alignment model training method and device is divided into different functional modules to complete all or part of the above-described functions.

[0191] In addition, the speech-text alignment model training method and device provided in the above embodiments belong to the same concept as the speech-text alignment model training method. The specific manner in which each module performs operations has been described in detail in the method embodiments, and will not be described here.

[0192] Figure 11 According to an exemplary embodiment, a structure of a server is shown. The server is suitable for Figure 1 The server 130 in the shown implementation environment.

[0193] It should be noted that the server is only an example suitable for the present application, and should not be considered as providing any limitation on the use range of the present application. The server should also not be interpreted as needing to rely on or must have Figure 11 One or more components in the shown exemplary server 2000.

[0194] The hardware structure of the server 2000 can vary greatly due to different configurations or performance, such as Figure 11 As shown, the server 2000 includes a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0195] Specifically, the power supply 210 is configured to provide working voltage for each hardware device on the server 2000.

[0196] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. For example, it is used to Figure 1 The interaction between the terminal 100 and the server 200 in the shown implementation environment.

[0197] Of course, in other examples suitable for the present application, the interface 230 can further include at least one serial-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc., such as Figure 11As shown, this does not constitute a specific limitation.

[0198] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0199] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the server 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0200] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task, and may include at least one module ( Figure 11 (Not shown), each module may contain computer-readable instructions for server 2000. For example, the speech-text alignment model training method apparatus can be considered as application 253 deployed on server 2000.

[0201] Data 255 can be photos, images, etc. stored on a disk, or recommendation information, etc., stored in memory 250.

[0202] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby enabling the computation and processing of massive amounts of data 255 in the memory 250. For example, a speech-text alignment model training method may be implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0203] Furthermore, this application can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or combination thereof.

[0204] Please see Figure 12 This application provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc.

[0205] exist Figure 12 In this context, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0206] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 can include a channel for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 12 Only one thick line is used in the middle, but it does not mean that there is only one bus or only one type of bus.

[0207] Optionally, the electronic device 4000 can also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that the transceiver 4004 is not limited to one in actual application, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0208] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can realize or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 4001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0209] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program instructions or code in the form of instructions or data structures and that can be accessed by the electronic device 400, but is not limited thereto.

[0210] The memory 4003 stores computer readable instructions, and the processor 4001 can read the computer readable instructions stored in the memory 4003 through the communication bus 4002.

[0211] The computer readable instructions are executed by the one or more processors 4001 to implement the voice text alignment model training method in the above embodiments.

[0212] In addition, a storage medium is provided in the embodiments of the present application, and the storage medium stores computer readable instructions. The computer readable instructions are executed by one or more processors to implement the voice text alignment model training method as described above.

[0213] A computer program product is provided in the embodiments of the present application, and the computer program product includes computer readable instructions stored in a storage medium. One or more processors of an electronic device read the computer readable instructions from the storage medium, load and execute the computer readable instructions, so that the electronic device implements the voice text alignment model training method as described above.

[0214] It should be understood that although the steps in the flowcharts of the drawings are shown in a sequential order following the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated otherwise herein, the execution of the steps is not strictly limited to the order indicated by the arrows, and can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of which is not necessarily sequential, but can be round-robin or alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0215] The above is only some embodiments of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered as the protection scope of the present application.

Claims

1. A method for training a speech-text alignment model, the method comprising: The method comprises the following steps: obtaining a pre-training model and text data, and performing time information prediction on each word in the text data based on the pre-training model to obtain a time prediction result corresponding to each word; obtaining speech data, and performing text reconstruction on the speech data based on the time prediction result to obtain reconstructed text data; performing optimal alignment path prediction on the reconstructed text data and the speech data based on a dynamic programming algorithm to obtain an optimal alignment path; adjusting parameters of the pre-training model based on the optimal alignment path to obtain a speech-text alignment model; the pre-training model comprises a text encoder and a fully connected layer; the time information prediction on each word in the text data based on the pre-training model to obtain a time prediction result corresponding to each word comprises: inputting the text data into the text encoder to obtain a text feature vector corresponding to each word in the text data; inputting the text feature vector into the fully connected layer to perform duration prediction on each word in the text data to obtain a length proportion of each word relative to the whole text data; determining the start time and the end time of each word based on the length proportion corresponding to each word to generate a time prediction result corresponding to each word; the pre-training model comprises a speech encoder and a fully connected layer; the text reconstruction on the speech data based on the time prediction result to obtain reconstructed text data comprises: inputting the speech data into the speech encoder to obtain a speech feature vector corresponding to each speech segment in the speech data; performing word prediction on each speech feature vector based on the fully connected layer to obtain a word corresponding to each speech feature vector to generate a word prediction result; performing paragraph division on the word prediction result based on the time prediction result to obtain reconstructed text data.

2. The method of claim 1, wherein, The text data comprises text embedding and absolute position embedding; the inputting of the text data into the text encoder to obtain a text feature vector corresponding to each word in the text data comprises: performing sequence division processing on the text data to obtain text paragraph embedding; performing summation processing on the text embedding, the absolute position embedding and the text paragraph embedding and inputting into the text encoder to obtain a text feature vector corresponding to each word.

3. The method of claim 1, wherein, The speech data comprises speech embedding; the inputting of the speech data into the speech encoder to obtain a speech feature vector corresponding to each speech segment in the speech data comprises: performing sequence division processing on the speech data to obtain speech paragraph embedding; performing summation processing on the speech paragraph embedding and the speech embedding and inputting into a self-attention layer in the speech encoder to obtain a speech feature vector corresponding to each speech segment in the speech data.

4. The method of claim 1, wherein, the optimal alignment path prediction on the reconstructed text data and the speech data based on the dynamic programming algorithm to obtain an optimal alignment path comprises: performing normalization processing on the reconstructed text data to obtain a probability distribution of each speech feature vector corresponding to each word; Based on the time sequence relationship of each speech feature vector and the probability distribution, the total probability of the speech data for each alignment path is calculated by using a dynamic programming algorithm, and the alignment path with the highest total probability is determined as the optimal alignment path.

5. The method of claim 1, wherein, The parameter adjustment of the pre-training model based on the optimal alignment path obtains a speech-text alignment model, which includes: Path backtracking is performed on the optimal alignment path to obtain the length proportion of each word in the reconstructed text data; Based on the length proportion of each word in the reconstructed text data relative to the overall length of the reconstructed text data and the length proportion of each word in the text data relative to the overall length of the text data, divergence calculation is performed to obtain a consistency loss; Based on the consistency loss, the model parameters of the pre-training model are adjusted, and the training process is repeated until the pre-training model meets the preset pre-training target, thereby obtaining a speech-text alignment model. 6.A voice text alignment model training apparatus, characterized by comprising: It includes: An information prediction module is configured to obtain a pre-training model and text data, and perform time information prediction on each word in the text data based on the pre-training model to obtain a time prediction result corresponding to each word; A reconstruction module is configured to obtain speech data and perform text reconstruction on the speech data based on the time prediction result to obtain reconstructed text data; A path prediction module is configured to predict an optimal alignment path based on a dynamic programming algorithm for the reconstructed text data and the speech data to obtain the optimal alignment path; A parameter adjustment module is configured to adjust the parameters of the pre-training model based on the optimal alignment path to obtain a speech-text alignment model. The pre-training model includes a text encoder and a fully connected layer; the time information prediction on each word in the text data based on the pre-training model to obtain a time prediction result corresponding to each word includes: The text data is input into the text encoder to obtain a text feature vector corresponding to each word in the text data; the text feature vector is input into the fully connected layer to perform duration prediction on each word in the text data to obtain the length proportion of each word relative to the overall length of the text data; the start time and the end time of each word are determined based on the length proportion corresponding to each word to generate a time prediction result corresponding to each word; The pre-training model includes a speech encoder and a fully connected layer; the text reconstruction on the speech data based on the time prediction result to obtain reconstructed text data includes: The speech data is input into the speech encoder to obtain a speech feature vector corresponding to each speech segment in the speech data; the fully connected layer is used to perform word prediction on each speech feature vector to obtain a word corresponding to each speech feature vector to generate a word prediction result; and the word prediction result is divided into paragraphs based on the time prediction result to obtain reconstructed text data.

7. An electronic device, comprising: It includes: At least one processor and at least one memory, wherein The memory stores computer readable instructions; The computer readable instructions are executed by one or more processors to enable the electronic device to implement the speech-text alignment model training method of any one of claims 1-5.

8. A storage medium having stored thereon computer readable instructions, characterized in that, The computer readable instructions are executed by one or more processors to implement the speech-text alignment model training method of any one of claims 1-5.

Images