Voice text alignment model training method and device, electronic equipment and storage medium

Abstract

The invention provides a voice text alignment model training method and device, electronic equipment and a storage medium, and relates to the field of large models. The method comprises the following steps: acquiring a pre-training model and text data, and performing time information prediction on each word in the text data based on the pre-training model to obtain a time prediction result corresponding to each word; obtaining voice data, and performing text reconstruction on the voice data based on the time prediction result to obtain reconstructed text data; performing optimal alignment path prediction on the reconstructed text data and the voice data based on a dynamic programming algorithm to obtain an optimal alignment path; and performing parameter adjustment on the pre-training model based on the optimal alignment path to obtain a voice text alignment model. According to the invention, the problem of low applicability of a voice text alignment model in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of large model technology. Specifically, the present application relates to a speech-text alignment model training method, device, electronic device and storage medium. Background Art

[0002] In recent years, multimodal human-computer dialogue has received widespread social attention, especially multimodal dialogue between speech and text. The speech-text alignment model can be used to realize the design of specific multimodal downstream tasks of speech and text, such as speech-to-text translation tasks and spoken language comprehension tasks.

[0003] However, traditional word-level alignment techniques usually convert text into phoneme sequences and align based on these phonemes. This method largely ignores the semantic information in the text, resulting in the inability to achieve ideal results in the task of multimodal dialogue between speech and text. Existing word-level alignment models such as mask modeling and cross-modal contrastive learning are mostly designed for specific downstream tasks, and it is difficult to maintain the same performance in a wider range of speech-to-text tasks. For example, in specific tasks such as speech-to-text translation or spoken language understanding, existing technologies may perform well, but they cannot maintain the same level of performance in other speech-to-text tasks, which limits the versatility and applicability of the model. In addition, the current word-level alignment model relies heavily on word-level annotated data during pre-training. The required word-level alignment annotation is costly and time-consuming. It is extremely difficult to obtain sufficient word-level annotated data, and it is impossible to fully utilize the large amount of available unlabeled data, which limits its application in a wider range of scenarios.

[0004] From the above, we can see that the problem of how to improve the applicability of speech-text alignment model training methods still needs to be solved. Summary of the invention

[0005] The present application provides a speech-text alignment model training method, device, electronic device and storage medium, which can solve the problem of low applicability of speech-text alignment model training methods in related technologies. The technical solution is as follows:

[0006] According to one aspect of the present application, a speech-text alignment model training method is characterized by comprising:

[0007] Acquire a pre-trained model and text data, and perform time information prediction for each word in the text data based on the pre-trained model to obtain a time prediction result corresponding to each word;

[0008] Acquire speech data, and reconstruct text of the speech data based on the time prediction result to obtain reconstructed text data;

[0009] Predicting an optimal alignment path for the reconstructed text data and the speech data based on a dynamic programming algorithm to obtain an optimal alignment path;

[0010] The parameters of the pre-trained model are adjusted based on the optimal alignment path to obtain a speech-text alignment model.

[0011] According to one aspect of the present application, a speech-text alignment model training device is characterized by comprising:

[0012] An information prediction module, used to obtain a pre-trained model and text data, and perform time information prediction for each word in the text data based on the pre-trained model to obtain a time prediction result corresponding to each word;

[0013] A reconstruction module, used for acquiring speech data, and reconstructing text of the speech data based on the time prediction result to obtain reconstructed text data;

[0014] A path prediction module, used to predict the optimal alignment path for the reconstructed text data and the voice data based on a dynamic programming algorithm to obtain an optimal alignment path;

[0015] A parameter adjustment module is used to adjust the parameters of the pre-trained model based on the optimal alignment path to obtain a speech-text alignment model.

[0016] In an exemplary embodiment, the pre-trained model includes a text encoder and a fully connected layer;

[0017] The information prediction module comprises:

[0018] A text feature vector unit, used for inputting text data into the text encoder to obtain a text feature vector corresponding to each word in the text data;

[0019] A duration prediction unit, used for inputting the text feature vector into the fully connected layer to perform duration prediction on each word in the text data to obtain the length ratio of each word relative to the entire text data;

[0020] The time prediction result unit is used to determine the start time and end time of each word based on the length ratio corresponding to each word, and generate a time prediction result corresponding to each word.

[0021] In an exemplary embodiment, the text data includes a text embedding and an absolute position embedding;

[0022] The text feature vector unit comprises:

[0023] The text paragraph embedding subunit is used to perform sequence division processing on the text data to obtain text paragraph embedding;

[0024] The text feature vector subunit is used to sum the text embedding, the absolute position embedding and the text paragraph embedding and input the sum into the text encoder to obtain the text feature vector corresponding to each word.

[0025] In an exemplary embodiment, the pre-trained model includes a speech encoder and a fully connected layer;

[0026] The reconstruction module comprises:

[0027] A speech feature vector unit, used for inputting the speech data into the speech encoder to obtain a speech feature vector corresponding to each speech segment in the speech data;

[0028] A word prediction unit, used to perform word prediction on each of the speech feature vectors based on the fully connected layer, obtain a word corresponding to each speech feature vector, and generate a word prediction result;

[0029] The text data reconstruction unit is used to divide the word prediction result into paragraphs based on the time prediction result to obtain reconstructed text data.

[0030] In an exemplary embodiment, the speech data includes speech embeddings;

[0031] The speech feature vector unit comprises:

[0032] The speech paragraph embedding subunit is used to perform sequence division processing on the speech data to obtain speech paragraph embedding;

[0033] The speech feature vector subunit is used to sum the speech segment embedding and the speech embedding and input them into the self-attention layer in the speech encoder to obtain the speech feature vector of each speech segment in the corresponding speech data.

[0034] In an exemplary embodiment, the path prediction module includes:

[0035] A probability distribution unit, used for normalizing the reconstructed text data to obtain a probability distribution of each word corresponding to each speech feature vector;

[0036] The total probability unit is used to calculate the total probability of the speech data for each alignment path based on the time series relationship of each of the speech feature vectors and the probability distribution using a dynamic programming algorithm, and determine the alignment path with the highest total probability as the optimal alignment path.

[0037] In an exemplary embodiment, the parameter adjustment module includes:

[0038] A path backtracking unit, used for backtracking the optimal alignment path to obtain the length proportion of each word in the reconstructed text data;

[0039] A consistency loss unit, used for performing divergence calculation based on the length ratio of each word in the reconstructed text data relative to the entire reconstructed text data and the length ratio of each word in the text data relative to the entire text data, to obtain a consistency loss;

[0040] A parameter adjustment unit is used to adjust the model parameters of the pre-trained model based on the consistency loss, repeat the training process until the pre-trained model meets the preset pre-training goals, and obtain a speech-text alignment model.

[0041] According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the speech-text alignment model training method as described above.

[0042] According to one aspect of the present application, a storage medium stores computer-readable instructions thereon, and the computer-readable instructions are executed by one or more processors to implement the speech-text alignment model training method as described above.

[0043] According to one aspect of the present application, a computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the speech-text alignment model training method as described above.

[0044] The beneficial effects of the technical solution provided by this application are:

[0045] In the above technical solution, the time information of each word in the text data is predicted, and the speech data is reconstructed, and the optimal alignment path for reconstructing the text data and the speech data is found through a dynamic programming algorithm, and finally a speech-text alignment model is obtained. By gradually optimizing the time alignment prediction of the speech-text alignment model through a dynamic programming algorithm, not only the prediction accuracy of the model is further improved, but also the model's fine-grained understanding of the input data is enhanced. At the same time, a large amount of unlabeled text data and speech data are used for training, breaking through the dependence on labeled data in the pre-training process, and significantly improving the accuracy and applicability of the model in speech-text downstream tasks.

[0046] This can effectively solve the problem of low applicability of speech-text alignment model training methods existing in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0048] Figure 1 It is a schematic diagram of the implementation environment involved in this application;

[0049] Figure 2 is a flow chart of a speech-text alignment model training method according to an exemplary embodiment;

[0050] Figure 3 yes Figure 2 A flowchart of step 210 in one embodiment corresponding to the embodiment;

[0051] Figure 4a yes Figure 3 A flowchart of an embodiment corresponding to step 211 in an embodiment;

[0052] Figure 4b is a schematic diagram of a text encoder according to an exemplary embodiment;

[0053] Figure 5 yes Figure 2 A flowchart of step 230 in one embodiment corresponding to the embodiment;

[0054] Figure 6a yes Figure 5 A flowchart of an embodiment corresponding to step 231 in an embodiment;

[0055] Figure 6b is a schematic diagram of a speech encoder according to an exemplary embodiment;

[0056] Figure 7 yes Figure 2 A flowchart of an embodiment corresponding to step 250 in an embodiment;

[0057] Figure 8 yes Figure 2 A flowchart of an embodiment corresponding to step 270 in an embodiment;

[0058] Fig. 9 This is a schematic diagram of a specific implementation of a speech-text alignment model training method in an application scenario;

[0059] Fig.10 is a structural block diagram of a speech-text alignment model training method device according to an exemplary embodiment;

[0060] Fig.11 is a hardware structure diagram of a server according to an exemplary embodiment;

[0061] Fig.12 The present invention is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0062] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0063] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0064] As mentioned above, the existing speech-text alignment model training method detects the differences between data sets, which results in complex calculations during the leakage detection process, poor detection results when disorder problems exist in the test data, and thus low accuracy.

[0065] From the above, it can be seen that the relevant technology still has the defect that the applicability of the speech-text alignment model training method is not high.

[0066] To this end, the speech-text alignment model training method provided in the present application can effectively improve the accuracy of the speech-text alignment model training method. Accordingly, the speech-text alignment model training method is suitable for a speech-text alignment model training method device, and the speech-text alignment model training method device can be deployed on an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, for example, the computer device includes a desktop computer, a laptop computer, a server, etc.; the electronic device can also be an electronic device with a central control function, for example, the electronic device includes a gateway, etc.; the electronic device can also refer to a portable and mobile electronic device, for example, the electronic device includes a smart phone, a tablet computer, etc.

[0067] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0068] Figure 1 It is a schematic diagram of an implementation environment involved in an image processing method. It should be noted that this implementation environment is only an example adapted to the present invention and cannot be considered as providing any limitation on the scope of use of the present invention.

[0069] The implementation environment includes a collection end 110 and a service end 130 .

[0070] Specifically, the acquisition end 110 can be considered as an information acquisition device, including but not limited to electronic devices with information acquisition functions such as cameras and locators, and can also be considered as a device that integrates information acquisition and speech-text alignment model training methods, including but not limited to electronic devices such as laptops and desktop computers that have both information acquisition functions and speech-text alignment model training methods. That is, in some embodiments, the acquisition end 110 can collect text data and voice data.

[0071] The server 130 can also be considered as a speech-text alignment model training device, including but not limited to smart phones, desktop computers, laptop computers, servers and other electronic devices with speech-text alignment model training method functions, and can also be a computer cluster composed of multiple servers, or even a cloud computing center composed of multiple servers. The server 130 is used to provide background services, for example, the background services include but are not limited to speech-text alignment model training services, etc.

[0072] The server 130 and the collection terminal 110 establish a network communication connection in advance through wired or wireless means, and the data transmission between the server 130 and the collection terminal 110 is realized through the network communication connection. The transmitted data includes but is not limited to: text data and voice data, etc.

[0073] In one application scenario, through the interaction between the collection end 110 and the server end 130, the collection end 110 collects text data and voice data, and uploads the text data and voice data to the server end 130 to request the server end 130 to provide a speech-text alignment model training service.

[0074] At this time, for the server 130, after receiving the text data and voice data uploaded by the acquisition end 110, it calls the speech-text alignment model training service to train the speech-text alignment model according to the text data and voice data, thereby solving the problem of low applicability of the speech-text alignment model training method existing in the related technology.

[0075] See also Figure 2The present application embodiment provides a speech-text alignment model training method, which is applicable to an electronic device, which may be Figure 1 The server 130 in the illustrated implementation environment may also be a desktop computer, a laptop computer, a server, and the like.

[0076] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0077] like Figure 2 As shown, the method may include the following steps:

[0078] Step 210, obtaining a pre-trained model and text data, and performing time information prediction for each word in the text data based on the pre-trained model to obtain a time prediction result corresponding to each word.

[0079] Among them, the text data includes text information that needs to be aligned with speech and text and text embedding and absolute position embedding generated by the text information. The pre-trained model is a preset model that needs to be trained for text-speech alignment capabilities. The text alignment model can be obtained by training the pre-trained model.

[0080] Specifically, the pre-training model includes a text encoder, a speech encoder, a fully connected layer and a modal fusion layer, wherein the text encoder is used to encode the text data in the input data to obtain a text feature vector, the speech encoder is used to encode the speech data in the input data to obtain a speech modal feature vector, the modal fusion layer is used to fuse the text modal feature vector and the speech modal feature vector across modal information to obtain a mixed modal feature vector for pre-training tasks or downstream prediction tasks in dialogue understanding, and the fully connected layer is used to perform text and speech prediction on the input data.

[0081] In one possible implementation, the pre-trained model is a SPECTRA model.

[0082] Step 230, acquiring speech data, and reconstructing the speech data into text based on the time prediction result to obtain reconstructed text data.

[0083] The speech data includes unlabeled speech information corresponding to the text data and speech embedding generated by the speech information, which needs to be aligned with the speech text. Text reconstruction can be used to reconstruct the speech data into text data, thereby aligning and predicting the text content corresponding to the speech data.

[0084] In a possible implementation, the voice data is voice waveform segments collected in multiple rounds of conversations, and the text data is the corresponding text sentences, wherein each word in the text sentence can be decomposed into a sub-word sequence.

[0085] Step 250 , predicting the optimal alignment path for the reconstructed text data and speech data based on a dynamic programming algorithm to obtain the optimal alignment path.

[0086] Among them, the optimal alignment path is the alignment path calculated according to the dynamic programming algorithm, which has the smallest loss in text reconstruction of speech data.

[0087] Step 270, adjusting the parameters of the pre-trained model based on the optimal alignment path to obtain a speech-text alignment model.

[0088] Among them, the pre-trained model is optimized by continuously adjusting the parameters of the pre-trained model through the optimal alignment path, so that the pre-trained model is trained in the direction of improving the quality of speech-text alignment result generation, and finally a speech-text generation model is obtained.

[0089] Through the above process, the time alignment prediction of the speech-text alignment model is gradually optimized through the dynamic programming algorithm, which not only further improves the prediction accuracy of the model, but also enhances the model's fine-grained understanding of the input data. At the same time, a large amount of unlabeled text data and speech data are used for training, and word-level alignment can be achieved through parallel corpora without word-level alignment annotations, which greatly reduces the cost and time of data preparation, breaks through the dependence on labeled data in the pre-training process, and significantly improves the accuracy and applicability of the model in the downstream task of speech-text alignment.

[0090] In an exemplary embodiment, if Figure 3 As shown, step 210 may include the following steps:

[0091] Step 211, input the text data into a text encoder to obtain a text feature vector corresponding to each word in the text data.

[0092] In an exemplary embodiment, if Figure 4a As shown, step 211 may include the following steps:

[0093] Step 2111, perform sequence segmentation processing on the text data to obtain text paragraph embedding.

[0094] Specifically, the text data is divided into sequences according to the intervals of conversations in the text data, and the text paragraph embedding is added to the text embedding in the text data to complete the sequence division processing of the text data.

[0095] In one possible implementation, special tags are used to divide the sequences in the text data and segment the conversations. For example, setting tags in the text data <s> Indicates the start of a sequence, setting a marker< / s> Indicates the end of each round of dialogue. At this time, we get the learnable paragraph embedding e A and eB , e B Add to and e The text paragraph embedding is obtained by using the text paragraph embedding to distinguish the conversation intervals in the text data. Yes i and the last token's text embedding.

[0096] Step 2113, summing up the text embedding, absolute position embedding and text paragraph embedding and inputting them into the text encoder to obtain the text feature vector corresponding to each word.

[0097] Specifically, the text embedding, absolute position embedding and paragraph embedding are summed and input into the text encoder to obtain a text feature vector, which is used to indicate the hidden state of the text data in the latent space.

[0098] In one possible implementation, Figure 4b As shown in Figure 1, the text encoder is a RoBERTa model. By inputting text embedding, absolute position embedding, and paragraph embedding into the RoBERTa model, the output hidden state of the RoBERTa model is obtained. in represents the output hidden state of RoBERTa, where n t is the length of the text input, and d h is the hidden state dimension.

[0099] Through the above process, the text data is divided into dialogues through sequence division processing, which improves the accuracy and efficiency of speech-text alignment.

[0100] Step 213, input the text feature vector into the fully connected layer to predict the duration of each word in the text data, and obtain the length ratio of each word relative to the entire text data.

[0101] In a possible implementation, a matrix consisting of text hidden representations of the first subword of each word is used as a text feature vector to predict the duration of each word in the text data.

[0102] The formula is as follows:

[0103]

[0104] in, Indicates the proportion of each word in the total length of the text data,

[0105] is a matrix consisting of the textual hidden representations of the first subword of each word.

[0106] Step 215, determining the start time and end time of each word based on the length ratio corresponding to each word, and generating a time prediction result corresponding to each word.

[0107] It should be noted that each word is aligned one by one by obtaining the start time and end time of each word in the text data. For example, under the premise that the first word of the voice data starts from 0.0 seconds and ends with the last word, each word is closely connected to the next word without overlap or gap, and each word lasts for at least 0.1 seconds, through the timing of each word in the text data, after determining that the first word is aligned with the voice data, the corresponding relationship between the subsequent words and the voice data is determined one by one and aligned.

[0108] In one possible implementation, after obtaining the time proportion of each word, the length of each word is obtained by multiplying the proportion of each word in the total length of the text data by the speech duration of the text information, and then calculating the start time and end time of each word in time sequence.

[0109] The calculation formula for the start time and end time of each word is as follows:

[0110]

[0111] Among them, w ij For words, s ij is the start time, e ij is the end time.

[0112] Through the above process, a fully connected layer is used to predict the duration of each word, thereby ensuring that the time alignment prediction of the speech-text alignment model remains monotonic, thereby improving the alignment accuracy and efficiency of the speech-text alignment model.

[0113] In an exemplary embodiment, if Figure 5 As shown, step 230 may include the following steps:

[0114] Step 231: input the speech data into a speech encoder to obtain a speech feature vector corresponding to each speech segment in the speech data.

[0115] In one possible implementation, the speech encoder is a WavLM structure including an additional convolutional layer.

[0116] In an exemplary embodiment, if Figure 6a As shown, step 231 may include the following steps:

[0117] Step 2311, perform sequence division processing on the speech data to obtain speech paragraph embedding.

[0118] By performing sequence division processing on the speech data, different speech sequences in the speech data are distinguished, and then the speech segment embedding is used to add the speech embedding in the speech data to complete the sequence division processing of the speech data.

[0119] In a possible implementation, the sequence segmentation process is performed by using [CLS] and [SEP] tags to mark the beginning and end of the speech sequence embedding.

[0120] Step 2313, the speech segment embedding and the speech embedding are summed and input into the self-attention layer in the speech encoder to obtain the speech feature vector of each speech segment in the corresponding speech data.

[0121] Specifically, the speech encoder is Figure 6b As shown in the figure, the additional convolutional layer is the convolutional layer added on the basis of WavLM, Conv represents the convolutional layer, Projection represents the projection layer, and Self-Attention represents the self-attention layer. By inputting the speech into the waveform. By connecting the output of the projection layer with the speech data after sequence division, the speech segment is embedded (e A 、e B ) is added to the concatenated speech embedding for summation. It is then input into the self-attention layer to produce in,

[0122] represents the output hidden state of the last self-attention layer, n s is the speech embedding length.

[0123] Through the above process, the corresponding text paragraph embedding and speech paragraph embedding are added to the output of the text encoder and the speech encoder, so that the text data and speech data can be distinguished. Finally, the self-attention layer is used to effectively integrate the two modalities to achieve the versatility of the speech-text alignment model in multimodality.

[0124] Step 233: perform word prediction on each speech feature vector based on the fully connected layer, obtain the word corresponding to each speech feature vector, and generate a word prediction result.

[0125] It should be noted that when predicting words on the speech feature vector, it is necessary to isolate the influence of text information. Therefore, it is necessary to mask the text information by adding tags and input the speech feature vector into the fully connected layer for prediction.

[0126] In one possible implementation, the prediction formula is as follows:

[0127]

[0128] in,

[0129] Among them, U i The predicted value V of the first subword of the text corresponding to each speech feature vector is the vocabulary parameter, d h is the size of the speech feature vector.

[0130] Step 235 , dividing the word prediction results into paragraphs based on the time prediction results to obtain reconstructed text data.

[0131] Specifically, the text data in the word prediction result is divided according to the predicted word start time and end time, and the word prediction result is aligned with the corresponding word.

[0132] It should be noted that after text reconstruction, the model parameters except the parameters of the fully connected layer in the pre-trained model are fixed, and the fully connected layer parameters are reversely updated with the cross entropy loss to revise the word prediction results.

[0133] Through the above process, independent prediction of text data can be achieved by relying solely on voice data, which ensures the alignment ability of the voice-text alignment model. By dividing the word prediction results into paragraphs, a tighter alignment can be achieved, thereby improving the voice-text alignment effect.

[0134] In an exemplary embodiment, if Figure 7 As shown, step 250 may include the following steps:

[0135] Step 251, normalize the reconstructed text data to obtain the probability distribution of each speech feature vector corresponding to each word.

[0136] Among them, the probability distribution is obtained by normalizing through the softmax function.

[0137] Specifically, by extracting the subword sequence corresponding to each word in the reconstructed text data from the word prediction results, determining the probability that each speech feature vector corresponds to the initial subword in the subword sequence in the reconstructed text data, and generating the probability distribution of each speech feature vector.

[0138] Among them, the probability distribution formula of each speech feature vector is as follows:

[0139]

[0140] Among them, F(x, y) represents the first x speech feature vectors and t i The first y text words of the sentence have the highest probability when they correspond.

[0141] Step 253, based on the time series relationship and probability distribution of each speech feature vector, the total probability of the speech data for each alignment path is calculated using a dynamic programming algorithm, and the alignment path with the highest total probability is determined as the optimal alignment path.

[0142] Among them, since each speech feature vector has a time sequence, after determining the word corresponding to the current speech feature vector, the word corresponding to the next speech feature vector can be determined to be a word coherent with the previous word sequence based on the time sequence of the speech feature vector. The total probability of the alignment path for generating the reconstructed text from each speech data is calculated through a dynamic programming algorithm, and the alignment path with the highest total probability is determined as the optimal alignment path.

[0143] Through the above process, the pre-trained model is iteratively trained through a dynamic programming algorithm, optimal alignment is performed in the latent space, and the cross-modal feature representation of the pre-trained model is gradually optimized. It can perform well in specific tasks and maintain a consistent high level of performance in a wide range of speech-to-text tasks, thereby improving the applicability of the model. When converting text to speech, it avoids the loss of semantic information in the text, so that it can be better applied to speech-to-text alignment pre-training that focuses on understanding tasks.

[0144] In an exemplary embodiment, if Figure 8 As shown, step 270 may include the following steps:

[0145] Step 271, backtracking the optimal alignment path to obtain the length ratio of each word in the reconstructed text data.

[0146] Specifically, by backtracking the selection of each node in the alignment path, the feature vector interval between the words corresponding to each node is derived, and then the time length of each word is determined by the length of the feature vector interval.

[0147] In one possible implementation, the path backtracking algorithm is as follows:

[0148]

[0149] Step 273, based on the length ratio of each word in the reconstructed text data relative to the entire reconstructed text data and the length ratio of each word in the text data relative to the entire text data, a divergence calculation is performed to obtain a consistency loss.

[0150] In a possible implementation, the consistency loss is the KL divergence of the duration of each word in the reconstructed text data and the duration of each word in the text data.

[0151] Specifically, the consistency loss function formula is as follows:

[0152]

[0153] Among them, y i To reconstruct the words and o in the text data i For each word in the text data.

[0154] In one possible implementation, when the word t i In the middle of the text sequence of the reconstructed text data, that is, word t i When it is neither the first nor the last sentence, the result can be predicted by the time of its context: {t i-k , ..., t i-1 , t i} and {t i-k+1 , ..., t i , t i+1}Calculate the consistency loss. The consistency loss formula is as follows:

[0155]

[0156] in, is a sequence of text {t i-k , ..., t i-1 , t i The word t in i The time prediction results, is the reconstructed text sequence {t i-k+1 , ..., t i , t i+1} time prediction results.

[0157] Step 275, adjust the model parameters of the pre-trained model based on the consistency loss, repeat the training process until the pre-trained model meets the preset pre-training goals, and obtain the speech-text alignment model.

[0158] Among them, the pre-training objectives of the pre-training model include the preset cross-modal response selection task, text mask modeling task, dialogue consistency task, timing alignment prediction task, modal reconstruction task, and speech mask modeling task.

[0159] The six pre-training objectives are combined to form a pre-training objective for speech-to-text pre-training. The pre-training objective formula is as follows:

[0160]

[0161] Among them, CRS represents the cross-modal response selection task, MLM represents the text mask modeling task, CON represents the dialogue consistency task, TAP represents the temporal alignment prediction task, RE represents the modal reconstruction task, and MAM represents the speech mask modeling task.

[0162] Through the above process, by constructing a consistency loss function, we can realize the training and optimization of unsupervised word-level alignment of a large speech-text model, and align the information in different rounds of the same conversation, so as to help the pre-trained model understand the text speech information in a more fine-grained manner, and enhance the consistency of the temporal output between contexts through consistency loss, thereby further constraining the model to produce correct alignment and improving the alignment performance of the speech-text alignment model.

[0163] Fig. 9 This is a schematic diagram of a specific implementation of a speech-text alignment model training method in an application scenario. In the figure, Text encoder represents the text encoder, Speech Encoder represents the speech encoder, Modality FusionModule represents the modality fusion module, CRS represents the cross-modal response selection task, MLM represents the text mask modeling task, CON represents the dialogue consistency task, TAP represents the time series alignment prediction task, DP represents the dynamic programming algorithm, RE represents the modality reconstruction task, and MAM represents the speech mask modeling task. In this application scenario, D={T1,T2,...,T n} to represent a conversation containing n conversation turns as text data and voice data, where T i The original speech waveform segment s in the speech data i and its corresponding text data Among them, w ij Represents sentence t i The jth word, m i Represents sentence t i length. Use s ij / e ij Indicates w ij The start / end time of each word w ij can be decomposed into a subword sequence, represented as

[0164] (q represents the length of the subword sequence). For each dialogue turn T i (where i>1), use k+1(k≥1) text dialogue rounds and 2 voice dialogue rounds to construct the text data input X i , which is expressed as {t i-k ,…,t i-1 , t i ,s i-1 ,s i}.

[0165] Since the length of speech representations usually exceeds the length of their corresponding text representations, the number of speech epochs is limited to two for increased efficiency during the pre-training process.

[0166] Using RoBERTa as a text encoder, we introduce learnable paragraph embeddings A and

[0167] e B . B Add to and e On, among them Yes i and the last token's text embedding.

[0168] Then the text embedding, absolute position embedding and paragraph embedding are summed and input into the text encoder to get the RoBERTa output hidden state

[0169] An additional convolutional layer is added to the speech encoder based on the WavLM structure so that the output token of each speech feature represents approximately 200 milliseconds of speech and the stride is 100 milliseconds. The [CLS] and [SEP] tokens are used to mark the beginning and end of the first speech sequence embedding. The output of the projection layer is then concatenated with the embeddings of the [CLS] and [SEP] tokens. The speech segment embedding (e A 、e B ) is added to the concatenated output. The sum of the speech segment embedding and the speech embedding is fed into the self-attention layer to produce the output hidden state of the last self-attention layer in,

[0170]

[0171] The pre-trained model accepts unmasked text and speech as input and generates time predictions using a time prediction head. A fully connected layer is used to predict the start and end time of each word in the text data to generate time prediction results.

[0172] All text input is masked with [MASK] tags. Through the fully connected layer W st Processing, predicting the result of the first subword of the text corresponding to each speech feature vector, and generating reconstructed text data.

[0173] We use the cross entropy loss to perform a reverse gradient update on the fully connected layer while freezing other parameters of the model. After the reverse update, we get the revised matrix U i From U i Extract and subword list The corresponding row. The elements in the list represent sentences t i We apply the softmax function to the extracted matrix to obtain the probability that each speech feature vector corresponds to the initial subword of each text word.

[0174] The dynamic programming algorithm is used to determine the alignment path that maximizes the total probability and obtain the optimal alignment path. The alignment path can be traced back and the selection of each node can be determined to derive the feature vector interval of each word. By measuring the length of these intervals, the length of each word is established. By Divide by the speech length L i to normalize, and calculate y i and i The KL divergence between them is used to calculate the consistency loss.

[0175] During pre-training, when optimizing for temporal prediction, the model is also optimized for Masked Language Modeling (MLM), Masked Acoustic Modeling (MAM), and Dialogue Response Selection (CRS). We combine the six pre-training objectives into one pre-training objective for speech-to-text pre-training.

[0176] Based on the above consistency loss and pre-training objectives, the unsupervised word-level alignment of the large speech-to-text model is trained and optimized.

[0177] In this application scenario, three innovative pre-training objectives (Masked Language Modeling (MLM), Masked Acoustic Modeling (MAM), and Dialogue Response Selection (CRS)) are proposed for the pre-training model, enabling it to be effectively trained on data lacking word-level annotations. Using the proposed dynamic programming-based speech-text unannotated word-level alignment method, two models of size USDP-base and USDP-large are trained to verify the effectiveness and advancement of the method proposed in this patent.

[0178] For different downstream tasks, we tested the performance of the pre-trained USDP model in different downstream tasks.

[0179] Among them, the MSA task aims to predict sentiment labels from multiple modalities. This patent conducts experiments on two multimodal datasets, MOSI and MOSEI, to evaluate the effectiveness of the speech-text alignment model for the MSA task. The positive / negative emotion classification accuracy (denoted as Acc2) and the seven-category classification task (denoted as Acc7) are used as evaluation indicators to compare the speech-text alignment model and the baseline method. The USDP-large model improves Acc7 by 7.03 percentage points on the MOSI dataset, surpassing the most advanced MIB method. The results were improved by 5.95 percentage points on the MOSEI dataset, surpassing the SOTA method BBFN.

[0180] The ERC task requires the model to predict the emotional category of speech based on speech clips with transcription and conversation history. Experiments were conducted on the speech-to-text alignment model on the ERC task, where the USDP-large model outperformed the SOTA method M2FNET by 1.72 and 2.77 percentage points on the MELD and IEMOCAP datasets, respectively, with significantly improved accuracy.

[0181] The SLU task aims to predict user intent based on spoken utterances and text transcriptions. The MIntRec dataset is used as the experimental dataset for SLU, and the 20-class and binary classification accuracy (denoted as Acc20 and Acc2, respectively) are used as evaluation indicators. It can be clearly seen from Table TODO that the USDP-large model has achieved 4.27 and 4.52 percentage point improvements in the 20-class results compared to the previous SOTA method MAG-BERT, respectively.

[0182] In the dialogue state tracking task, the SpokenWoZ dataset is used to evaluate USDP. The results show that the speech-to-text alignment model USDP-large surpasses the previous SOTA method SPACE+WavLM+TripPy in the joint objective accuracy (JGA), with an improvement of 2.13 percentage points.

[0183] The speech-to-text alignment model is compared with other baseline models on all downstream tasks. The comparison results are shown in the following table:

[0184]

[0185] See also Fig.10 In an embodiment of the present application, a speech-text alignment model training device 900 is provided, including but not limited to: an information prediction module 910, a reconstruction module 930, a path prediction module 950 and a parameter adjustment module 970.

[0186] The information prediction module 910 is used to obtain a pre-trained model and text data, and perform time information prediction for each word in the text data based on the pre-trained model to obtain a time prediction result corresponding to each word;

[0187] A reconstruction module 930, used to obtain speech data, and perform text reconstruction on the speech data based on the time prediction result to obtain reconstructed text data;

[0188] A path prediction module 950 is used to predict the optimal alignment path for the reconstructed text data and the voice data based on a dynamic programming algorithm to obtain an optimal alignment path;

[0189] The parameter adjustment module 970 is used to adjust the parameters of the pre-trained model based on the optimal alignment path to obtain a speech-text alignment model.

[0190] It should be noted that the speech-text alignment model training method device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when performing the speech-text alignment model training method. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the speech-text alignment model training method device will be divided into different functional modules to complete all or part of the functions described above.

[0191] In addition, the speech-text alignment model training method device provided in the above embodiment and the embodiment of the speech-text alignment model training method belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiment and will not be repeated here.

[0192] Fig.11 A schematic diagram of the structure of a server is shown according to an exemplary embodiment. The server is suitable for Figure 1 A server 130 in an implementation environment is shown.

[0193] It should be noted that the server is only an example adapted to the present application and cannot be considered to provide any limitation on the scope of use of the present application. The server cannot be interpreted as requiring dependence on or having Fig.11 One or more components of exemplary server 2000 are shown.

[0194] The hardware structure of the server 2000 may vary greatly due to different configurations or performances, such as Fig.11 As shown, the server 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0195] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the server 2000 .

[0196] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Figure 1 The interaction between the terminal 100 and the server 200 in the implementation environment is shown.

[0197] Of course, in other examples adapted by the present application, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Fig.11As shown, this is not intended to be a specific limitation.

[0198] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0199] Among them, the operating system 251 is used to manage and control the hardware devices and application programs 253 on the server 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0200] The application program 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Fig.11 (not shown), each module may respectively include computer-readable instructions for the server 2000. For example, the speech-text alignment model training method device may be regarded as an application 253 deployed on the server 2000.

[0201] The data 255 may be photos, pictures, etc. stored in a disk, or may be recommendation information, etc., stored in the memory 250 .

[0202] The central processor 270 may include one or more processors, and is configured to communicate with the memory 250 through at least one communication bus to read the computer-readable instructions stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the speech-text alignment model training method is completed by the central processor 270 reading a series of computer-readable instructions stored in the memory 250.

[0203] In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present application is not limited to any specific hardware circuits, software, or a combination of the two.

[0204] See also Fig.12 In an embodiment of the present application, an electronic device 4000 is provided, and the electronic device 400 may include: a desktop computer, a laptop computer, a server, etc.

[0205] exist Fig.12 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0206] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0207] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0208] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0209] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 400, but is not limited to this.

[0210] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0211] The computer-readable instructions are executed by one or more processors 4001 to implement the speech-text alignment model training method in the above-mentioned embodiments.

[0212] In addition, a storage medium is provided in an embodiment of the present application, on which computer-readable instructions are stored, and the computer-readable instructions are executed by one or more processors to implement the above-mentioned speech-text alignment model training method.

[0213] A computer program product is provided in an embodiment of the present application. The computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the above-mentioned speech-text alignment model training method.

[0214] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0215] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A speech-text alignment model training method, characterized in that: include: Acquire a pre-trained model and text data, and perform time information prediction for each word in the text data based on the pre-trained model to obtain a time prediction result corresponding to each word; Acquire speech data, and reconstruct text of the speech data based on the time prediction result to obtain reconstructed text data; Predicting an optimal alignment path for the reconstructed text data and the speech data based on a dynamic programming algorithm to obtain an optimal alignment path; The parameters of the pre-trained model are adjusted based on the optimal alignment path to obtain a speech-text alignment model.

2. The method according to claim 1, characterized in that The pre-trained model includes a text encoder and a fully connected layer; The step of performing time information prediction for each word in the text data based on the pre-trained model to obtain a time prediction result corresponding to each word includes: Inputting text data into the text encoder to obtain a text feature vector corresponding to each word in the text data; Input the text feature vector into the fully connected layer to predict the duration of each word in the text data and obtain the length ratio of each word relative to the entire text data; The start time and the end time of each word are determined based on the length ratio corresponding to each word, and a time prediction result corresponding to each word is generated.

3. The method according to claim 2, characterized in that The text data includes text embedding and absolute position embedding; The step of inputting the text data into the text encoder to obtain a text feature vector of each word in the corresponding text data includes: Perform sequence division processing on text data to obtain text paragraph embedding; The text embedding, the absolute position embedding and the text paragraph embedding are summed and input into the text encoder to obtain the text feature vector corresponding to each word.

4. The method according to claim 1, characterized in that The pre-trained model includes a speech encoder and a fully connected layer; The step of reconstructing the speech data into text based on the time prediction result to obtain reconstructed text data comprises: Inputting the speech data into the speech encoder to obtain a speech feature vector corresponding to each speech segment in the speech data; Perform word prediction on each of the speech feature vectors based on the fully connected layer, obtain words corresponding to each speech feature vector, and generate word prediction results; The word prediction result is divided into paragraphs based on the time prediction result to obtain reconstructed text data.

5. The method according to claim 4, characterized in that The speech data includes speech embedding; The step of inputting the speech data into the speech encoder to obtain a speech feature vector corresponding to each speech segment in the speech data comprises: Sequence the speech data and obtain speech segment embedding; The speech segment embedding and the speech embedding are summed and input into the self-attention layer in the speech encoder to obtain the speech feature vector of each speech segment in the corresponding speech data.

6. The method according to claim 1, characterized in that The step of predicting the optimal alignment path for the reconstructed text data and the speech data based on a dynamic programming algorithm to obtain the optimal alignment path includes: Normalizing the reconstructed text data to obtain the probability distribution of each word corresponding to each speech feature vector; Based on the temporal relationship of each of the speech feature vectors and the probability distribution, a dynamic programming algorithm is used to calculate the total probability of the speech data for each alignment path, and the alignment path with the highest total probability is determined as the optimal alignment path.

7. The method according to claim 1, characterized in that The step of adjusting parameters of the pre-trained model based on the optimal alignment path to obtain a speech-text alignment model includes: Perform path backtracking on the optimal alignment path to obtain the length proportion of each word in the reconstructed text data; Based on the length ratio of each word in the reconstructed text data relative to the entire reconstructed text data and the length ratio of each word in the text data relative to the entire text data, a divergence calculation is performed to obtain a consistency loss; The model parameters of the pre-trained model are adjusted based on the consistency loss, and the training process is repeated until the pre-trained model meets the preset pre-training goals to obtain a speech-text alignment model.

8. A speech-text alignment model training device, characterized in that: include: An information prediction module, used to obtain a pre-trained model and text data, and perform time information prediction for each word in the text data based on the pre-trained model to obtain a time prediction result corresponding to each word; A reconstruction module, used for acquiring speech data, and reconstructing text of the speech data based on the time prediction result to obtain reconstructed text data; A path prediction module, used to predict the optimal alignment path for the reconstructed text data and the voice data based on a dynamic programming algorithm to obtain an optimal alignment path; A parameter adjustment module is used to adjust the parameters of the pre-trained model based on the optimal alignment path to obtain a speech-text alignment model.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the speech-text alignment model training method as described in any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the speech-text alignment model training method as described in any one of claims 1 to 7.

Images