Reinforcement learning voice text alignment model training method and device, electronic equipment and storage medium

Abstract

The invention provides a reinforcement learning voice text alignment model training method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring an annotation training set, and performing text alignment training on a pre-training model based on the annotation training set to obtain a voice text alignment model; pre-training a value function model corresponding to the voice text alignment model based on the annotation training set to obtain a first reward model; obtaining an unlabeled training set, and pre-training the voice text alignment model based on the unlabeled training set to obtain a second reward model; and performing reinforcement learning training on the text alignment model based on the first reward model, the second reward model and the unlabeled training set to obtain a target model for voice text alignment. According to the method and the device, the problem of low voice text alignment applicability of a voice text alignment model in related technologies is solved.

Description

Reinforcement learning speech-text alignment model training methods, devices, electronic equipment, and storage media Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus, electronic device, and storage medium for training a reinforcement learning speech-text alignment model. Background Technology

[0002] In recent years, significant progress has been made in training speech-to-text alignment models. These models can align the feature vectors of speech segments with the corresponding feature vectors of text sentences, enabling the design of specific downstream multimodal speech-to-text tasks, such as speech-to-text translation and spoken language comprehension.

[0003] However, current speech-to-text alignment models learn cross-modal feature vector representations using data from large-scale training corpora, primarily employing multimodal self-supervised pre-training loss functions such as masking modeling and cross-modal contrastive learning. However, this approach requires a large amount of word-level alignment annotations. Since this word-by-word annotation necessitates precise temporal representation down to extremely small time segments, the required amount of word-level alignment annotation data is extremely time-consuming and expensive. Furthermore, the model performs poorly when only a small amount of word-level annotation data is available, resulting in limited applicability.

[0004] As can be seen from the above, the problem of how to improve the applicability of reinforcement learning speech-text alignment model training methods still needs to be solved. Summary of the Invention

[0005] This application provides a reinforcement learning speech-to-text alignment model training method, apparatus, electronic device, and storage medium, which can solve the problem of low applicability of reinforcement learning speech-to-text alignment model training methods in related technologies. The technical solutions are as follows:

[0006] According to one aspect of this application, a method for training a reinforcement learning speech-text alignment model includes:

[0007] Obtain a labeled training set, and perform text alignment training on the pre-trained model based on the labeled training set to obtain a speech-text alignment model;

[0008] Based on the labeled training set, the value function model corresponding to the speech-text alignment model is trained by comparison to obtain the first reward model.

[0009] Obtain an unlabeled training set, and train a reward scoring model corresponding to the speech-text alignment model based on the unlabeled training set to obtain a second reward model;

[0010] The speech-text alignment model is trained using reinforcement learning based on the first reward model, the second reward model, and the unlabeled training set to obtain a target model, which is used to perform speech-text alignment on text data and speech data.

[0011] According to one aspect of this application, an apparatus for training a reinforcement learning speech-text alignment model includes:

[0012] The first training module is used to obtain a labeled training set and perform text alignment training on the pre-trained model based on the labeled training set to obtain a speech-text alignment model.

[0013] The second training module is used to pre-train the value function model corresponding to the speech-text alignment model based on the labeled training set to obtain the first reward model.

[0014] The third training module is used to obtain an unlabeled training set and pre-train the speech-text alignment model based on the unlabeled training set to obtain a second reward model.

[0015] The fourth training module is used to perform reinforcement learning training on the text alignment model based on the first reward model, the second reward model, and the unlabeled training set to obtain a target model for speech-text alignment.

[0016] In one exemplary embodiment, the first training module includes:

[0017] The data acquisition unit is used to acquire text data and corresponding voice data;

[0018] The segmentation unit is used to segment the speech data based on the word time annotations in the text data using a preset time threshold, to obtain speech data segments and corresponding text data segments to generate an annotated training set.

[0019] In one exemplary embodiment, the first training module further includes:

[0020] The prediction unit is used to input the labeled training set into the pre-trained model to predict the word time labels of the text data segment and obtain the probability distribution prediction value.

[0021] The loss value calculation unit is used to calculate the loss value between the predicted value of the probability distribution and the word time annotation of the preset text data segment based on the loss function;

[0022] The parameter update unit is used to update the model parameters of the pre-trained model based on the loss value, and repeatedly predict the word time labels of the text data segment until the pre-trained model meets the preset pre-training target, then stop training and obtain the text alignment model.

[0023] In one exemplary embodiment, the pre-trained model includes a text encoder, a speech encoder, and a modality fusion layer, wherein the text encoder is used to encode text data in the input data to obtain text modality feature vectors, the speech encoder is used to encode speech data in the input data to obtain speech modality features, and the modality fusion layer is used to fuse the text modality feature vectors and the speech modality feature vectors to obtain a hybrid modality feature vector.

[0024] In one exemplary embodiment, the second training module includes:

[0025] A value function model unit is used to construct a value function model based on the speech-text alignment model.

[0026] The similarity calculation unit is used to input the labeled training set into the value function model to calculate the similarity and obtain the similarity between the text data segments and the speech data segments in the labeled training set.

[0027] The first parameter tuning unit is used to input the similarity into the loss function to tune the parameter of the value function model, repeat the similarity calculation until the value function model meets the preset training objective, then stop training and obtain the first reward model.

[0028] In one exemplary embodiment, the third training module includes:

[0029] The scoring model unit is used to construct a scoring model based on the speech-text alignment model.

[0030] The matching calculation unit is used to acquire an unlabeled training set and input the unlabeled training set into the scoring model to perform speech-text matching calculation, thereby obtaining a reward value for the temporal correspondence between speech data and text data in the unlabeled training set.

[0031] The second parameter tuning unit is used to repeatedly input the reward value into the optimization algorithm to tune the scoring model, repeat the matching calculation until the scoring model meets the preset training objective, then stop training and obtain the second reward model.

[0032] In one exemplary embodiment, the fourth training module includes:

[0033] The temporal prediction unit is used to input the unlabeled training set into the text alignment model to perform temporal prediction and obtain the temporal prediction result.

[0034] The value function estimation unit is used to perform similarity calculation on the unlabeled training set based on the first reward model to obtain the value function estimate corresponding to the time series prediction result;

[0035] The reward value unit is used to perform speech-text matching calculation on the time-series prediction result based on the second reward model to obtain the reward value corresponding to the time-series prediction result.

[0036] The third parameter tuning unit is used to tune the speech-text alignment model and the first reward model based on the value function estimation and the reward value, repeat the time-series prediction until the speech-text alignment model meets the preset training objective, then stop training and obtain the target model.

[0037] According to one aspect of this application, an electronic device includes at least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; the computer-readable instructions are executed by one or more of the processors to cause the electronic device to implement the reinforcement learning speech-text alignment model training method as described above.

[0038] According to one aspect of this application, a storage medium stores computer-readable instructions thereon, which are executed by one or more processors to implement the reinforcement learning speech-text alignment model training method as described above.

[0039] According to one aspect of this application, a computer program product includes computer-readable instructions stored in a storage medium, wherein one or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, causing the electronic device to implement the reinforcement learning speech-text alignment model training method as described above.

[0040] The beneficial effects of the technical solution provided in this application are:

[0041] In the above technical solution, a first reward model, a second reward model, and a speech-to-text alignment model are trained using labeled and unlabeled training sets. The speech-to-text alignment model is then trained using reinforcement learning to obtain the target model. By using the first and second reward models to train the speech-to-text alignment model at different levels, the speech-to-text alignment performance of the target model is improved. Furthermore, only a small amount of labeled data from the labeled training set is needed to fully utilize a large amount of unlabeled data from the unlabeled training set, reducing dependence on labeled data and improving the applicability of the target model in environments with limited labeled training sets. This effectively solves the problem of low applicability of reinforcement learning speech-to-text alignment model training methods in related technologies. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 is a schematic diagram of the implementation environment according to this application;

[0044] Figure 2 is a flowchart illustrating a reinforcement learning speech-text alignment model training method according to an exemplary embodiment;

[0045] Figure 3 is a flowchart of step 210 in one embodiment of the embodiment corresponding to Figure 2;

[0046] Figure 4a is a flowchart of step 210 in one embodiment of the embodiment corresponding to Figure 2;

[0047] Figure 4b is a structural diagram of a pre-trained model according to an exemplary embodiment;

[0048] Figure 5a is a flowchart of step 230 in one embodiment of the embodiment corresponding to Figure 2;

[0049] Figure 5b is a structural diagram of a value function model according to an exemplary embodiment;

[0050] Figure 6a is a flowchart of step 250 in one embodiment of the embodiment corresponding to Figure 2;

[0051] Figure 6b is a structural diagram of a scoring model according to an exemplary embodiment;

[0052] Figure 7 is a flowchart of step 270 in one embodiment of the embodiment corresponding to Figure 2;

[0053] Figure 8 is a schematic diagram of a specific implementation of a reinforcement learning speech-text alignment model training method in an application scenario.

[0054] Figure 9 is a structural block diagram of a reinforcement learning speech-text alignment model training method apparatus according to an exemplary embodiment;

[0055] Figure 10 is a hardware structure diagram of a server according to an exemplary embodiment;

[0056] Figure 11 is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0057] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0058] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0059] As mentioned earlier, existing reinforcement learning speech-to-text alignment model training methods require a large amount of word-level alignment annotations. Because this word-by-word annotation requires precise time segmentation, the large amount of word-level alignment annotation data required is extremely time-consuming and expensive. Furthermore, the model performs poorly when only a small amount of word-level annotation data is available, resulting in limited applicability. Therefore, it is clear that related technologies still suffer from the drawback of limited applicability in reinforcement learning speech-to-text alignment model training methods.

[0060] Therefore, the reinforcement learning speech-text alignment model training method provided in this application can effectively improve the accuracy of the reinforcement learning speech-text alignment model training method. Accordingly, the reinforcement learning speech-text alignment model training method is applicable to the reinforcement learning speech-text alignment model training method device, which can be deployed on an electronic device. The electronic device can be a computer device configured with a von Neumann architecture, such as a desktop computer, laptop computer, server, etc.; the electronic device can also be an electronic device with central control function, such as a gateway; the electronic device can also refer to a portable mobile electronic device, such as a smartphone, tablet computer, etc.

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0062] Figure 1 is a schematic diagram of an implementation environment involved in an image processing method. It should be noted that this implementation environment is merely an example adapted to the present invention and should not be considered as providing any limitation on the scope of the invention.

[0063] The implementation environment includes a data acquisition terminal 110 and a server terminal 130.

[0064] Specifically, the acquisition terminal 110 can be considered a training set acquisition device, including but not limited to electronic devices with information acquisition functions such as recorders and text input devices. It can also be considered a device that integrates information acquisition and reinforcement learning speech-text alignment model training methods, including but not limited to electronic devices such as laptops and desktop computers that combine information acquisition and reinforcement learning speech-text alignment model training methods. That is to say, in some embodiments, the acquisition terminal 110 can acquire both labeled training sets and unlabeled training sets.

[0065] Server 130 can also be considered as the device for training reinforcement learning speech-to-text alignment models, including but not limited to electronic devices with reinforcement learning speech-to-text alignment model training capabilities such as smartphones, desktop computers, laptops, and servers. It can also be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide backend services, such as reinforcement learning speech-to-text alignment model training services, etc.

[0066] The server 130 and the data acquisition terminal 110 establish a network communication connection in advance via wired or wireless means, and data transmission between the server 130 and the data acquisition terminal 110 is realized through this network communication connection. The transmitted data includes, but is not limited to, labeled training sets and unlabeled training sets, etc.

[0067] In one application scenario, through the interaction between the acquisition terminal 110 and the server terminal 130, the acquisition terminal 110 acquires labeled training sets and unlabeled training sets, and uploads the labeled training sets and unlabeled training sets to the server terminal 130 to request the server terminal 130 to provide a reinforcement learning speech-text alignment model training method service.

[0068] At this point, for server 130, after receiving the labeled training set and unlabeled training set uploaded by acquisition terminal 110, it calls the reinforcement learning speech-text alignment model training method service, and trains the pre-trained model according to the labeled training set and unlabeled training set to obtain the target model for speech-text alignment of text data and speech data, thus solving the problem of low applicability of reinforcement learning speech-text alignment model training methods in related technologies.

[0069] Please refer to Figure 2. This application embodiment provides a reinforcement learning speech-text alignment model training method. This method is applicable to electronic devices, which can be the server 130 in the implementation environment shown in Figure 1, or a desktop computer, laptop computer, server, etc.

[0070] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0071] As shown in Figure 2, the method may include the following steps:

[0072] Step 210: Obtain the labeled training set, and perform text alignment training on the pre-trained model based on the labeled training set to obtain the speech-text alignment model.

[0073] The labeled training set includes text data and speech data that has a temporal correspondence with the text data. Furthermore, the start and end times of the corresponding speech data are labeled for different levels of text data, generating labeled data at different levels. For example, labeling the start and end times of each word in the text data with its corresponding speech data constitutes a word-level labeled training set.

[0074] In one possible implementation, the pre-trained model is the SPECTRA model.

[0075] Step 230: Based on the labeled training set, perform comparative learning training on the value function model corresponding to the speech-text alignment model to obtain the first reward model.

[0076] The value function model is a model built for speech-text alignment that can calculate the similarity of feature vectors from speech and text data segments at different scales. The first reward model, obtained by training the value function model on a labeled training set through comparative learning of speech and text data, can calculate the similarity between text and speech data and output the similarity as a reward value.

[0077] One possible implementation involves training the speech-text alignment model through comparative learning using a word-level labeled training set.

[0078] Step 250: Obtain an unlabeled training set, and train a reward scoring model on the scoring model corresponding to the speech-text alignment model based on the unlabeled training set to obtain a second reward model.

[0079] The unlabeled training set includes text data and speech data that has a temporal correspondence with the text data, but the start and end times of the speech data are not labeled in the text data. The scoring model is a model built for speech-text alignment models that scores the degree of matching between speech data and text data.

[0080] Step 270: Reinforcement learning training is performed on the speech-text alignment model based on the first reward model, the second reward model, and the unlabeled training set to obtain the target model.

[0081] The target model is used to perform speech-text alignment on text and speech data.

[0082] Specifically, the speech-to-text alignment model is trained using reinforcement learning on a high-level unlabeled training set based on the first and second reward models. During training, the speech-to-text alignment model outputs speech-to-text alignment results for the unlabeled training set. Then, the first and second reward models output reward values ​​based on the speech-to-text alignment results to perform gradient optimization on the speech-to-text alignment model, thereby training the model in a direction that improves the quality of speech-to-text alignment results, ultimately obtaining the target model.

[0083] One possible implementation involves training the speech-text alignment model using reinforcement learning via the PPO algorithm.

[0084] Through the above process, the first reward model trained on the labeled training set can output reward values ​​for text and speech data at a lower level than the second reward model trained on the unlabeled training set. By using the first and second reward models to perform reinforcement learning training on the speech-text alignment model at different levels, the speech-text alignment effect of the target model is improved. Furthermore, through multi-level training, the target model can perform speech-text alignment at different levels, improving the model's generality. Moreover, it only requires a small amount of labeled data in the labeled training set to fully utilize a large amount of unlabeled data in the unlabeled training set, thus reducing its dependence on labeled data and improving the applicability of the target model in environments with a small amount of labeled training set.

[0085] In an exemplary embodiment, as shown in FIG3, step 210 may include the following steps:

[0086] Step 211: Obtain the text data and the corresponding audio data of the text data.

[0087] Step 213: Based on the word time annotations in the text data, the speech data is segmented according to a preset time threshold to obtain speech data segments and corresponding text data segments to generate an annotated training set.

[0088] The text data in the labeled training set is sequential, and the text data has corresponding word time annotations for the speech data. The temporal correspondence between the text data and the speech data can be determined through word time annotations.

[0089] In one possible implementation, a word-level time threshold is preset, and the text data is then divided into multiple text data segments with a duration within the preset time threshold. The corresponding speech data segment is determined by the word time annotation of the text data segment.

[0090] Through the above process, by dividing the text data using a preset threshold, the text data is transformed from sentence-level to lower-level word-level text data segments, thereby enabling annotation of the text data at the lower level. At the same time, the division of text data can also truncate the text data from the beginning or end of words, thus preventing errors due to the annotation of the starting position of the speech data and the text data.

[0091] In an exemplary embodiment, as shown in FIG4a, step 210 may further include the following steps:

[0092] Step 215: Input the labeled training set into the pre-trained model to predict the word time annotations of the text data segment and obtain the probability distribution prediction value.

[0093] In an exemplary embodiment, as shown in FIG4b, the pre-trained model includes a text encoder, a speech encoder, and a modality fusion layer. The text encoder is used to encode text data in the input data to obtain text modality feature vectors, the speech encoder is used to encode speech data in the input data to obtain speech modality features, and the modality fusion layer is used to fuse the text modality feature vectors and the speech modality feature vectors to obtain a mixed modality feature vector.

[0094] It should be noted that the word time labels for text data segments can be real-valued start and end times. When training a pre-trained model using the policy gradient algorithm, it is necessary to convert the real-valued start and end time predictions into probability distributions over the entire "action space".

[0095] For example, when the maximum length of word time annotation is 10 seconds, the convolution can produce a feature vector sequence of corresponding word time annotations with a maximum length of 100. Each word in the text data corresponds to at least one feature vector in the speech data, and the duration of each word is greater than or equal to 0.1 seconds. Each feature vector corresponds to approximately 0.1 seconds of audio, thus transforming the time prediction form into a 100-classification form.

[0096] Step 217: Calculate the loss value between the predicted probability distribution value and the word time annotation of the preset text data segment based on the loss function.

[0097] In one possible implementation, the pre-trained model is trained with MLM, MAM, TPP, and CRS, and the total loss function is the sum of the four pre-training objectives.

[0098] Step 219: Update the model parameters of the pre-trained model based on the loss value, and repeat the prediction of word time labels for text data segments until the pre-trained model meets the preset pre-training target, then stop training and obtain the text alignment model.

[0099] The pre-training objectives of the pre-trained model include predicting the start and end times of the text words corresponding to each text data segment in the corresponding speech data waveform for the Temporal Position Prediction (TPP) task of speech-text alignment, and the Cross-modal Response Selection (CRS) task for judging whether the dialogue response conforms to the dialogue history in two modalities.

[0100] Specifically, the overall pre-training objective function is formulated as follows:

[0101]

[0102] In one possible implementation, the start and end times of the word time annotations in the labeled training set are multiplied by 10 and rounded down. The result of multiplying the end time by 10 is an integer between 0 and 99, and the result of multiplying the start time by 10 is an integer between 1 and 100. The loss function formula for TPP in this case is:

[0103]

[0104] Where L is 10 seconds, CE 100 (x, y) represents the cross-entropy loss for 100 categories, where the true label y is an integer between [0, 99], and x is a vector of length 100, representing the probability distribution over 100 possible starting times.

[0105] Through the above process, the word time annotations of text data segments are predicted using probability distribution prediction values, which ensures the effectiveness of subsequent reinforcement learning training. At the same time, the monotonic temporal characteristics of speech data relative to text data are used to train the pre-trained model, thereby improving the model training effect.

[0106] In an exemplary embodiment, as shown in FIG5a, step 230 may include the following steps:

[0107] Step 231: Construct a value function model based on the speech-text alignment model.

[0108] In one possible implementation, as shown in Figure 5b, the value function model consists of a speech encoder and a text encoder, and the initial parameters of the value function model are the same as those of the speech encoder and text encoder in the speech-text alignment model.

[0109] Step 233: Calculate the similarity between the input value function model of the labeled training set to obtain the similarity between the corresponding text data segments and speech data segments in the labeled training set.

[0110] Specifically, the similarity calculation process of the value function model is as follows: input text data into a text encoder to obtain text modal feature vectors, input speech data into a speech encoder to obtain speech modal feature vectors, and then calculate the feature vector similarity between each text modal feature vector and the speech modal feature vector.

[0111] Step 235: Input the similarity into the loss function to tune the parameter of the value function model. Repeat the similarity calculation until the value function model meets the preset training objective, then stop training and obtain the first reward model.

[0112] Specifically, by selecting speech data segments aligned with text data segments as positive samples and inputting speech data segments misaligned with text data segments as negative samples into the value function model for similarity calculation, the difference between the output of the value function model and the positive and negative samples guides the parameter tuning process of the value function model. This reduces the distance between matching speech and text feature vectors and increases the distance between non-matching vectors in the cross-modal feature space, thereby making the feature representation spaces of text data and speech data similar.

[0113] The training objective formula is as follows:

[0114]

[0115] in, For negative samples, σsample represents the Sigmoid activation function.

[0116] In one possible implementation, the Bradley-Terry win rate model is used as the loss function in the value function model training process.

[0117] Through the above process, a value function model is trained at a low level using a labeled training set, enabling comparative learning of speech and text data. The similarity between text data segments and speech segments is calculated and used as a word-level reward to improve model accuracy.

[0118] In an exemplary embodiment, as shown in FIG6a, step 250 may include the following steps:

[0119] Step 251: Construct a scoring model based on the speech-text alignment model.

[0120] In one possible implementation, as shown in Figure 6b, the scoring model consists of a speech encoder, a text encoder, a modality fusion layer, and a reward output head. Except for the reward output head, the model structure is the same as that of the speech-text alignment model.

[0121] In one possible implementation, the reward output head is a fully connected layer with an output size of 1. During initialization, the reward output head is initialized randomly, and the speech encoder, text encoder, and modality fusion layer are initialized using the parameters of the speech-text alignment model.

[0122] Step 253: Obtain an unlabeled training set and input it into the scoring model to calculate the speech-text matching, thereby obtaining a reward value for the temporal correspondence between the speech data and the text data in the unlabeled training set.

[0123] Specifically, text data from the unlabeled training set is input into the text encoder, and speech data is input into the speech encoder. A sequence of mixed feature vectors is obtained from the modality fusion layer. The first feature vector is passed through a fully connected layer with an output size of 1 to obtain the reward value.

[0124] Step 255: Repeatedly input the reward value into the optimization algorithm to tune the scoring model. Repeat the matching calculation until the scoring model meets the preset training objective, then stop training and obtain the second reward model.

[0125] Specifically, speech and text data with correct temporal correspondences in the unlabeled training set are used as positive samples, while speech and text data with varying numbers of incorrect temporal correspondences are used as negative samples and input into the scoring model. The scoring model is then tuned based on the difference between the reward value of the scoring model and the positive and negative samples.

[0126] It should be noted that the inventors realized that in the existing technology, the prediction result for speech-text matching is a binary classification task. The way to judge speech-text alignment is whether the speech and text correspond to each other. If they correspond, a high score is given, and if they do not correspond, a low score is given. However, in the training process of the scoring model, the scoring model needs to make correct estimations and calculate reward values ​​for the entire time-series prediction results. Therefore, it is necessary to pay attention to all samples that are not predicted completely correctly. For example, the reward value of samples with high prediction accuracy should be greater than that of samples with low prediction accuracy.

[0127] Therefore, the rating model is trained using the Preference Rank Optimization (PRO) algorithm. Specifically, PRO optimizes the rating model using multi-level optimization algorithms based on the preference sequences of samples in the unlabeled training set. For example, given a preference sequence... The preference sequence optimization algorithm decomposes it into: Superior Superior And so on, until... Superior

[0128] The training objective formula for the scoring model is:

[0129]

[0130] in, The maximum reward value set for the experiment.

[0131] In one possible implementation, the scoring model loss is optimized using the Bradley-Terry win rate model.

[0132] Through the above process, a second reward model is obtained that can calculate reward values ​​for multiple sample sorting types, thereby improving the accuracy of text-to-speech alignment.

[0133] In an exemplary embodiment, as shown in FIG7, step 270 may include the following steps:

[0134] Step 271: Input the unlabeled training set into the text alignment model for time-series prediction and obtain the time-series prediction results.

[0135] Specifically, text and speech data are input into the text encoder and speech encoder of the text alignment model to obtain temporal prediction results. The temporal prediction results are obtained by ranking the text and speech data in the unlabeled training set, and the reward model is trained based on this ranking to provide rewards according to the order of the ranking.

[0136] Step 273: Calculate the similarity of the unlabeled training set based on the first reward model to obtain the value function estimate of the corresponding time series prediction result.

[0137] Specifically, text data and speech data are input into the first reward model, and the corresponding text sub-words and speech intervals are extracted from the model output based on the time-series prediction results to calculate the value function estimation.

[0138] Step 275: Calculate the speech-text matching of the time-series prediction results based on the second reward model to obtain the reward value of the corresponding time-series prediction results.

[0139] In one possible implementation, the speech data is transformed into a sequence based on the temporal prediction results. <sep>The input sequence, along with the text data, is input into the second reward model to obtain the reward value.

[0140] Step 277: Based on the value function estimation and reward value, tune the parameters of the speech-text alignment model and the first reward model, repeating the temporal prediction until the speech-text alignment model meets the preset training objective.

[0141] Stop training and obtain the target model.

[0142] Specifically, the parameters of the second reward model are fixed, and the first reward model is iteratively evaluated using value function estimation. Simultaneously, a new speech-to-text alignment reference model with unoptimized parameters is created. During training, the first reward model continuously uses the value function estimation objective function to optimize the model parameters of the speech-to-text alignment model until the training objective is met.

[0143] The optimization objective formula for the speech-text alignment model is:

[0144]

[0145] in:

[0146]

[0147]

[0148] In one possible implementation, the FVI method is used to iteratively estimate the value function of the first reward model, where the objective function of the first reward model is formulated as follows:

[0149]

[0150] In one possible implementation, pre-trained tasks such as MLM, MAM, and CRS are incorporated into the speech-text alignment model training process to prevent overfitting. The objective function is:

[0151]

[0152] Through the above process, the first and second reward models generate corresponding reward values ​​for each speech-to-text alignment attempt. Reward values ​​at different levels are correlated, and the contextual fitting information of higher-level rewards is integrated into the lower-level reward models to guide the training of the speech-to-text alignment model, thereby enhancing the speech-to-text alignment capability of the generated target model. Figure 8 is a schematic diagram illustrating the specific implementation of a reinforcement learning speech-to-text alignment model training method in an application scenario. In this application scenario, the SPECTRA pre-trained model is first trained with word-level alignment annotations using a training set to obtain the speech-to-text alignment model.

[0153] Based on the speech-text alignment model, the modality fusion layer is removed to obtain the value function model. Then, a contrastive learning training method with word-level alignment annotations is used to train the word-level alignment, resulting in the first reward model at the word level. A fully connected layer with an output size of 1 is added to the speech-text alignment model to obtain the scoring model. The scoring model is trained using one positive sample with all correct temporal sequences and Ns negative samples with varying numbers of temporal errors to determine whether text sentences and speech segments match, resulting in the second reward model at the sentence level. Here, the positive sample is denoted as x0=< negative samples Where j = 1, 2, ..., Ns, For any j < k, we have The number of timing errors is less than Let rj be the sentence-level reward output corresponding to sample xj. Where Ns = 5 and M = 5.

[0154] In the training process of the speech-text alignment model, the text {t(ik), t(ik-1), ..., t(i-1), ti} and the speech {s(i-1), si} are first input into the text encoder and speech encoder of the speech-text alignment model to obtain the temporal prediction result P. The text t(i-1), ti and the speech s(i-1), si are then input into the first reward model, and the corresponding text sub-words and speech intervals are extracted at the model output based on the temporal result to calculate the estimate of the value function. The speech data is transformed into an input sequence with <SEP> according to the temporal prediction result, and the text and speech are input into the second reward model to obtain the sentence-level reward ri. The FVI method is used to iteratively estimate the value function of the first reward model, and a new reference model with unoptimized parameters is created. During the training process, the first reward model continuously uses the value function estimation objective function for optimization, while the parameters of the second reward model remain unchanged. In addition, pre-training tasks such as MLM, MAM, and CRS are also added to prevent overfitting. Finally, the target model is obtained after training.

[0155] In this application scenario, fine-tuning and testing were conducted on internationally recognized speech-text multimodal datasets. The RLMR model, encompassing six datasets (MOSI, MOSEI, MELD, IEMOCAP, MINTREC, SPOKENWOZ) with four downstream tasks (MSA, ERC, SLU, DST), was compared with existing methods SPECTRA, the best models on individual datasets (MIB, BBFN, M2FNET, MAG-BERT, SPACE), and various baseline models (RoBERTa-base, WavLM-base-plus, CTAL-base). This application outperformed other models in various accuracy metrics and outperformed SPECTRA in most indicators, with only slight differences in a few. Further ablation experiments also demonstrated the effectiveness and contribution of each part of this application. These experiments provide strong support for the feasibility and effectiveness of this application, showcasing its application potential in speech-text alignment scenarios with limited word-level labeled data.

[0156] Please refer to Figure 9. In this embodiment of the application, a reinforcement learning speech-text alignment model training device 700 is provided, including but not limited to: a first training module 910, a second training module 930, a third training module 950, and a fourth training module 970.

[0157] The first training module 910 is used to obtain a labeled training set and to perform text alignment training on the pre-trained model based on the labeled training set to obtain a speech-text alignment model.

[0158] The second training module 930 is used to pre-train the value function model corresponding to the speech-text alignment model based on the labeled training set to obtain the first reward model.

[0159] The third training module 950 is used to obtain an unlabeled training set and pre-train the speech-text alignment model based on the unlabeled training set to obtain the second reward model.

[0160] The fourth training module 970 is used to perform reinforcement learning training on the text alignment model based on the first reward model, the second reward model and the unlabeled training set to obtain the target model for speech-text alignment.

[0161] It should be noted that the reinforcement learning speech-text alignment model training method device provided in the above embodiments is only illustrated by the division of the above functional modules when performing the reinforcement learning speech-text alignment model training method. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the reinforcement learning speech-text alignment model training method device will be divided into different functional modules to complete all or part of the functions described above.

[0162] Furthermore, the reinforcement learning speech-text alignment model training method apparatus provided in the above embodiments and the embodiments of the reinforcement learning speech-text alignment model training method belong to the same concept. The specific way each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0163] Figure 10 shows a schematic diagram of a server according to an exemplary embodiment. This server is suitable for server 130 in the implementation environment shown in Figure 1.

[0164] It should be noted that this server is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Nor should this server be interpreted as requiring or needing to have one or more of the components of the exemplary server 2000 shown in Figure 10.

[0165] The hardware structure of server 2000 can vary greatly depending on the configuration or performance. As shown in Figure 10, server 2000 includes: power supply 210, interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0166] Specifically, power supply 210 is used to provide operating voltage for the various hardware devices on server 2000.

[0167] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. For example, it enables interaction between terminal 100 and server 200 in the implementation environment shown in Figure 1.

[0168] Of course, in other examples adapted in this application, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, as shown in Figure 10, but this is not intended to be a specific limitation.

[0169] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0170] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the server 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0171] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task. It may include at least one module (not shown in Figure 10), each module may contain computer-readable instructions for server 2000. For example, a reinforcement learning speech-text alignment model training method apparatus can be regarded as application 253 deployed on server 2000.

[0172] Data 255 can be voice data, text data, etc. stored on a disk, and stored in memory 250.

[0173] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby enabling the computation and processing of massive amounts of data 255 in the memory 250. For example, a reinforcement learning speech-text alignment model training method can be implemented by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0174] Furthermore, this application can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or combination thereof.

[0175] Please refer to Figure 11. This application embodiment provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc.

[0176] In Figure 11, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0177] Data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 11, but this does not indicate that there is only one bus or one type of bus.

[0178] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0179] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0180] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 400, but not limited thereto.

[0181] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.

[0182] The computer-readable instructions are executed by one or more processors 4001 to implement the reinforcement learning speech-text alignment model training method in the above embodiments.

[0183] Furthermore, this application provides a storage medium storing computer-readable instructions, which are executed by one or more processors to implement the reinforcement learning speech-text alignment model training method described above.

[0184] This application provides a computer program product, which includes computer-readable instructions stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the reinforcement learning speech-text alignment model training method described above.

[0185] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0186] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.< / sep>

Claims

1. A method for training a reinforcement learning speech-text alignment model, characterized in that, include: Obtain a labeled training set, and perform text alignment training on the pre-trained model based on the labeled training set to obtain a speech-text alignment model; perform comparative learning training on the value function model corresponding to the speech-text alignment model based on the labeled training set to obtain a first reward model; Obtain an unlabeled training set, and train a reward scoring model corresponding to the speech-text alignment model based on the unlabeled training set to obtain a second reward model; train the speech-text alignment model using reinforcement learning based on the first reward model, the second reward model, and the unlabeled training set to obtain a target model, which is used to perform speech-text alignment on text data and speech data.

2. The method as described in claim 1, characterized in that, The process of obtaining the labeled training set includes: obtaining text data and corresponding speech data; segmenting the speech data based on word time annotations in the text data using a preset time threshold to obtain speech data segments and corresponding text data segments to generate a labeled training set.

3. The method as described in claim 1, characterized in that, The step of training a pre-trained model for text alignment based on the labeled training set to obtain a speech-text alignment model includes: inputting the labeled training set into the pre-trained model to predict word time annotations of the text data segment and obtaining a probability distribution prediction value; calculating a loss value between the probability distribution prediction value and the preset word time annotations of the text data segment based on a loss function; updating the model parameters of the pre-trained model based on the loss value; repeating the prediction of word time annotations of the text data segment until the pre-trained model meets the preset pre-training objective, then stopping the training to obtain the text alignment model.

4. The method as described in claim 1, characterized in that, The pre-trained model includes a text encoder, a speech encoder, and a modality fusion layer. The text encoder encodes the text data in the input data to obtain text modality feature vectors, the speech encoder encodes the speech data in the input data to obtain speech modality features, and the modality fusion layer fuses the text modality feature vectors and the speech modality feature vectors to obtain a mixed modality feature vector.

5. The method as described in claim 1, characterized in that, The step of performing comparative learning training on the value function model corresponding to the speech-text alignment model based on the labeled training set to obtain a first reward model includes: constructing a value function model based on the speech-text alignment model; inputting the labeled training set into the value function model to calculate similarity, obtaining the similarity between text data segments and speech data segments in the labeled training set; inputting the similarity into a loss function to tune the parameters of the value function model, repeating the similarity calculation until the value function model meets the preset training objective, stopping the training, and obtaining the first reward model.

6. The method as described in claim 1, characterized in that, The step of obtaining an unlabeled training set and training a scoring model corresponding to the speech-text alignment model with a reward score based on the unlabeled training set to obtain a second reward model includes: constructing a scoring model based on the speech-text alignment model; obtaining an unlabeled training set and inputting the unlabeled training set into the scoring model to perform speech-text matching calculation to obtain a reward value for the temporal correspondence between speech data and text data in the unlabeled training set; repeatedly inputting the reward value into an optimization algorithm to tune the scoring model; repeating the matching calculation until the scoring model meets a preset training objective, then stopping the training to obtain the second reward model.

7. The method as described in claim 1, characterized in that, The step of training the text alignment model using reinforcement learning based on the first reward model, the second reward model, and the unlabeled training set to obtain a target model for speech-text alignment includes: inputting the unlabeled training set into the text alignment model for temporal prediction to obtain a temporal prediction result; calculating similarity on the unlabeled training set based on the first reward model to obtain a value function estimate corresponding to the temporal prediction result; calculating speech-text matching based on the second reward model to obtain a reward value corresponding to the temporal prediction result; tuning the parameters of the speech-text alignment model and the first reward model based on the value function estimate and the reward value; repeating the temporal prediction until the speech-text alignment model meets a preset training objective, then stopping training to obtain the target model.

8. A device for training a reinforcement learning speech-text alignment model, characterized in that, include: The first training module is used to obtain a labeled training set and perform text alignment training on the pre-trained model based on the labeled training set to obtain a speech-text alignment model. The second training module is used to pre-train the value function model corresponding to the speech-text alignment model based on the labeled training set to obtain the first reward model. The third training module is used to obtain an unlabeled training set and pre-train the speech-text alignment model based on the unlabeled training set to obtain a second reward model. The fourth training module is used to perform reinforcement learning training on the text alignment model based on the first reward model, the second reward model, and the unlabeled training set to obtain a target model for speech-text alignment.

9. An electronic device, characterized in that, include: At least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; the computer-readable instructions are executed by one or more of the processors to cause the electronic device to implement the reinforcement learning speech-text alignment model training method as described in any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that, The computer-readable instructions are executed by one or more processors to implement the reinforcement learning speech-text alignment model training method as described in any one of claims 1 to 7.

Images