Model training method and device, voice text conversion method and device, equipment and medium
Through a unified model architecture and training sample set, the speech and text models are jointly trained, which solves the independent and cross-domain problems of the model architecture in the existing technology, and improves the accuracy and generalization capabilities of the model.
Patent Information
- Application Number
- CN202411538777.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-02-18
AI Technical Summary
In the prior art, the cross-modal task model architecture between speech and text is independent, resulting in the inability to share model parameters, increasing model complexity and memory footprint, and cross-domain problems, resulting in the degradation of the model's performance on specific tasks.
By obtaining the training sample set, including source text data and source speech data, as well as textual task labels and speech-based task labels for each data, the model is jointly trained and multi-task learning is used to eliminate training data for different tasks. inconsistencies between sources.
It improves the accuracy and generalization ability of model training, avoids error transmission caused by cross-domain problems and cascading operations, and enhances the accuracy of the model's processing of multi-task speech text conversion.
Smart Images
Figure CN119479612A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training method, a speech-to-text conversion method, a device, a device and a medium. Background Art
[0002] With the continuous development of artificial intelligence technology, cross-modal tasks between speech and text have been widely used. These tasks cover a variety of task types such as speech to text, text to text, text to speech, and speech to speech, aiming to achieve effective conversion and understanding between different modal information. In the prior art, independent model architectures, training objectives, and reasoning methods are usually used to train models of corresponding task types to handle the corresponding tasks. However, this method has certain limitations. For example, the model architectures between different tasks are independent of each other, resulting in the inability to share model parameters, thereby increasing the complexity and memory usage of the model; the training data sources of different tasks are often inconsistent, and cross-domain problems may occur, which may lead to a decrease in the performance of the model on specific tasks, or even failure to work properly.
[0003] Therefore, how to use voice data and text data together to improve the accuracy of model training and cope with multi-task processing has become an urgent problem to be solved. Summary of the invention
[0004] Based on this, a model training method, a speech-to-text conversion method, an apparatus, a device and a medium are provided to solve the problem of how to use speech data and text data together to improve the accuracy of model training, thereby coping with the problem of multi-task processing.
[0005] In a first aspect, an embodiment of the present invention provides a model training method, comprising: Acquire a training sample set, wherein the training sample set includes source text data and source voice data, as well as a textual task label and a voiced task label of each data; Input the source text data into a text encoder of a preset model, decode it through a first decoder of the preset model to output a first result, input the source voice data into a voice encoder of the preset model, decode it through the first decoder of the preset model to output a second result; According to the first result, the second result and the textual task label of the corresponding data, the parameters of the preset model are optimized to obtain a pre-trained preset model; Input the source text data into the text encoder of the pre-trained preset model, decode the data through the first decoder and the second decoder of the pre-trained preset model to output a third result, input the source voice data into the voice encoder of the pre-trained preset model, decode the data through the first decoder and the second decoder of the pre-trained preset model to output a fourth result; According to the third result, the fourth result and the voice-based task label of the corresponding data, the parameters of the pre-trained preset model are optimized to obtain a trained preset model.
[0006] In a second aspect, an embodiment of the present invention provides a speech-to-text conversion method, comprising: Obtaining a trained preset model obtained by the model training method of the first aspect above; Get the data to be converted and the corresponding task type; Detecting whether the data to be converted is text data or voice data, and detecting whether the task type is a text-based task or a voice-based task; If it is detected that the data to be converted is text data or voice data, and it is detected that the task type is a textual task or a voice task, the data to be converted is input into the trained preset model, so that the trained preset model performs conversion according to the task type to obtain a conversion result.
[0007] In a third aspect, an embodiment of the present invention provides a model training device, comprising: A sample acquisition module, used to acquire a training sample set, wherein the training sample set includes source text data and source voice data, as well as a textual task label and a voiced task label of each data; A first training module is used to input the source text data into a text encoder of a preset model, decode the source text data through a first decoder of the preset model to output a first result, and input the source speech data into a speech encoder of the preset model, decode the source speech data through the first decoder of the preset model to output a second result; A first optimization module, configured to optimize parameters of the preset model according to the first result, the second result, and the textual task label of the corresponding data, to obtain a pre-trained preset model; A second training module is used to input the source text data into the text encoder of the pre-trained preset model, decode and output a third result through the first decoder and the second decoder of the pre-trained preset model, input the source voice data into the voice encoder of the pre-trained preset model, and decode and output a fourth result through the first decoder and the second decoder of the pre-trained preset model; The second optimization module is used to optimize the parameters of the pre-trained preset model according to the third result, the fourth result and the voice-based task label of the corresponding data to obtain a trained preset model.
[0008] In a fourth aspect, an embodiment of the present invention provides a speech-to-text conversion device, comprising: A model acquisition module, used to acquire a trained preset model obtained by the model training method of the first aspect above; A data acquisition module is used to obtain the data to be converted and the corresponding task type; A detection module, used to detect whether the data to be converted is text data or voice data, and to detect whether the task type is a text-based task or a voice-based task; A conversion module is used to input the data to be converted into the trained preset model if it is detected that the data to be converted is text data or voice data, and if it is detected that the task type is a textual task or a voice-based task, so that the trained preset model performs conversion according to the task type to obtain a conversion result.
[0009] In a fifth aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the model training method of the first aspect or the speech-to-text conversion method of the second aspect is implemented.
[0010] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the model training method of the first aspect or the speech-to-text conversion method of the second aspect is implemented.
[0011] The present invention is different from the technical effects achieved by the prior art scheme: the model training method of the present invention inputs the source text data into the text encoder of the preset model, decodes and outputs the first result through the first decoder of the preset model, inputs the source voice data into the voice encoder of the preset model, decodes and outputs the second result through the first decoder of the preset model, optimizes the parameters of the preset model according to the first result, the second result and the textual task label of the corresponding data, obtains the pre-trained preset model, inputs the source text data into the text encoder of the pre-trained preset model, decodes and outputs the third result through the first decoder and the second decoder of the pre-trained preset model, inputs the source voice data into the voice encoder of the pre-trained preset model, decodes and outputs the fourth result through the first decoder and the second decoder of the pre-trained preset model, optimizes the parameters of the pre-trained preset model according to the third result, the fourth result and the voiced task label of the corresponding data, and obtains the trained preset model. When performing voice-to-text conversion according to the trained preset model, if it is detected that the data to be converted is text data or voice data, and the task type is detected to be a textual task or a voiced task, the data to be converted is input into the trained preset model to obtain the conversion result.
[0012] Among them, the voice data and text data are used together, and the model is jointly trained and multi-task learned through a unified training sample set and a unified model architecture, which eliminates the inconsistency between the training data sources of different tasks, avoids cross-domain problems, and avoids the error transmission caused by the cascade operation of first completing a task to obtain the result and then using another model to complete the next task. This enables the model to learn the characteristics of different tasks at the same time, improves the accuracy and generalization ability of model training, and thus improves the processing accuracy of the trained model for multi-task speech-to-text conversion. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0014] Figure 1 It is a schematic diagram of an application environment of a model training method and a speech-to-text conversion method provided in the first embodiment of the present invention; Figure 2 It is a flowchart of a model training method provided in Embodiment 2 of the present invention; Figure 3 This is an overall architecture diagram of a preset model provided in Embodiment 2 of the present invention; Figure 4It is a flowchart of a model training method provided in Embodiment 3 of the present invention; Figure 5 It is a flowchart of a model training method provided by Embodiment 4 of the present invention; Figure 6 1 is a flow chart of a method for converting speech to text provided in Embodiment 5 of the present invention; Figure 7 It is a flowchart of a speech-to-text conversion method provided by Embodiment 6 of the present invention; Figure 8 1 is a flow chart of a method for converting speech to text provided in Embodiment 7 of the present invention; Fig. 9 is a structural schematic diagram of a model training device provided by Embodiment 8 of the present invention; Fig.10 It is a structural schematic diagram of a speech-to-text conversion device provided by Embodiment 9 of the present invention; Fig.11 It is a structural diagram of a computer device provided in Embodiment 10 of the present invention. DETAILED DESCRIPTION
[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0016] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0017] like Figure 1As shown, it is a schematic diagram of an application environment of a model training method and a speech-to-text conversion method provided in the first embodiment of the present invention, wherein the server communicates with the client, the server provides the functions of model training and speech-to-text conversion, the client sends a training sample set to the server so that the server can perform model training, and the client sends the data to be converted and the corresponding task type to the server so that the server can perform speech-to-text conversion. The client includes but is not limited to PDAs, desktop computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, cloud computer devices, personal digital assistants (PDAs), and other devices. The computer device corresponding to the server can be implemented with an independent server or a server cluster consisting of multiple servers.
[0018] like Figure 2 FIG. 1 is a flow chart of a model training method provided in Embodiment 2 of the present invention, comprising the following steps: Step S201: Obtain a training sample set.
[0019] In this embodiment, the training sample set includes source text data and source voice data, as well as a textual task label and a voiced task label for each data. The training model is to train multiple associated voice-to-text conversion tasks. The voice-to-text conversion tasks can be divided into two categories: textual tasks and voiced tasks. Textual tasks can include text-to-text tasks and voiced-to-text tasks. Voiced tasks can include text-to-speech tasks and voiced-to-speech tasks. The source text data can refer to the original text data that has not been converted into textual tasks and voiced tasks. The source voice data can refer to the original voice data that has not been converted into textual tasks and voiced tasks. The textual task label can be used to represent the goal that the model should achieve in the textual task, and the voiced task label can be used to represent the goal that the model should achieve in the voiced task.
[0020] For example, for a text-to-text task, English text data is converted into Chinese text data, then the corresponding source text data is English text data, and the textualization task label corresponding to the source text data is Chinese text data.
[0021] Step S202: input the source text data into the text encoder of the preset model, decode it through the first decoder of the preset model to output a first result, input the source voice data into the voice encoder of the preset model, decode it through the first decoder of the preset model to output a second result.
[0022] In this embodiment, the preset model may refer to an initial model that is pre-set for model training, the text encoder may refer to an encoder that vectorizes text data, the speech encoder may refer to an encoder that vectorizes speech data, the first decoder may refer to decoding the vector representation generated by the text encoder and the speech encoder to generate a textual task label corresponding to the source data, the first result may refer to the training prediction result of the text encoder and the first decoder of the preset model performing textual conversion on the source text data during the training process, and the second result may refer to the training prediction result of the speech encoder and the first decoder of the preset model performing textual conversion on the source speech data during the training process.
[0023] Specifically, in the process of training the preset model, the source text data is input into the text encoder of the preset model, and the first result is output through the first decoder of the preset model; the source voice data is input into the voice encoder of the preset model, and the second result is output through the first decoder of the preset model.
[0024] Step S203: Optimize the parameters of the preset model according to the first result, the second result and the textual task labels of the corresponding data to obtain a pre-trained preset model.
[0025] In this embodiment, the pre-trained preset model may refer to a preset model after parameter optimization of the text encoder, speech encoder and first decoder of the preset model according to the first result, the second result and the textual task label of the corresponding data.
[0026] Specifically, according to the preset loss function, the error value between the first result and the textual task label corresponding to the source text data is calculated, and the error value between the second result and the textual task label corresponding to the source speech data is calculated. Based on the two calculated error values, the text encoder, speech encoder and first decoder of the preset model are optimized in parameters to obtain a pre-trained preset model.
[0027] Step S204: Input the source text data into the text encoder of the pre-trained preset model, decode and output the third result through the first decoder and the second decoder of the pre-trained preset model, input the source voice data into the voice encoder of the pre-trained preset model, decode and output the fourth result through the first decoder and the second decoder of the pre-trained preset model.
[0028] In this embodiment, the second decoder may refer to decoding the textual data generated by the first decoder to generate a voiced task label corresponding to the source data, wherein the second decoder may include a text-to-unit encoder, a unit decoder and a voice coder, the third result may refer to the training prediction result of the speech conversion of the source text data by the text encoder, the first decoder and the second decoder of the pre-trained preset model during the training process, and the fourth result may refer to the training prediction result of the speech conversion of the source speech data by the pre-trained preset model during the training process.
[0029] Specifically, in the process of training the pre-trained preset model, the source text data is input into the text encoder, and the third result is output through the second decoder of the pre-trained preset model; the source voice data is input into the voice encoder, and the fourth result is output through the second decoder of the pre-trained preset model.
[0030] Step S205: Optimize the parameters of the pre-trained preset model according to the third result, the fourth result and the voice-based task label of the corresponding data to obtain a trained preset model.
[0031] In this embodiment, the trained preset model may refer to a preset model that meets the model training termination condition. The training termination condition may be that the loss function value drops below a preset threshold. At this time, it can be considered that the model has achieved sufficient accuracy and the training can be terminated.
[0032] Specifically, according to the preset loss function, the error value between the third result and the speech task label corresponding to the source text data is calculated, and the error value between the fourth result and the speech task label corresponding to the source speech data is calculated. Based on the two calculated error values, the text encoder, speech encoder, first decoder and second decoder of the pre-trained preset model are optimized in parameters to obtain a trained preset model.
[0033] For example, Figure 3 As shown, it is an overall architecture diagram of a preset model provided in Embodiment 2 of the present invention. The preset model is composed of a text encoder, a speech encoder, a first decoder and a second decoder, wherein the second decoder is composed of a text-to-unit encoder, a unit decoder and a vocoder, and the preset model is trained according to the training sample set to obtain a trained preset model, and the trained preset model can process textualization tasks and speech tasks of text data and speech data, wherein the textualization tasks may include text-to-text tasks and speech-to-text tasks, and the speech tasks may include text-to-speech tasks and speech-to-speech tasks.
[0034] The overall process of voice-to-text conversion based on the trained preset model can be: 1) When it is detected that the data to be converted is text data and the task type is a text conversion task, the text data is input into a text encoder of a trained preset model, the text data is encoded by the text encoder to obtain a text vector, and the first decoder decodes the text vector generated by the text encoder to generate a conversion result of the text-to-text task corresponding to the text data; 2) When it is detected that the data to be converted is speech data and the task type is a speech conversion task, the speech data is input into a speech encoder of a trained preset model, the speech data is encoded by the speech encoder to obtain a speech vector, and the first decoder decodes the speech vector generated by the speech encoder to generate a conversion result of the speech-to-text task corresponding to the speech data; 3) When it is detected that the data to be converted is text data and the task type is a speech task, the text data is input into the text encoder of the trained preset model, the text data is encoded by the text encoder to obtain a text vector, the first decoder decodes the text vector generated by the text encoder to generate a subword sequence, the text-to-unit encoder encodes the subword sequence into a speech unit, the unit decoder decodes the speech unit to generate an audio feature, and the vocoder synthesizes the conversion result of the corresponding text-to-speech task according to the audio feature; 4) When it is detected that the data to be converted is speech data and the task type is a voice task, the speech data is input into the speech encoder of the trained preset model, the speech data is encoded by the speech encoder to obtain a speech vector, the first decoder decodes the text vector generated by the text encoder to generate a subword sequence, the text-to-unit encoder encodes the subword sequence into a speech unit, the unit decoder decodes the speech unit to generate audio features, and the vocoder synthesizes the conversion result of the corresponding speech-to-speech task based on the audio features.
[0035] In this embodiment, by jointly using voice data and text data, the model is jointly trained and multi-task learned through a unified training sample set and a unified model architecture, which eliminates the inconsistency between different task training data sources, avoids cross-domain problems, and avoids the error transmission caused by the cascade operation of first completing a task to obtain a result and then using another model to complete the next task. This enables the model to learn the characteristics of different tasks at the same time, improves the accuracy and generalization ability of model training, and thus improves the processing accuracy of the trained model for multi-task speech-to-text conversion.
[0036] like Figure 4 As shown, it is a flow chart of a model training method provided in the third embodiment of the present invention. In step S203 of the second embodiment above, the parameters of the preset model are optimized according to the first result, the second result and the textual task label of the corresponding data to obtain a pre-trained preset model, which may include the following steps: Step S401: by presetting a first loss function, a loss calculation is performed on the first result and the textual task label of the corresponding source text data to obtain a first loss.
[0037] Step S402: by presetting a second loss function, performing loss calculation on the second result and the textual task label of the corresponding source speech data, to obtain a second loss.
[0038] Step S403: Optimize the parameters of the text encoder, speech encoder and first decoder of the preset model according to the first loss and the second loss to obtain a pre-trained preset model.
[0039] In this embodiment, the preset first loss function may refer to a preset function for measuring the difference between the first result and the textual task label corresponding to the source text data, the preset second loss function may refer to a preset function for measuring the difference between the second result and the textual task label corresponding to the source voice data, the first loss may refer to the error value between the first result and the textual task label corresponding to the source text data, and the second loss may refer to the error value between the second result and the textual task label corresponding to the source voice data.
[0040] For example, after calculating the first loss by using the preset first loss function and calculating the second loss by using the preset second loss function, the parameters of the text encoder, speech encoder and first decoder of the preset model are optimized according to the sum or weighted sum of the first loss and the second loss to obtain a pre-trained preset model.
[0041] In this embodiment, by calculating the first loss and the second loss, the model can learn how to more accurately generate text output that matches the textual task label, thereby improving the accuracy of the pre-trained preset model.
[0042] like Figure 5 As shown, a flow chart of a model training method provided by the fourth embodiment of the present invention, in step S205 of the second embodiment above, the parameters of the pre-trained preset model are optimized according to the third result, the fourth result and the voiced task label of the corresponding data to obtain the trained preset model, which may include the following steps: Step S501: by presetting a third loss function, a loss is calculated for the third result and the voiced task label of the corresponding source text data to obtain a third loss.
[0043] Step S502: By presetting a fourth loss function, a loss is calculated for the fourth result and the voiced task label of the corresponding source voice data to obtain a fourth loss.
[0044] Step S503: According to the third loss and the fourth loss, parameters of the text encoder, speech encoder, first decoder and second decoder of the pre-trained preset model are optimized to obtain a trained preset model.
[0045] In this embodiment, the preset third loss function may refer to a preset function for measuring the difference between the third result and the speech task label corresponding to the source text data, the preset fourth loss function may refer to a preset function for measuring the difference between the fourth result and the speech task label corresponding to the source voice data, the third loss may refer to the error value between the third result and the speech task label corresponding to the source text data, and the fourth loss may refer to the error value between the fourth result and the speech task label corresponding to the source voice data.
[0046] For example, after the third loss is calculated by using the preset third loss function and the fourth loss is calculated by using the preset fourth loss function, the parameters of the text encoder, speech encoder, first decoder and second decoder of the trained preset model are optimized according to the sum or weighted sum of the third loss and the fourth loss to obtain the trained preset model.
[0047] In this embodiment, by calculating the third loss and the fourth loss, the model can learn how to more accurately generate speech output that matches the voiced task label, thereby improving the accuracy of the trained preset model.
[0048] like Figure 6 FIG. 1 is a flow chart of a method for converting speech to text provided in Embodiment 5 of the present invention, comprising the following steps: Step S601: Obtain a trained preset model obtained by a model training method.
[0049] Step S602: Acquire the data to be converted and the corresponding task type.
[0050] Step S603: Detect whether the data to be converted is text data or voice data, and detect whether the task type is a text conversion task or a voice conversion task.
[0051] Step S604: If it is detected that the data to be converted is text data or voice data, and the task type is detected to be a text task or a voice task, the data to be converted is input into a trained preset model, so that the trained preset model performs conversion according to the task type to obtain a conversion result.
[0052] In this embodiment, the data to be converted may refer to data for speech-to-text conversion, the task type may refer to the conversion task type corresponding to the data to be converted, and the conversion result may refer to the output result of the trained preset model for converting the data to be converted into a text task or a speech task.
[0053] Specifically, when the data to be converted is obtained, it is detected whether the data to be converted is of text type or voice type, and whether the task type corresponding to the data to be converted is a text-to-text task of text to text or voice to text, or a voice-to-voice task of text to voice to voice. If it is detected that the data to be converted is text data or voice data, and the task type is detected to be a text-to-text task or a voice-to-voice task, the data to be converted is input into a trained preset model, so that the trained preset model performs conversion according to the task type to obtain a conversion result.
[0054] In this embodiment, by using a preset model trained by a model training method, the data to be converted is subjected to speech-to-text conversion according to the task type of the data to be converted to obtain a conversion result, thereby improving the processing efficiency and accuracy of multi-task speech-to-text conversion.
[0055] like Figure 7 As shown, it is a flow chart of a speech-to-text conversion method provided by Embodiment 6 of the present invention. In step S604 of the above Embodiment 5, if it is detected that the data to be converted is text data or speech data, and the task type is detected to be a text-to-text task or a speech-to-text task, the data to be converted is input into a trained preset model, so that the trained preset model performs conversion according to the task type to obtain a conversion result, which may include the following steps: Step S701: If it is detected that the data to be converted is text data and the task type is a text conversion task, the data to be converted is input into the text encoder of the trained preset model, and the text-to-text result is output by decoding through the first decoder of the trained preset model.
[0056] Specifically, if the data to be converted is text data, and the corresponding task type is a text-to-text task, the data to be converted is input into the text encoder of the trained preset model, and the text encoder of the trained preset model is used to encode the data to be converted to obtain vectorized text features. The first decoder of the trained preset model decodes the text features and outputs the conversion result of the text-to-text task.
[0057] Step S702: If it is detected that the data to be converted is text data and the task type is a speech task, the data to be converted is input into the text encoder of the trained preset model, and the text-to-speech result is decoded and output by the first decoder and the second decoder of the trained preset model.
[0058] Specifically, if the data to be converted is text data, and the corresponding task type is a speech-to-text voice-based task, including converting the data to be converted into a speech result of the same language type and converting the data to be converted into a speech result of a different language type, the data to be converted is input into the text encoder of the trained preset model, the text encoder of the trained preset model is used to encode the data to be converted to obtain vectorized text features, the first decoder of the trained preset model decodes the text features to obtain corresponding sub-word sequences, the text-to-unit encoder of the second decoder of the trained preset model encodes the sub-word sequences to obtain speech units, the unit decoder of the second decoder of the trained preset model decodes the speech units to obtain audio features, the vocoder of the second decoder of the trained preset model performs speech synthesis on the audio features, and outputs the conversion result of the text-to-speech voice-based task.
[0059] In this embodiment, by using a preset model trained by a model training method, text conversion and speech conversion are performed on the text data to be converted to obtain corresponding conversion results, thereby improving the processing efficiency and accuracy of the text data text conversion task and speech conversion task.
[0060] like Figure 8 As shown, it is a flow chart of a speech-to-text conversion method provided in Embodiment 7 of the present invention. In step S604 of the above Embodiment 5, if it is detected that the data to be converted is text data or speech data, and the task type is detected to be a text task or a speech task, the data to be converted is input into a trained preset model, so that the trained preset model performs conversion according to the task type to obtain a conversion result, which may include the following steps: Step S801: If it is detected that the data to be converted is speech data and the task type is a text conversion task, the data to be converted is input into the speech encoder of the trained preset model, and the speech-to-text result is output by decoding through the first decoder of the trained preset model.
[0061] Specifically, if the data to be converted is speech data, and the corresponding task type is a text-to-text task, the data to be converted is input into the speech encoder of the trained preset model, the speech encoder of the trained preset model encodes the data to be converted to obtain vectorized speech features, the first decoder of the trained preset model decodes the speech features, and outputs the conversion result of the speech-to-text task.
[0062] Step S802: If it is detected that the data to be converted is speech data and the task type is a speech task, the data to be converted is input into the speech encoder of the trained preset model, and the speech-to-speech result is output by decoding the first decoder and the second decoder of the trained preset model.
[0063] Specifically, if the data to be converted is speech data, and the corresponding task type is a speech-to-speech task, including converting the data to be converted into a speech result of the same language type and converting the data to be converted into a speech result of a different language type, the data to be converted is input into the speech encoder of the trained preset model, the speech encoder of the trained preset model encodes the data to be converted to obtain vectorized speech features, the first decoder of the trained preset model decodes the speech features to obtain the corresponding sub-word sequence, the text-to-unit encoder of the second decoder of the trained preset model encodes the sub-word sequence to obtain unit features, the unit decoder of the second decoder of the trained preset model decodes the unit features to obtain audio features, the vocoder of the second decoder of the trained preset model performs speech synthesis on the audio features, and outputs the conversion result of the speech-to-speech task.
[0064] In this embodiment, by using a preset model trained by a model training method, the speech data to be converted is converted into text and speech to obtain corresponding conversion results, thereby improving the processing efficiency and accuracy of the text and speech tasks of the speech data.
[0065] like Fig. 9 As shown, a model training device is provided in Embodiment 8 of the present invention, and the model training device corresponds to the model training method in the above embodiment. The model training device includes a sample acquisition module 91, a first training module 92, a first optimization module 93, a second training module 94, and a second optimization module 95. Each functional module is described in detail as follows: A sample acquisition module 91 is used to acquire a training sample set, wherein the training sample set includes source text data and source voice data, as well as a textual task label and a voiced task label of each data; A first training module 92 is used to input the source text data into a text encoder of a preset model, decode the source text data through a first decoder of the preset model to output a first result, and input the source speech data into a speech encoder of the preset model, decode the source speech data through the first decoder of the preset model to output a second result; A first optimization module 93, configured to optimize parameters of the preset model according to the first result, the second result and the textual task label of the corresponding data to obtain a pre-trained preset model; The second training module 94 is used to input the source text data into the text encoder of the pre-trained preset model, decode the source text data through the first decoder and the second decoder of the pre-trained preset model to output a third result, and input the source speech data into the speech encoder of the pre-trained preset model, decode the source speech data through the first decoder and the second decoder of the pre-trained preset model to output a fourth result; The second optimization module 95 is used to optimize the parameters of the pre-trained preset model according to the third result, the fourth result and the voice-based task label of the corresponding data to obtain a trained preset model.
[0066] Optionally, the first optimization module 93 includes: A first loss calculation unit, configured to calculate the loss of the first result and the textual task label of the corresponding source text data by using a preset first loss function to obtain a first loss; A second loss calculation unit, configured to perform loss calculation on the second result and the textual task label of the corresponding source speech data by using a preset second loss function to obtain a second loss; The third optimization unit is used to optimize the parameters of the text encoder, the speech encoder and the first decoder of the preset model according to the first loss and the second loss to obtain the pre-trained preset model.
[0067] Optionally, the second optimization module 95 includes: A third loss calculation unit, configured to perform loss calculation on the third result and the voiced task label of the corresponding source text data by using a preset third loss function to obtain a third loss; a fourth loss calculation unit, configured to perform loss calculation on the fourth result and the voiced task label of the corresponding source voice data by using a preset fourth loss function to obtain a fourth loss; The fourth optimization unit is used to optimize the parameters of the text encoder, speech encoder, first decoder and second decoder of the pre-trained preset model according to the third loss and the fourth loss to obtain the trained preset model.
[0068] For the specific definition of the model training device, please refer to the definition of the model training method above, which will not be repeated here. Each module in the above-mentioned model training device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0069] like Fig.10As shown, a speech-to-text conversion device provided in Embodiment 9 of the present invention corresponds to the speech-to-text conversion method in the above embodiment. The speech-to-text conversion device includes a model acquisition module 1001, a data acquisition module 1002, a detection module 1003 and a conversion module 1004. Each functional module is described in detail as follows: A model acquisition module 1001 is used to acquire a trained preset model obtained by the model training method; The data acquisition module 1002 is used to acquire the data to be converted and the corresponding task type; A detection module 1003 is used to detect whether the data to be converted is text data or voice data, and to detect whether the task type is a text-based task or a voice-based task; The conversion module 1004 is used to input the data to be converted into the trained preset model if it is detected that the data to be converted is text data or voice data, and if it is detected that the task type is a textual task or a voice-based task, so that the trained preset model performs conversion according to the task type to obtain a conversion result.
[0070] Optionally, the conversion module 1004 includes: A first task conversion unit is used for, if it is detected that the data to be converted is text data and the task type is a text conversion task, inputting the data to be converted into a text encoder of the trained preset model, and decoding and outputting a text-to-text result through a first decoder of the trained preset model; The second task conversion unit is used to input the data to be converted into the text encoder of the trained preset model if it is detected that the data to be converted is text data and the task type is a speech task, and decode and output the text-to-speech result through the first decoder and the second decoder of the trained preset model.
[0071] Optionally, the conversion module 1004 includes: A third task conversion unit is configured to, if it is detected that the data to be converted is speech data and the task type is a text conversion task, input the data to be converted into the speech encoder of the trained preset model, and decode and output a speech-to-text result through the first decoder of the trained preset model; The fourth task conversion unit is used to input the data to be converted into the speech encoder of the trained preset model if it is detected that the data to be converted is speech data and the task type is a speech task, and decode and output the speech-to-speech result through the first decoder and the second decoder of the trained preset model.
[0072] The specific definition of the speech-to-text conversion device can be found in the definition of the vortex speech-to-text conversion method above, which will not be repeated here. Each module in the above-mentioned speech-to-text conversion device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0073] Fig.11 This is a schematic diagram of the structure of a computer device provided by Embodiment 10 of the present invention. Fig.11 As shown, the computer device of this embodiment includes: at least one processor ( Fig.11 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned model training method embodiments or the steps in any of the above-mentioned speech-to-text conversion method embodiments are implemented.
[0074] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Fig.11 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.
[0075] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0076] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.
[0077] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0078] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.
[0079] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0080] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0081] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0083] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein by equivalents, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention. The non-company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.
Claims
1. A model training method, characterized in that: include: Acquire a training sample set, wherein the training sample set includes source text data and source voice data, as well as a textual task label and a voiced task label of each data; Input the source text data into a text encoder of a preset model, decode it through a first decoder of the preset model to output a first result, input the source voice data into a voice encoder of the preset model, decode it through the first decoder of the preset model to output a second result; According to the first result, the second result and the textual task label of the corresponding data, the parameters of the preset model are optimized to obtain a pre-trained preset model; Input the source text data into the text encoder of the pre-trained preset model, decode the data through the first decoder and the second decoder of the pre-trained preset model to output a third result, input the source voice data into the voice encoder of the pre-trained preset model, decode the data through the first decoder and the second decoder of the pre-trained preset model to output a fourth result; According to the third result, the fourth result and the voice-based task label of the corresponding data, the parameters of the pre-trained preset model are optimized to obtain a trained preset model.
2. The model training method according to claim 1, characterized in that: The step of optimizing parameters of the preset model according to the first result, the second result, and the textual task label of the corresponding data to obtain a pre-trained preset model includes: By presetting a first loss function, performing loss calculation on the first result and the textual task label of the corresponding source text data, a first loss is obtained; By presetting a second loss function, performing loss calculation on the second result and the textual task label of the corresponding source speech data, a second loss is obtained; According to the first loss and the second loss, parameters of the text encoder, the speech encoder and the first decoder of the preset model are optimized to obtain the pre-trained preset model.
3. The model training method according to claim 1, characterized in that: The step of optimizing parameters of the pre-trained preset model according to the third result, the fourth result and the voice-based task label of the corresponding data to obtain the trained preset model includes: By presetting a third loss function, performing loss calculation on the third result and the voiced task label of the corresponding source text data, to obtain a third loss; By presetting a fourth loss function, performing loss calculation on the fourth result and the voiced task label of the corresponding source voice data, a fourth loss is obtained; According to the third loss and the fourth loss, parameters of the text encoder, speech encoder, first decoder and second decoder of the pre-trained preset model are optimized to obtain the trained preset model.
4. A speech-to-text conversion method, characterized in that: include: Obtain a trained preset model obtained by the model training method according to any one of claims 1 to 3; Get the data to be converted and the corresponding task type; Detecting whether the data to be converted is text data or voice data, and detecting whether the task type is a text-based task or a voice-based task; If it is detected that the data to be converted is text data or voice data, and it is detected that the task type is a textual task or a voice task, the data to be converted is input into the trained preset model, so that the trained preset model performs conversion according to the task type to obtain a conversion result.
5. The speech-to-text conversion method according to claim 4, characterized in that: If it is detected that the data to be converted is text data or voice data, and the task type is detected to be a text-based task or a voice-based task, the data to be converted is input into the trained preset model, so that the trained preset model performs conversion according to the task type to obtain a conversion result, including: If it is detected that the data to be converted is text data and the task type is a text conversion task, the data to be converted is input into the text encoder of the trained preset model, and the text-to-text result is output by decoding the first decoder of the trained preset model; If it is detected that the data to be converted is text data and the task type is a speech task, the data to be converted is input into the text encoder of the trained preset model, and the text-to-speech result is decoded and output by the first decoder and the second decoder of the trained preset model.
6. The speech-to-text conversion method according to claim 4, wherein: If it is detected that the data to be converted is text data or voice data, and the task type is detected to be a text-based task or a voice-based task, the data to be converted is input into the trained preset model, so that the trained preset model performs conversion according to the task type to obtain a conversion result, including: If it is detected that the data to be converted is speech data and the task type is a text conversion task, the data to be converted is input into the speech encoder of the trained preset model, and the speech-to-text conversion result is output by decoding the first decoder of the trained preset model; If it is detected that the data to be converted is speech data and the task type is a speech task, the data to be converted is input into the speech encoder of the trained preset model, and the speech-to-speech result is decoded and output by the first decoder and the second decoder of the trained preset model.
7. A model training device, characterized in that: include: A sample acquisition module, used to acquire a training sample set, wherein the training sample set includes source text data and source voice data, as well as a textual task label and a voiced task label of each data; A first training module is used to input the source text data into a text encoder of a preset model, decode the source text data through a first decoder of the preset model to output a first result, and input the source speech data into a speech encoder of the preset model, decode the source speech data through the first decoder of the preset model to output a second result; A first optimization module, configured to optimize parameters of the preset model according to the first result, the second result, and the textual task label of the corresponding data, to obtain a pre-trained preset model; A second training module is used to input the source text data into the text encoder of the pre-trained preset model, decode and output a third result through the first decoder and the second decoder of the pre-trained preset model, input the source voice data into the voice encoder of the pre-trained preset model, and decode and output a fourth result through the first decoder and the second decoder of the pre-trained preset model; The second optimization module is used to optimize the parameters of the pre-trained preset model according to the third result, the fourth result and the voice-based task label of the corresponding data to obtain a trained preset model.
8. A speech-to-text conversion device, characterized in that: include: A model acquisition module, used to acquire a trained preset model obtained by the model training method according to any one of claims 1 to 3; A data acquisition module is used to obtain the data to be converted and the corresponding task type; A detection module, used to detect whether the data to be converted is text data or voice data, and to detect whether the task type is a text-based task or a voice-based task; A conversion module is used to input the data to be converted into the trained preset model if it is detected that the data to be converted is text data or voice data, and if it is detected that the task type is a textual task or a voice-based task, so that the trained preset model performs conversion according to the task type to obtain a conversion result.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the model training method as described in any one of claims 1 to 3, or the speech-to-text conversion method as described in any one of claims 4 to 6.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 3, or the speech-to-text conversion method as described in any one of claims 4 to 6.