Method, device and computer program product for voice data processing

Abstract

The present disclosure provides methods, devices, electronic devices, storage media and computer program products for speech data processing, specifically relating to speech processing and deep learning technology, which can be used in speech recognition and translation scenarios. The specific implementation scheme is: obtaining a training sample set, the training samples in the training sample set include corresponding speech data, phoneme data and text data; for the training samples in the training sample set, the speech data and text data therein are used as the input and expected output of the initial speech-to-text network in turn, and the phoneme data and text data therein are used as the input and expected output of the initial text-to-text network in turn, and the initial speech-to-text network and the initial text-to-text network with partially the same network structure are jointly trained, and the speech-to-text network obtained by training based on the initial speech-to-text network is determined as the speech data processing model. The present disclosure improves the accuracy of the speech-to-text network in processing speech data.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, specifically to the field of speech processing and deep learning technology, and in particular to methods, devices, electronic devices, storage media, and computer program products for speech data processing, which can be used in speech recognition and speech translation scenarios. Background Art

[0002] At present, with the development of artificial intelligence technology, voice data processing technologies such as speech recognition and speech translation have made great progress. However, the accuracy of current voice data processing results still needs to be improved. Summary of the invention

[0003] The present disclosure provides a method, an apparatus, an electronic device, a storage medium and a computer program product for voice data processing.

[0004] According to a first aspect, a method for speech data processing is provided, comprising: obtaining a training sample set, wherein the training samples in the training sample set include corresponding speech data, phoneme data and text data; for the training samples in the training sample set, using the speech data and text data in the training samples as input and expected output of an initial speech-to-text network in turn, using the phoneme data and text data in the training samples as input and expected output of an initial text-to-text network in turn, jointly training the initial speech-to-text network and the initial text-to-text network, wherein the initial speech-to-text network and the initial text-to-text network include partially identical network structures; and determining a speech-to-text network obtained by training the initial speech-to-text network as a speech data processing model.

[0005] According to the second aspect, a speech data processing method is provided, comprising: obtaining speech to be processed; inputting the speech to be processed into a speech data processing model to obtain a speech data processing result, wherein the speech data processing model is trained by any implementation method of the first aspect.

[0006] According to a third aspect, a device for speech data processing is provided, comprising: a first acquisition unit, configured to acquire a training sample set, wherein the training samples in the training sample set include corresponding speech data, phoneme data and text data; a training unit, configured to, for the training samples in the training sample set, use the speech data and text data in the training samples as input and expected output of an initial speech-to-text network in turn, use the phoneme data and text data in the training samples as input and expected output of an initial text-to-text network in turn, and jointly train the initial speech-to-text network and the initial text-to-text network, wherein the initial speech-to-text network and the initial text-to-text network include partially identical network structures; a determination unit, configured to determine the speech-to-text network obtained by training the initial speech-to-text network as a speech data processing model.

[0007] According to the fourth aspect, a speech data processing device is provided, comprising: a second acquisition unit, configured to acquire speech to be processed; a processing unit, configured to input the speech to be processed into a speech data processing model to obtain a speech data processing result, wherein the speech data processing model is trained by any implementation method of the third aspect.

[0008] According to the fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation of the first aspect or the second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect or the second aspect.

[0010] According to a seventh aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation of the first aspect or the second aspect when executed by a processor.

[0011] According to the technology disclosed in the present invention, the joint training includes an initial speech-to-text network and an initial text-to-text network with partially identical network structures. By encoding and transmitting text information of the initial text-to-text network during the training process, the speech data processing task corresponding to the initial speech-to-text network is assisted, thereby improving the accuracy of the speech-to-text network in processing speech data.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0015] Figure 2 is a flow chart of an embodiment of a method for speech data processing according to the present disclosure;

[0016] Figure 3 A schematic diagram of a network framework applicable to the method for voice data processing according to the present disclosure;

[0017] Figure 4 is a schematic diagram of an application scenario of the method for voice data processing according to this embodiment;

[0018] Figure 5 is a flow chart of another embodiment of a method for voice data processing according to the present disclosure;

[0019] Figure 6 is a flow chart of another embodiment of a method for voice data processing according to the present disclosure;

[0020] Figure 7 is a flow chart of an embodiment of a method for processing speech data according to the present disclosure;

[0021] Figure 8 is a structural diagram of an embodiment of a device for voice data processing according to the present disclosure;

[0022] Fig. 9 is a structural diagram of an embodiment of a speech data processing device according to the present disclosure;

[0023] Fig.10 It is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0026] Figure 1 An exemplary architecture 100 to which the method and apparatus for speech data processing and the speech data processing method and apparatus of the present disclosure can be applied is shown.

[0027] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The communication connection between the terminal devices 101, 102, 103 constitutes a topological network, and the network 104 is used to provide a medium for the communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0028] Terminal devices 101, 102, 103 may be hardware devices or software that support network connection for data interaction and data processing. When terminal devices 101, 102, 103 are hardware, they may be various electronic devices that support network connection, information acquisition, interaction, display, processing and other functions, including but not limited to voice acquisition equipment, vehicle-mounted intelligent devices, monitoring equipment, smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. It may be implemented as multiple software or software modules for providing distributed services, for example, or it may be implemented as a single software or software module. No specific limitation is made here.

[0029] The server 105 may be a server that provides various services, for example, a server that receives training sample sets sent by the terminal devices 101, 102, and 103, and jointly trains an initial speech-to-text network and an initial text-to-text network that have partially the same network structure to obtain a background processing server for a high-accuracy speech-to-text network. Optionally, the server may use the obtained speech-to-text network to process voice data. As an example, the server 105 may be a cloud server.

[0030] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0031] It should also be noted that the method for voice data processing and the voice data processing method provided by the embodiments of the present disclosure can be executed by a server, or by a terminal device, or by a server and a terminal device in cooperation with each other. Accordingly, the various parts (such as various units) included in the device for voice data processing and the voice data processing device can be all set in the server, or all set in the terminal device, or can be set in the server and the terminal device respectively.

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system is only illustrative. Any number of terminal devices, networks and servers may be provided as required. When the method for voice data processing and the electronic device on which the voice data processing method is running do not need to transmit data with other electronic devices, the system architecture may only include the method for voice data processing and the electronic device (e.g., server or terminal device) on which the voice data processing method is running.

[0033] Please refer to Figure 2 , Figure 2 A flowchart of a method for voice data processing provided by an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0034] Step 201: Obtain a training sample set.

[0035] In this embodiment, the execution subject of the method for voice data processing (for example, Figure 1 The terminal device or server in the network can obtain the training sample set remotely or locally through a wired network connection or a wireless network connection.

[0036] The training sample set includes multiple training samples, and each training sample includes corresponding speech data, phoneme data and text data. The speech data, phoneme data and text data in a training sample correspond to each other, and the information indicated by the three types of data is information with the same semantics. As an example, the speech data is speech data belonging to any language (such as Chinese, English), the phoneme data is a phoneme text sequence obtained by dividing the speech represented by the speech data into the smallest speech unit, and the text data is a text representation form of the speech data, which can be text data in the same language as the speech data, or text data in a different language from the speech data. A training sample can represent different data amounts such as single words, words, short sentences or long sentences, which are not limited here.

[0037] In some optional implementations of this embodiment, for a training sample in a training sample set, the execution subject may perform noise replacement on a target phoneme in initial phoneme data representing speech data in the training sample to obtain phoneme data.

[0038] As an example, the execution subject may select a preset number of target phonemes from the phoneme data in each training sample, and replace the target phonemes with noise, thereby obtaining the phoneme data after noise replacement. The preset number may be specifically set according to actual conditions (e.g., the number of phonemes in the phoneme data), and is not limited here.

[0039] In this implementation, by introducing noise into the phoneme data in the training samples, the noise resistance of the model trained based on the phoneme data with the introduced noise is improved.

[0040] Step 202, for the training samples in the training sample set, the speech data and text data in the training samples are used as the input and expected output of the initial speech-to-text network in turn, and the phoneme data and text data in the training samples are used as the input and expected output of the initial text-to-text network in turn, and the initial speech-to-text network and the initial text-to-text network are jointly trained.

[0041] In this embodiment, for the training samples in the training sample set, the execution subject can use the speech data and text data in the training samples as the input and expected output of the initial speech-to-text network, and use the phoneme data and text data in the training samples as the input and expected output of the initial text-to-text network, and jointly train the initial speech-to-text network and the initial text-to-text network. The initial speech-to-text network and the initial text-to-text network include part of the same network structure.

[0042] The initial speech-to-text network is used to convert speech data into text data. Specifically, the initial speech-to-text network is a speech recognition network that converts speech data into text data in the same language as the speech data, or a speech translation network that converts speech data into text data in a language different from the speech data. For example, the initial speech-to-text network is used to convert Chinese speech into text in English, which is a language different from Chinese.

[0043] The speech-to-text network may adopt an existing network model, or a hybrid model based on a combination of existing network models, including but not limited to an N-gram language model and an RNN (Recurrent Neural Network) language model.

[0044] The initial text-to-text network is used to convert one text data into another text data. For example, the text-to-text network can be used for conversion between different forms of text in the same language (for example, conversion between phoneme data and text data in the same language), and can also be used for conversion between texts in different languages.

[0045] Specifically, for the training samples in the training sample set, the speech data in the training samples is used as the input of the initial speech-to-text network to obtain the actual output of the initial speech-to-text network, and the first loss between the actual output of the initial speech-to-text network and the text data (expected output) corresponding to the input speech data is calculated; the phoneme data in the training samples is used as the input of the initial text-to-text network to obtain the actual output of the initial text-to-text network, and the second loss between the actual output of the initial text-to-text network and the text data (expected output) corresponding to the input speech data is calculated; the total loss is calculated based on the first loss and the second loss; the gradient of the training initial speech-to-text network and the initial text-to-text network is calculated based on the total loss, and the parameters of the initial speech-to-text network and the initial text-to-text network are adjusted according to the obtained gradient using methods such as the gradient descent method. The above-mentioned execution subject can cyclically execute the above-mentioned training steps until the preset end condition is reached, and the training of the initial speech-to-text network and the initial text-to-text network is completed.

[0046] The preset termination condition may be, for example, that the number of training times exceeds a number threshold, the training time exceeds a time threshold, and the total loss tends to converge.

[0047] In some optional implementations of this embodiment, the initial speech-to-text network includes a speech encoding module, and the initial text-to-text network includes a text encoding module, wherein the speech encoding module and the text encoding module include partially identical network structures.

[0048] In some optional implementations of this embodiment, the initial speech-to-text network and the initial text-to-text network include the same attention module; and / or, the initial speech-to-text network and the initial text-to-text network include the same text decoding module.

[0049] like Figure 3 As shown, the initial speech-to-text network includes a speech encoding module 301, an attention module 302 and a text decoding module 303, and the initial text-to-text network includes a text encoding module 304, an attention module 302 and a text decoding module 303. Among them, the speech encoding module 301 and the text encoding module 304 include a partially identical network structure 305. The speech encoding module is used to encode data in speech form, the text encoding module is used to encode data in text form, the attention module (such as a self-attention module) is used to adjust the attention to information during the model learning process, and help the speech data processing model learn feature information useful for speech data processing, and the text decoding module is used to decode the information learned by the model into text data.

[0050] The text encoding module is mainly composed of multiple Transformer layers, with the input being phoneme data, and the subsequent attention module and text decoding module as a text-to-text network; the speech encoding module is mainly composed of multiple Transformer layers, and the subsequent attention module and text decoding module as a speech-to-text network. The speech encoding module and the text encoding module include some of the same Transformer layers.

[0051] In this embodiment, a network framework is provided that can jointly train an initial speech-to-text network and an initial text-to-text network that include partially identical network structures.

[0052] Based on the above network framework, the above execution entity can calculate the total loss through the following formula:

[0053]

[0054] Among them, L represents the total loss, θ dec represents the network parameters of the speech encoding module, the text encoding module, and the text decoding module respectively, X represents the input audio data, Y represents the input phoneme data, W is the predicted text data, and P is the corresponding probability.

[0055] In the prior art, the speech recognition system and the speech translation system are separated and do not use each other's effective information. In order to allow the intermediate effective information to be transferred and used in the speech recognition system or the speech translation system and reduce the dependence of the model training process on a large amount of annotated data, in some optional implementations of this embodiment, the text data includes first text data in the same language as the speech data and second text data in a different language from the speech data, and the initial speech-to-text network includes a first initial speech recognition network and a first initial speech translation network obtained based on the first initial speech recognition network.

[0056] The execution subject may execute step 202 in the following manner:

[0057] First, for the training samples in the training sample set, the speech data and the first text data in the training samples are used as the input and expected output of the first initial speech recognition network, respectively, and the phoneme data and the first text data in the training samples are used as the input and expected output of the initial text-to-text network, respectively, and the first initial speech recognition network and the initial text-to-text network are jointly trained to obtain the trained speech recognition network and text-to-text network.

[0058] As an example, the speech data in the training sample is Chinese speech, the first text data is Chinese text, and the second text data is English text.

[0059] The above-mentioned execution entity can use the speech data and the first text data in the training sample as the input and expected output of the first initial speech recognition network in turn, and calculate the first loss between the actual output and the expected output of the first initial speech recognition network; use the phoneme data and the first text data in the training sample as the input and expected output of the initial text-to-text network in turn, and calculate the second loss between the actual output and the expected output of the initial text-to-text network, and then calculate the total loss based on the first loss and the second loss, and update the first initial speech recognition network and the initial text-to-text network based on the total loss, and iteratively execute the above-mentioned training process until the trained speech recognition network and text-to-text network are obtained.

[0060] After inputting the speech data, the initial speech recognition network can extract features from the audio frames obtained by framing the speech data, obtain feature information such as fundamental frequency (pitch), Mel-frequency cepstrum (MFCC), and perform differential enhancement on the feature information.

[0061] Second, the output units of the trained speech recognition network are adjusted to output units for speech translation, thereby obtaining a first initial speech translation network.

[0062] The output unit for speech translation may be, for example, an output layer of a speech translation model for converting speech data into text data in a language different from that of the speech data.

[0063] Third, for the training samples in the training sample set, the speech data and the second text data in the training samples are used as the input and expected output of the first initial speech translation network in turn, and the phoneme data and the second text data in the training samples are used as the input and expected output of the text-to-text network in turn, and the first initial speech translation network and the text-to-text network are jointly trained.

[0064] Specifically, the execution subject may train the first initial speech translation network and the text-to-text network by adopting a training process similar to the training process of the first initial speech recognition network and the initial text-to-text network in the first step.

[0065] In this implementation, the final target task model is a speech translation network. Based on a reusable network framework, the speech recognition model is first trained, and the output unit is adjusted on the basis of the speech recognition model to obtain the first initial speech translation model, and further training on the translation task is performed, so that the effective information learned by the speech recognition network can be transferred to the speech translation system for use, reducing the dependence of the model training process on a large amount of labeled data and improving the translation accuracy of the final speech translation network.

[0066] In some optional implementations of this embodiment, the initial speech-to-text network includes a second initial speech translation network and a second initial speech recognition network obtained based on the second initial speech translation network. The execution subject may perform step 202 in the following manner:

[0067] First, for the training samples in the training sample set, the speech data and the second text data in the training samples are used as the input and expected output of the second initial speech translation network in turn, and the phoneme data and the second text data in the training samples are used as the input and expected output of the initial text-to-text network in turn, and the second initial speech translation network and the initial text-to-text network are jointly trained to obtain the trained speech translation network and text-to-text network.

[0068] Specifically, for the training samples in the training sample set, the speech data and the second text data in the training samples are used as the input and expected output of the second initial speech translation network in turn, and the first loss between the actual output and the expected output of the second initial speech translation network is calculated; the phoneme data and the second text data in the training samples are used as the input and expected output of the initial text-to-text network in turn, and the second loss between the actual output and the expected output of the initial text-to-text network is calculated, and then the total loss is calculated according to the first loss and the second loss, and the second initial speech translation network and the initial text-to-text network are updated according to the total loss, and the above training process is iteratively performed until the trained speech translation network and text-to-text network are obtained.

[0069] Second, the output units of the trained speech translation network are adjusted to output units for speech recognition, thereby obtaining a second initial speech recognition network.

[0070] The output unit for speech recognition may be, for example, an output layer of a speech recognition model for converting speech data into text data in the same language as the speech data.

[0071] Third, for the training samples in the training sample set, the speech data and the first text data in the training samples are used as the input and expected output of the second initial speech recognition network in turn, and the phoneme data and the first text data in the training samples are used as the input and expected output of the text-to-text network in turn, and the second initial speech recognition network and the text-to-text network are jointly trained.

[0072] Specifically, the execution subject may train the second initial speech recognition network and the text-to-text network by adopting a training process similar to the training process of the second initial speech translation network and the initial text-to-text network in the first step.

[0073] In this implementation, the final target task model is a speech recognition network. Based on the reused network framework, the speech translation model is first trained, and the output unit is adjusted on the basis of the speech translation model to obtain the second initial speech recognition model, and further training on the recognition task is performed, so that the effective information learned by the speech translation network can be transferred to the speech recognition system for use, reducing the dependence of the model training process on a large amount of labeled data and improving the recognition accuracy of the final speech recognition network.

[0074] Step 203: determine the speech-to-text network obtained based on the initial speech-to-text network training as the speech data processing model.

[0075] In this embodiment, the above-mentioned execution subject can determine the speech-to-text network obtained based on the initial speech-to-text network training as the speech data processing model, and the speech data processing model can be used for speech processing tasks.

[0076] In some optional implementations of this embodiment, when the final target task model obtained through training in step 202 is a speech translation network, the execution subject may determine the speech translation network obtained through training based on the first initial speech translation network as the speech data processing model. In this way, a speech data processing model with high translation accuracy is obtained.

[0077] In some optional implementations of this embodiment, when the final target task model obtained through training in step 202 is a speech recognition network, the execution subject may determine the speech recognition network obtained through training based on the second initial speech recognition network as the speech data processing model. In this way, a speech data processing model with high recognition accuracy is obtained.

[0078] Continue to see Figure 4 , Figure 4 FIG. 4 is a schematic diagram 400 of an application scenario of the method for voice data processing according to this embodiment. Figure 4In the application scenario, the server 401 first obtains a training sample set 403 from a database 402. The training samples in the training sample set 403 include corresponding speech data 4031, phoneme data 4032, and text data 4033. For the training samples in the training sample set 403, the speech data 4031 and text data 4033 in the training samples are used as the input and expected output of the initial speech-to-text network 404 in turn, and the phoneme data 4032 and text data 4033 in the training samples are used as the input and expected output of the initial text-to-text network 405 in turn, and the initial speech-to-text network 404 and the initial text-to-text network 405 are jointly trained. The initial speech-to-text network 404 and the initial text-to-text network 405 include partially identical network structures. Finally, the speech-to-text network 406 trained based on the initial speech-to-text network 404 is determined as the speech data processing model.

[0079] In this embodiment, the joint training includes an initial speech-to-text network and an initial text-to-text network with partially identical network structures. By encoding and transmitting text information of the initial text-to-text network during the training process, the speech data processing task corresponding to the initial speech-to-text network is assisted, thereby improving the accuracy of speech data processing by the speech-to-text network.

[0080] Continue to refer Figure 5 , shows a schematic process 500 of a method embodiment of the method of the present disclosure applied to speech data processing, comprising the following steps:

[0081] Step 501: Obtain a training sample set.

[0082] The training samples in the training sample set include corresponding speech data, phoneme data and text data, and the text data includes first text data in the same language as the speech data and second text data in a different language from the speech data. For the training samples in the training sample set, the target phonemes in the initial phoneme data representing the speech data in the training sample are replaced with noise to obtain the phoneme data. The initial speech-to-text network includes a first initial speech recognition network and a first initial speech translation network obtained based on the first initial speech recognition network.

[0083] Step 502: for the training samples in the training sample set, the speech data and the first text data in the training samples are used as the input and expected output of the first initial speech recognition network, respectively; the phoneme data and the first text data in the training samples are used as the input and expected output of the initial text-to-text network, respectively; the first initial speech recognition network and the initial text-to-text network are jointly trained to obtain the trained speech recognition network and text-to-text network.

[0084] The initial speech-to-text network includes a speech encoding module, an attention module and a text decoding module, and the initial text-to-text network includes a text encoding module, an attention module and a text decoding module, wherein the speech encoding module and the text encoding module include partially identical network structures.

[0085] Step 503: Adjust the output unit of the trained speech recognition network to an output unit for speech translation, and obtain a first initial speech translation network.

[0086] Step 504: for the training samples in the training sample set, the speech data and the second text data in the training samples are used as the input and expected output of the first initial speech translation network in sequence, and the phoneme data and the second text data in the training samples are used as the input and expected output of the text-to-text network in sequence, and the first initial speech translation network and the text-to-text network are jointly trained.

[0087] Step 505: determine the speech translation network obtained based on the first initial speech translation network training as the speech data processing model.

[0088] It can be seen from this embodiment that Figure 2 Compared with the corresponding embodiments, the process 500 of the method applied to speech data processing in this embodiment specifically illustrates a network framework suitable for jointly training an initial speech-to-text network and an initial text-to-text network, a process of introducing noise into training samples, a process of assisting the speech-to-text network in performing speech data processing tasks corresponding to the speech-to-text network through encoding and transmitting text information of the initial text-to-text network during the training process, and a process of migrating and using effective information learned by the speech recognition network in a speech translation system, thereby improving the model's anti-noise capability, reducing the model training process's dependence on a large amount of annotated data, and improving the translation accuracy of the final speech translation network.

[0089] Continue to refer Figure 6 , shows a schematic process 600 of a method embodiment of the method of the present disclosure applied to speech data processing, comprising the following steps:

[0090] Step 601, obtaining a training sample set.

[0091] The training samples in the training sample set include corresponding speech data, phoneme data and text data, and the text data includes first text data in the same language as the speech data and second text data in a different language from the speech data. For the training samples in the training sample set, the target phonemes in the initial phoneme data representing the speech data in the training sample are replaced with noise to obtain the phoneme data. The initial speech-to-text network includes a second initial speech translation network and a second initial speech recognition network obtained based on the second initial speech translation network.

[0092] Step 602: for the training samples in the training sample set, the speech data and the second text data in the training samples are used as the input and expected output of the second initial speech translation network in sequence, and the phoneme data and the second text data in the training samples are used as the input and expected output of the initial text-to-text network in sequence, and the second initial speech translation network and the initial text-to-text network are jointly trained to obtain the trained speech translation network and text-to-text network.

[0093] The initial speech-to-text network includes a speech encoding module, an attention module and a text decoding module, and the initial text-to-text network includes a text encoding module, an attention module and a text decoding module, wherein the speech encoding module and the text encoding module include partially identical network structures.

[0094] Step 603: Adjust the output unit of the trained speech translation network to an output unit for speech recognition, thereby obtaining a second initial speech recognition network.

[0095] Step 604: For the training samples in the training sample set, the speech data and the first text data in the training samples are used as the input and expected output of the second initial speech recognition network, respectively; the phoneme data and the first text data in the training samples are used as the input and expected output of the text-to-text network, respectively; and the second initial speech recognition network and the text-to-text network are jointly trained.

[0096] Step 605: determine the speech recognition network obtained based on the training of the second initial speech recognition network as the speech data processing model.

[0097] It can be seen from this embodiment that Figure 2 Compared with the corresponding embodiments, the process 600 of the method applied to speech data processing in this embodiment specifically illustrates a framework suitable for jointly training an initial speech-to-text network and an initial text-to-text network, a process of introducing noise into training samples, a process of assisting the speech-to-text network in performing speech data processing tasks corresponding to the speech-to-text network through encoding and transmitting text information of the initial text-to-text network during the training process, and a process of migrating and using effective information learned by the speech translation network in a speech recognition system, thereby improving the model's anti-noise capability, reducing the model training process's dependence on a large amount of annotated data, and improving the translation accuracy of the resulting speech recognition network.

[0098] Continue to refer Figure 7 , Figure 7 A flowchart of a voice data processing method provided by an embodiment of the present disclosure, wherein process 700 includes the following steps:

[0099] Step 701, obtaining the speech to be processed.

[0100] In this embodiment, the execution subject of the voice data processing method (for example Figure 1 The terminal device or server in the system can obtain the speech to be processed remotely or locally through a wired network connection or a wireless network connection. The speech to be processed can be speech data in any language.

[0101] Step 702: input the speech to be processed into the speech data processing model to obtain the speech data processing result.

[0102] In this embodiment, the execution subject can input the speech to be processed into the speech data processing model to obtain the speech data processing result. The speech data processing model is obtained by training in the above embodiments 200, 500, and 600.

[0103] Furthermore, when the processing task of the speech to be processed is a speech translation task, the speech data processing model can be the speech translation model trained in Example 500; when the processing task of the speech to be processed is a speech recognition task, the speech data processing model can be the speech recognition model trained in Example 600.

[0104] Continue to refer Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for voice data processing, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0105] like Figure 8 As shown, the device applied to speech data processing includes: a first acquisition unit 801, configured to acquire a training sample set, wherein the training samples in the training sample set include corresponding speech data, phoneme data and text data; a training unit 802, configured to, for the training samples in the training sample set, use the speech data and text data in the training samples as the input and expected output of the initial speech-to-text network in turn, use the phoneme data and text data in the training samples as the input and expected output of the initial text-to-text network in turn, and jointly train the initial speech-to-text network and the initial text-to-text network, wherein the initial speech-to-text network and the initial text-to-text network include partially identical network structures; a determination unit 803, configured to determine the speech-to-text network obtained by training the initial speech-to-text network as a speech data processing model.

[0106] In some optional implementations of this embodiment, the text data includes first text data in the same language as the speech data and second text data in a different language from the speech data, and the initial speech-to-text network includes a first initial speech recognition network and a first initial speech translation network obtained based on the first initial speech recognition network; and the training unit 802 is further configured to: for training samples in the training sample set, use the speech data and the first text data in the training sample as the input and expected output of the first initial speech recognition network in sequence, use the phoneme data and the first text data in the training sample as the input and expected output of the initial text-to-text network in sequence, jointly train the first initial speech recognition network and the initial text-to-text network to obtain a trained speech recognition network and a text-to-text network; adjust the output unit of the trained speech recognition network to an output unit for speech translation to obtain a first initial speech translation network; for training samples in the training sample set, use the speech data and the second text data in the training sample as the input and expected output of the first initial speech translation network in sequence, use the phoneme data and the second text data in the training sample as the input and expected output of the text-to-text network in sequence, and jointly train the first initial speech translation network and the text-to-text network.

[0107] In some optional implementations of this embodiment, the determining unit 803 is further configured to: determine the speech translation network obtained by training the first initial speech translation network as the speech data processing model.

[0108] In some optional implementations of this embodiment, the text data includes first text data in the same language as the speech data and second text data in a different language from the speech data, and the initial speech-to-text network includes a second initial speech translation network and a second initial speech recognition network obtained based on the second initial speech translation network; and the training unit 802 is further configured to: for training samples in the training sample set, use the speech data and the second text data in the training sample as the input and expected output of the second initial speech translation network in sequence, use the phoneme data and the second text data in the training sample as the input and expected output of the initial text-to-text network in sequence, jointly train the second initial speech translation network and the initial text-to-text network to obtain a trained speech translation network and a text-to-text network; adjust the output unit of the trained speech translation network to an output unit for speech recognition to obtain a second initial speech recognition network; for training samples in the training sample set, use the speech data and the first text data in the training sample as the input and expected output of the second initial speech recognition network in sequence, use the phoneme data and the first text data in the training sample as the input and expected output of the text-to-text network in sequence, and jointly train the second initial speech recognition network and the text-to-text network.

[0109] In some optional implementations of this embodiment, the determination unit 803 is further configured to: determine the speech recognition network obtained based on the training of the second initial speech recognition network as the speech data processing model.

[0110] In some optional implementations of this embodiment, the above-mentioned device also includes: a replacement unit (not shown in the figure), which is configured to replace the target phonemes in the initial phoneme data representing the speech data in the training sample in the training sample set with noise to obtain phoneme data.

[0111] In some optional implementations of this embodiment, the initial speech-to-text network includes a speech encoding module, and the initial text-to-text network includes a text encoding module, wherein the speech encoding module and the text encoding module include partially identical network structures.

[0112] In some optional implementations of this embodiment, the initial speech-to-text network and the initial text-to-text network include the same attention module; and / or, the initial speech-to-text network and the initial text-to-text network include the same text decoding module.

[0113] In this embodiment, the joint training includes an initial speech-to-text network and an initial text-to-text network with partially identical network structures. By encoding and transmitting text information of the initial text-to-text network during the training process, the speech data processing task corresponding to the initial speech-to-text network is assisted, thereby improving the accuracy of speech data processing by the speech-to-text network.

[0114] Continue to refer Fig. 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for voice data processing, and the device embodiment is Figure 7 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0115] like Fig. 9 As shown, the speech data processing device includes: a second acquisition unit 901, configured to acquire speech to be processed; and a processing unit 902, configured to input the speech to be processed into a speech data processing model to obtain a speech data processing result. The speech data processing model is obtained by training in the embodiment 800.

[0116] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the method applied to voice data processing and the voice data processing method described in any of the above embodiments when executing.

[0117] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the method for voice data processing and the voice data processing method described in any of the above embodiments when executed.

[0118] The embodiments of the present disclosure provide a computer program product. When the computer program is executed by a processor, the method for voice data processing and the voice data processing method described in any of the above embodiments can be implemented.

[0119] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0120] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0121] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0122] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as methods applied to voice data processing. For example, in some embodiments, the method applied to voice data processing may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the method applied to voice data processing described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the method applied to speech data processing in any other appropriate manner (eg, by means of firmware).

[0123] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0125] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0127] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0128] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services; it may also be a server for a distributed system, or a server combined with a blockchain.

[0129] According to the technical solution of the embodiment of the present disclosure, the joint training includes an initial speech-to-text network and an initial text-to-text network with partially identical network structures. By encoding and transmitting text information of the initial text-to-text network during the training process, the speech data processing task corresponding to the initial speech-to-text network is assisted, thereby improving the accuracy of the speech-to-text network in processing speech data.

[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0131] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for speech data processing, comprising: Acquire a training sample set, wherein the training samples in the training sample set include corresponding speech data, phoneme data and text data; For the training samples in the training sample set, the speech data and text data in the training samples are used as the input and expected output of the initial speech-to-text network in sequence, and the phoneme data and text data in the training samples are used as the input and expected output of the initial text-to-text network in sequence, and the initial speech-to-text network and the initial text-to-text network are jointly trained, wherein the initial speech-to-text network and the initial text-to-text network include a partially identical network structure; A speech-to-text network obtained by training the initial speech-to-text network is determined as a speech data processing model.

2. The method according to claim 1, wherein: The text data includes first text data in the same language as the voice data and second text data in a different language from the voice data, and the initial speech-to-text network includes a first initial speech recognition network and a first initial speech translation network obtained based on the first initial speech recognition network; and For the training samples in the training sample set, the speech data and text data in the training samples are used as the input and expected output of the initial speech-to-text network in sequence, the phoneme data and text data in the training samples are used as the input and expected output of the initial text-to-text network in sequence, and the initial speech-to-text network and the initial text-to-text network are jointly trained, including: For the training samples in the training sample set, using the speech data and the first text data in the training samples as the input and expected output of the first initial speech recognition network in sequence, using the phoneme data and the first text data in the training samples as the input and expected output of the initial text-to-text network in sequence, jointly training the first initial speech recognition network and the initial text-to-text network, and obtaining a trained speech recognition network and a text-to-text network; Adjusting the output unit of the trained speech recognition network to an output unit for speech translation, to obtain the first initial speech translation network; For the training samples in the training sample set, the speech data and the second text data in the training samples are used as the input and expected output of the first initial speech translation network in turn, and the phoneme data and the second text data in the training samples are used as the input and expected output of the text-to-text network in turn, and the first initial speech translation network and the text-to-text network are jointly trained.

3. The method according to claim 2, wherein: The step of determining the speech-to-text network obtained by training the initial speech-to-text network as the speech data processing model comprises: A speech translation network obtained by training the first initial speech translation network is determined as the speech data processing model.

4. The method according to claim 1, wherein: The text data includes first text data in the same language as the voice data and second text data in a different language from the voice data, and the initial speech-to-text network includes a second initial speech translation network and a second initial speech recognition network obtained based on the second initial speech translation network; as well as For the training samples in the training sample set, the speech data and text data in the training samples are used as the input and expected output of the initial speech-to-text network in sequence, the phoneme data and text data in the training samples are used as the input and expected output of the initial text-to-text network in sequence, and the initial speech-to-text network and the initial text-to-text network are jointly trained, including: For the training samples in the training sample set, using the speech data and the second text data in the training samples as the input and expected output of the second initial speech translation network in sequence, using the phoneme data and the second text data in the training samples as the input and expected output of the initial text-to-text network in sequence, jointly training the second initial speech translation network and the initial text-to-text network, and obtaining a trained speech translation network and a text-to-text network; Adjusting the output unit of the trained speech translation network to an output unit for speech recognition to obtain the second initial speech recognition network; For the training samples in the training sample set, the speech data and the first text data in the training samples are used as the input and expected output of the second initial speech recognition network in turn, and the phoneme data and the first text data in the training samples are used as the input and expected output of the text-to-text network in turn, and the second initial speech recognition network and the text-to-text network are jointly trained.

5. The method according to claim 4, wherein: The step of determining the speech-to-text network obtained by training the initial speech-to-text network as the speech data processing model comprises: A speech recognition network obtained by training the second initial speech recognition network is determined as the speech data processing model.

6. The method according to any one of claims 1 to 5, wherein: Also includes: For a training sample in the training sample set, noise is replaced on a target phoneme in initial phoneme data representing speech data in the training sample to obtain phoneme data.

7. The method according to claim 1, wherein: The initial speech-to-text network includes a speech encoding module, and the initial text-to-text network includes a text encoding module, wherein the speech encoding module and the text encoding module include partially identical network structures.

8. The method according to claim 7, wherein: The initial speech-to-text network and the initial text-to-text network include the same attention module; and / or, the initial speech-to-text network and the initial text-to-text network include the same text decoding module.

9. A method for processing speech data, comprising: Get the voice to be processed; The speech to be processed is input into a speech data processing model to obtain a speech data processing result, wherein the speech data processing model is trained by the method described in any one of claims 1-8.

10. A device for voice data processing, comprising: A first acquisition unit is configured to acquire a training sample set, wherein the training samples in the training sample set include corresponding speech data, phoneme data and text data; A training unit is configured to, for a training sample in the training sample set, use the speech data and text data in the training sample as the input and expected output of an initial speech-to-text network in sequence, use the phoneme data and text data in the training sample as the input and expected output of an initial text-to-text network in sequence, and jointly train the initial speech-to-text network and the initial text-to-text network, wherein the initial speech-to-text network and the initial text-to-text network include a partially identical network structure; The determining unit is configured to determine the speech-to-text network obtained by training the initial speech-to-text network as a speech data processing model.

11. The device according to claim 10, wherein: The text data includes first text data in the same language as the voice data and second text data in a different language from the voice data, and the initial speech-to-text network includes a first initial speech recognition network and a first initial speech translation network obtained based on the first initial speech recognition network; and The training unit is further configured to: For the training samples in the training sample set, the speech data and the first text data in the training samples are used as the input and expected output of the first initial speech recognition network in sequence, and the phoneme data and the first text data in the training samples are used as the input and expected output of the initial text-to-text network in sequence, and the first initial speech recognition network and the initial text-to-text network are jointly trained to obtain the trained speech recognition network and the text-to-text network; the output unit of the trained speech recognition network is adjusted to an output unit for speech translation to obtain the first initial speech translation network; for the training samples in the training sample set, the speech data and the second text data in the training samples are used as the input and expected output of the first initial speech translation network in sequence, and the phoneme data and the second text data in the training samples are used as the input and expected output of the text-to-text network in sequence, and the first initial speech translation network and the text-to-text network are jointly trained.

12. The device according to claim 11, wherein The determining unit is further configured to: A speech translation network obtained by training the first initial speech translation network is determined as the speech data processing model.

13. The device according to claim 10, wherein: The text data includes first text data in the same language as the voice data and second text data in a different language from the voice data, and the initial speech-to-text network includes a second initial speech translation network and a second initial speech recognition network obtained based on the second initial speech translation network; and The training unit is further configured to: For the training samples in the training sample set, the speech data and the second text data in the training samples are used as the input and expected output of the second initial speech translation network in sequence, and the phoneme data and the second text data in the training samples are used as the input and expected output of the initial text-to-text network in sequence, and the second initial speech translation network and the initial text-to-text network are jointly trained to obtain a trained speech translation network and a text-to-text network; the output unit of the trained speech translation network is adjusted to an output unit for speech recognition to obtain the second initial speech recognition network; for the training samples in the training sample set, the speech data and the first text data in the training samples are used as the input and expected output of the second initial speech recognition network in sequence, and the phoneme data and the first text data in the training samples are used as the input and expected output of the text-to-text network in sequence, and the second initial speech recognition network and the text-to-text network are jointly trained.

14. The device according to claim 13, wherein: The determining unit is further configured to: A speech recognition network obtained by training the second initial speech recognition network is determined as the speech data processing model.

15. The device according to any one of claims 10 to 14, wherein: Also includes: The replacement unit is configured to perform noise replacement on the target phoneme in the initial phoneme data representing the speech data in the training sample in the training sample set to obtain the phoneme data.

16. The device according to claim 10, wherein: The initial speech-to-text network includes a speech encoding module, and the initial text-to-text network includes a text encoding module, wherein the speech encoding module and the text encoding module include partially identical network structures.

17. The device according to claim 16, wherein: The initial speech-to-text network and the initial text-to-text network include the same attention module; and / or, the initial speech-to-text network and the initial text-to-text network include the same text decoding module.

18. A speech data processing device, comprising: A second acquisition unit is configured to acquire the speech to be processed; The processing unit is configured to input the speech to be processed into a speech data processing model to obtain a speech data processing result, wherein the speech data processing model is trained by any one of claims 10-17.

19. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

21. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images