Speech translation model training method, device, electronic device and storage medium

Abstract

The present invention provides a method, apparatus, electronic device and storage medium for training a speech translation model, which are applied to the technical field of natural language processing. The method includes: acquiring source language speech data; determining semantic representation information and acoustic representation information of the source language speech data, and performing fusion processing on the semantic representation information and the acoustic representation information to obtain a speech representation sequence; training a speech translation model based on the speech representation sequence; wherein, the semantic representation information is used to indicate the semantic features of the source language speech data, and the acoustic representation information is used to indicate the acoustic features of the source language speech data.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular, to a method, device, electronic device and storage medium for training a speech translation model. Background Art

[0002] With the increasingly extensive international exchanges, the demand for automatic conversion between different languages through speech translation has become increasingly strong.

[0003] In the prior art, a speech recognition model can be used to first transcribe speech into text, and then a text machine translation model can be used to translate the transcribed text.

[0004] However, this translation method can only extract the general semantic representation in the speech, resulting in the loss of some non-verbal features, thus causing problems such as understanding ambiguity and translation errors in the translated text, and being unable to correctly convey the true intention of the source language speaker. Summary of the Invention

[0005] The present invention provides a method, device, electronic device and storage medium for training a speech translation model, so as to solve the problems that the translated text in the prior art has understanding ambiguity, translation errors, etc., and is unable to correctly convey the true intention of the source language speaker.

[0006] The present invention provides a method for training a speech translation model, including: obtaining source language speech data; determining semantic representation information and acoustic representation information of the source language speech data, and performing fusion processing on the semantic representation information and the acoustic representation information to obtain a speech representation sequence; training a speech translation model based on the speech representation sequence; wherein, the semantic representation information is used to indicate the semantic features of the source language speech data, and the acoustic representation information is used to indicate the acoustic features of the source language speech data.

[0007] According to the method for training a speech translation model provided by the present invention, the determining the semantic representation information and the acoustic representation information of the source language speech data includes: inputting the source language speech data into a speech representation separation model to obtain pitch features and prosody features; determining the pitch features and the prosody features as the acoustic representation information; wherein, the pitch features are used to indicate the intonation information during speaking, and the prosody features are used to indicate the speed and rhythm of speaking each syllable.

[0008] According to the present invention, a method for training a speech translation model is provided. Determining the semantic representation information and acoustic representation information of the source language speech data includes: inputting the source language speech data into a semantic discrete model to obtain a first speech encoding; converting the first speech encoding into discrete labels and deleting adjacent repeated discrete labels to obtain a discrete unit sequence; and determining the discrete unit sequence as the semantic representation information.

[0009] According to the present invention, a method for training a speech translation model is provided. The acoustic representation information includes pitch features and prosody features. The method for fusing the semantic representation information and the acoustic representation information to obtain a speech representation sequence includes: establishing a word embedding matrix based on the discrete unit sequence, where one discrete unit in the discrete unit sequence corresponds to one word vector in the word embedding matrix; and splicing the pitch features, the prosody features, and the word vectors in the word embedding matrix to obtain the speech representation sequence.

[0010] Before splicing the pitch features, the prosody features, and the word vectors in the word embedding matrix, the method further includes: performing a mapping process on the vector dimensions of the pitch features and the vector dimensions of the prosody features through a feedforward network to achieve unification with the vector dimensions of the word vectors.

[0011] The present invention also provides a speech translation model training device, including: an acquisition module and a processing module; the acquisition module is configured to acquire source language speech data; the processing module is configured to determine the semantic representation information and acoustic representation information of the source language speech data, fuse the semantic representation information and the acoustic representation information to obtain a speech representation sequence, and train a speech translation model based on the speech representation sequence; wherein the semantic representation information is used to indicate the semantic features of the source language speech data, and the acoustic representation information is used to indicate the acoustic features of the source language speech data.

[0012] According to the present invention, a speech translation model training device is provided. The processing module is specifically configured to input the source language speech data into a speech representation separation model to obtain pitch features and prosody features, and determine the pitch features and the prosody features as the acoustic representation information; wherein the pitch features are used to indicate intonation information during speech, and the prosody features are used to indicate the speed and rhythm of each syllable spoken.

[0013] According to the present invention, a voice translation model training device is provided. The processing module is specifically configured to input the source language voice data into a semantic discrete model to obtain a first voice encoding; convert the first voice encoding into discrete labels, and delete adjacent repeated discrete labels to obtain a discrete unit sequence; and determine the discrete unit sequence as the semantic representation information.

[0014] According to the present invention, a voice translation model training device is provided. The acoustic representation information includes pitch features and prosody features; the processing module is specifically configured to establish a word embedding matrix based on the discrete unit sequence, where one discrete unit in the discrete unit sequence corresponds to one word vector in the word embedding matrix; splice the pitch features, the prosody features and the word vectors in the word embedding matrix to obtain the voice representation sequence.

[0015] According to the present invention, a voice translation model training device is provided. The processing module is further configured to perform mapping processing on the vector dimensions of the pitch features and the vector dimensions of the prosody features through a feed-forward network to achieve unification with the vector dimensions of the word vectors.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the voice translation model training method as described in any one of the above are implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the voice translation model training method as described in any one of the above are implemented.

[0018] The voice translation model training method, device, electronic device and storage medium provided by the present invention can obtain source language voice data; determine the semantic representation information and acoustic representation information of the source language voice data, and perform fusion processing on the semantic representation information and the acoustic representation information to obtain a voice representation sequence; train a voice translation model based on the voice representation sequence; wherein, the semantic representation information is used to indicate the semantic features of the source language voice data, and the acoustic representation information is used to indicate the acoustic features of the source language voice data. Through this solution, since the voice translation model can be trained based on the voice representation sequence, and the voice representation sequence integrates semantic representation information and acoustic representation information, therefore, both explicit modeling of speech features such as semantic representation information and explicit modeling of non-speech features such as acoustic representation information can be achieved, so as to fully utilize the effective information in the source language voice data, correctly convey the true intention of the source language speaker, and reduce the occurrence probability of problems such as understanding ambiguity and translation errors. Description of the Drawings

[0019] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0020] Figure 1 is one of the schematic flowcharts of the speech translation model training method provided by the present invention;

[0021] Figure 2 is the second of the schematic flowcharts of the speech translation model training method provided by the present invention;

[0022] Figure 3 is the schematic data processing flowchart of the speech representation separation model provided by the present invention;

[0023] Figure 4 is the schematic structural diagram of the speech translation model training device provided by the present invention;

[0024] Figure 5 is the schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0026] It should be noted that in the embodiments of the present invention, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0027] It should be noted that in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or elements inherent to such a process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0028] For the convenience of clearly describing the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, terms such as "first" and "second" are used to distinguish the same or similar items with basically the same functions and roles. Those skilled in the art can understand that the terms "first" and "second" do not limit the quantity and execution order.

[0029] Some exemplary embodiments of the present invention are described for purposes of illustration. It should be understood that the present invention can be implemented in other ways not specifically shown in the drawings.

[0030] The above implementation manners will be elaborated in detail below in conjunction with specific embodiments and drawings.

[0031] As Figure 1 shown, the embodiments of the present invention provide a method for training a speech translation model. This method for training a speech translation model can be applied to a device for training a speech translation model. The method for training a speech translation model may include S101 - S103:

[0032] S101. The device for training a speech translation model acquires source language speech data.

[0033] Optionally, the device for training a speech translation model may acquire source language speech data from the training samples of the speech translation model. The training samples may further include translation texts corresponding to the source language speech data.

[0034] Optionally, the above source language speech data and translation texts may correspond to different languages. For example, the source language speech data may be English speech data, and the translation text may be Chinese text, or the source language speech data may be Chinese speech data, and the translation text may be English text.

[0035] S102. The speech translation model training device determines the semantic representation information and acoustic representation information of the source language speech data, and performs fusion processing on the semantic representation information and the acoustic representation information to obtain a speech representation sequence.

[0036] Among them, the above-mentioned semantic representation information is used to indicate the semantic features of the source language speech data, and the acoustic representation information is used to indicate the acoustic features of the source language speech data.

[0037] Optionally, as Figure 2 shown, the speech translation model training device may include a speech representation separation model, a feed-forward network, a semantic discrete model, and a speech translation model. The speech representation separation model can be used to separate the acoustic representation information from the source language speech data, and the acoustic representation information may include pitch features and prosody features. The semantic discrete model can be used to separate the semantic representation information from the source language speech data, the feed-forward network is used to unify the vector dimensions of the acoustic representation information and the semantic representation information, and the speech translation model is used to translate the source language speech data according to the semantic representation information and the acoustic representation information, so as to obtain the target speech text.

[0038] Optionally, the speech representation separation model may include six modules such as a prosody encoder, a pitch encoder, a content encoder, a timbre encoder, a pitch decoder, and a speech decoder. Among them, the prosody encoder, the pitch encoder, the content encoder, and the timbre encoder can all be stacked by 5×1 convolutional layers and bidirectional long short-term memory layers.

[0039] Optionally, the above-mentioned semantic discrete model may be a HuBERT model, and the HuBERT model may include a one-dimensional convolutional layer and 12 layers of Transformer encoders.

[0040] Optionally, the speech translation model training device may input the source language speech data into the speech representation separation model to obtain pitch features and prosody features; determine the pitch features and the prosody features as the acoustic representation information; among them, the pitch features are used to indicate the intonation information during speaking, and the prosody features are used to indicate the speed rhythm of each syllable spoken.

[0041] Optionally, before inputting the source language speech data into the speech representation separation model, the speech translation model training device may preprocess the source language speech data to obtain the Mel spectrogram feature S of the source language speech and the normalized pitch contour , where the Mel spectrogram feature is used to represent speech features, and the normalized pitch contour means having the same mean and variance for different speakers.

[0042] Specifically, as Figure 3As shown, the speech translation model training device can input the Mel-spectrum features S of the source language speech data into the prosody encoder, content encoder, timbre encoder, and pitch decoder in the speech representation separation model respectively, and input the normalized pitch contour into the pitch encoder in the speech representation separation model. It should be noted that, in order to eliminate the timbre information and prosody information, before inputting into the pitch encoder and content encoder, a random resampling operation can be performed on the normalized pitch contour and Mel-spectrum features along the time dimension. After that, the output prosody features of the prosody encoder , the output pitch features of the pitch decoder , the output content features of the content decoder , and the output timbre features of the timbre decoder can be obtained. Among them, the prosody features are used to indicate the fast and slow rhythm of each syllable spoken, the pitch features are used to indicate the intonation information during speech, the content features are used to indicate the phonemes in the speech, and the timbre features are used to indicate the voice characteristics of the speaker. After that, the speech translation model training device can select the pitch features and prosody features from the four output features of the speech representation separation model as the acoustic representation information of the source language speech data.

[0043] Optionally, the decoder of the speech representation separation model can take all the outputs of the 4 encoders as inputs and generate speech Mel-spectrum features as outputs, that is, . It should be noted that the content encoder can be represented as , the prosody encoder can be represented as , the pitch encoder can be represented as , and the timbre encoder can be represented as ; the content features can be represented as , the prosody features can be represented as , the pitch features can be represented as , and the timbre features can be represented as , where RR represents the random resampling operation.

[0044] It should be noted that after eliminating the prosody information in the content features and pitch features through the random resampling operation, the prosody features can be used to restore the prosody information.

[0045] Optionally, the speech translation model training device can also train the speech representation separation model, and the speech translation model training device can update the model parameters of the speech representation separation model based on the loss function until the function converges.

[0046] Optionally, continuing to refer to Figure 3 , the speech translation model training device can use adversarial mutual information learning to train the speech representation separation model to avoid cross-information between prosodic features , pitch features , content features and timbre features .

[0047] Optionally, the speech translation model training device can input the source language speech data into the semantic discretization model to obtain a first speech encoding; convert the first speech encoding into discrete labels, and delete adjacent repeated discrete labels to obtain a discrete unit sequence; and determine the discrete unit sequence as the semantic representation information.

[0048] Specifically, continuing to refer to Figure 2 , the speech translation model training device can input the source language speech data into the semantic discretization model, and after passing through a one-dimensional convolutional layer and 12 layers of Transformer encoders, encode the source language speech data into a first speech encoding u. This first speech encoding u is a continuous intermediate representation. Then, use k-means clustering to convert the first speech encoding u into discrete labels, and then delete adjacent repeated discrete labels to obtain a discrete unit sequence, which can be used as the semantic representation information of the source language speech data.

[0049] It should be noted that the semantic discretization model can achieve downsampling of the source language speech data. For example, for a source language speech segment with a sampling rate of 16 kHz, the sampling rate of the discrete units obtained after passing through the semantic discretization model is 25 Hz, which is equivalent to downsampling by 640 times.

[0050] Optionally, the speech translation model training device can establish a word embedding matrix based on the discrete unit sequence, where one discrete unit in the discrete unit sequence corresponds to one word vector in the word embedding matrix; splice the pitch feature, the prosodic feature with the word vectors in the word embedding matrix to obtain the speech representation sequence.

[0051] Specifically, continuing to refer to Figure 2, the discrete unit sequence includes multiple discretized semantic representation units. The speech translation model training device can establish an N-dimensional word embedding matrix with the size of the discrete unit corresponding vocabulary for the discretized semantic representation units, and each discretized semantic representation unit can be mapped to an N-dimensional word vector; then, the prosodic features and pitch features can be concatenated with the word vectors of the semantic representation units and passed through special markers <sep>Perform a segmentation process to obtain a speech representation sequence.

[0052] Optionally, before concatenating the pitch feature, prosody feature with the word vectors in the word embedding matrix, the speech translation model training device may perform a mapping process on the vector dimensions of the pitch feature and the vector dimensions of the prosody feature through a feed-forward network to achieve unification with the vector dimensions of the word vectors.

[0053] Specifically, referring to Figure 2 further, the speech translation model training device may perform a mapping process on the vector dimensions of the pitch feature and the vector dimensions of the prosody feature through a feed-forward network to make them consistent with the vector dimensions of the word vectors in the word embedding matrix, thereby providing a basis for realizing the fusion process of vectors.

[0054] S103. The speech translation model training device trains the speech translation model based on the speech representation sequence.

[0055] After obtaining the speech representation sequence, the speech translation model training device may update the source language speech data in the training sample to the speech representation sequence, thereby obtaining a new training sample. Then, the speech translation model training device may input the new training sample into the speech translation model to obtain a target language text, and the speech translation model includes a semantic encoder and a semantic decoder. Finally, the speech translation model training device may calculate a loss based on the translation text and the target language text, and through minimizing the loss and backpropagation, obtain a converged speech translation model.

[0056] It should be noted that the speech translation model training device may first train the speech translation model through a first training set, and the training samples in the first training set are samples that do not include acoustic representation information. Then, the speech translation model is corrected and trained through a second training set, and the training samples in the second training set are samples that include acoustic representation information.

[0057] In the embodiments of the present invention, since the speech translation model can be trained based on the speech representation sequence, and the speech representation sequence fuses semantic representation information and acoustic representation information, therefore, it is possible to not only explicitly model speech features such as semantic representation information, but also explicitly model non-speech features such as acoustic representation information, thereby fully utilizing the effective information in the source language speech data, correctly conveying the true intention of the source language speaker, and reducing the occurrence probability of problems such as understanding ambiguity and translation errors.

[0058] The above mainly introduces the solution provided by the embodiments of the present invention from the perspective of methods. To implement the above functions, it includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0059] For the voice translation model training method provided by the embodiments of the present invention, the execution subject can be a voice translation model training device, or a control module for voice translation model training in the voice translation model training device. In the embodiments of the present invention, taking the voice translation model training device executing the voice translation model training method as an example, the voice translation model training device provided by the embodiments of the present invention is described.

[0060] It should be noted that the embodiments of the present invention can divide the functional modules of the voice translation model training device according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiments of the present invention is illustrative, and is only a logical function division. There may be other division methods in actual implementation.

[0061] As Figure 4 shown, the embodiments of the present invention provide a voice translation model training device 400. The voice translation model training device 400 includes: an acquisition module 401 and a processing module 402. The acquisition module 401 can be used to acquire source language speech data; the processing module 402 can be used to determine the semantic representation information and acoustic representation information of the source language speech data, and perform fusion processing on the semantic representation information and the acoustic representation information to obtain a speech representation sequence; and train a voice translation model based on the speech representation sequence; wherein, the semantic representation information is used to indicate the semantic features of the source language speech data, and the acoustic representation information is used to indicate the acoustic features of the source language speech data.

[0062] Optionally, the above-mentioned processing module 402 may specifically be configured to input the source language speech data into a speech feature separation model to obtain a pitch feature and a prosody feature; and determine the pitch feature and the prosody feature as the acoustic feature information; wherein, the pitch feature is used to indicate intonation information during speaking, and the prosody feature is used to indicate the fast and slow rhythm of each syllable spoken.

[0063] Optionally, the above-mentioned processing module 402 may specifically be configured to input the source language speech data into a semantic discretization model to obtain a first speech encoding; convert the first speech encoding into discrete labels, and delete adjacent repeated discrete labels to obtain a discrete unit sequence; and determine the discrete unit sequence as the semantic feature information.

[0064] Optionally, the acoustic feature information includes a pitch feature and a prosody feature; the above-mentioned processing module 402 may specifically be configured to establish a word embedding matrix based on the discrete unit sequence, where a discrete unit in the discrete unit sequence corresponds to a word vector in the word embedding matrix; splice the pitch feature, the prosody feature, and the word vectors in the word embedding matrix to obtain the speech feature sequence.

[0065] Optionally, the above-mentioned processing module 402 may further be configured to perform mapping processing on the vector dimensions of the pitch feature and the vector dimensions of the prosody feature through a feed-forward network to achieve unification with the vector dimensions of the word vectors.

[0066] In the embodiments of the present invention, since the speech translation model can be trained based on the speech feature sequence, and the speech feature sequence integrates semantic feature information and acoustic feature information, it is possible to not only perform explicit modeling on speech features such as semantic feature information, but also perform explicit modeling on non-speech features such as acoustic feature information, thereby completing the full utilization of the effective information in the source language speech data, correctly conveying the true intention of the source language speaker, and reducing the occurrence probability of problems such as understanding ambiguity and translation errors.

[0067] Figure 5 Schematically shows a physical structure diagram of an electronic device, such as Figure 5 As shown in the figure, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540. Among them, the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logical instructions in the memory 530 to execute the method for training a speech translation model. The method includes: obtaining source language speech data; determining semantic representation information and acoustic representation information of the source language speech data, and performing fusion processing on the semantic representation information and the acoustic representation information to obtain a speech representation sequence; training a speech translation model based on the speech representation sequence; where the semantic representation information is used to indicate the semantic features of the source language speech data, and the acoustic representation information is used to indicate the acoustic features of the source language speech data.

[0068] In addition, when the logical instructions in the above-mentioned memory 530 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0069] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the method for training a speech translation model provided by the above-mentioned various methods. The method includes: obtaining source language speech data; determining semantic representation information and acoustic representation information of the source language speech data, and performing fusion processing on the semantic representation information and the acoustic representation information to obtain a speech representation sequence; training a speech translation model based on the speech representation sequence; where the semantic representation information is used to indicate the semantic features of the source language speech data, and the acoustic representation information is used to indicate the acoustic features of the source language speech data.

[0070] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the voice translation model training method provided above. The method includes: obtaining source language speech data; determining semantic representation information and acoustic representation information of the source language speech data, and performing fusion processing on the semantic representation information and the acoustic representation information to obtain a speech representation sequence; training a voice translation model based on the speech representation sequence; wherein, the semantic representation information is used to indicate the semantic features of the source language speech data, and the acoustic representation information is used to indicate the acoustic features of the source language speech data.

[0071] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

[0072] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / sep>

Claims

1. A method for training a speech translation model, characterized in that, Including: Obtain source language speech data; Determine the semantic representation information and acoustic representation information of the source language speech data, and perform fusion processing on the semantic representation information and the acoustic representation information to obtain a speech representation sequence; Train a speech translation model based on the speech representation sequence; Wherein, the semantic representation information is used to indicate the semantic features of the source language speech data, and the acoustic representation information is used to indicate the acoustic features of the source language speech data; The determining the semantic representation information and acoustic representation information of the source language speech data includes: Preprocess the source language speech data to obtain the Mel spectrum features and normalized pitch contour of the source language speech; Input the Mel spectrum features into the prosody encoder of the speech representation separation model to obtain prosody features, and input the normalized pitch contour into the pitch encoder of the speech representation separation model to obtain pitch features; Determine the pitch features and the prosody features as the acoustic representation information; Wherein, the pitch features are used to indicate the intonation information during speech, and the prosody features are used to indicate the speed and rhythm of each syllable spoken; The determining the semantic representation information and acoustic representation information of the source language speech data includes: Input the source language speech data into a semantic discrete model to obtain a first speech encoding; Convert the first speech encoding into discrete labels, and delete adjacent repeated discrete labels to obtain a discrete unit sequence; Determine the discrete unit sequence as the semantic representation information.

2. The method for training a speech translation model according to claim 1, wherein The acoustic representation information includes pitch features and prosody features; The performing fusion processing on the semantic representation information and the acoustic representation information to obtain a speech representation sequence includes: Establish a word embedding matrix based on the discrete unit sequence, and one discrete unit in the discrete unit sequence corresponds to one word vector in the word embedding matrix; Perform concatenation processing on the pitch features, the prosody features and the word vectors in the word embedding matrix to obtain the speech representation sequence.

3. The method for training a speech translation model according to claim 2, wherein Before the performing concatenation processing on the pitch features, the prosody features and the word vectors in the word embedding matrix, the method further includes: Perform mapping processing on the vector dimensions of the pitch features and the vector dimensions of the prosody features through a feed-forward network to achieve unification with the vector dimensions of the word vectors.

4. A voice translation model training device, characterized in that, Including: An acquisition module and a processing module; The acquisition module is used to obtain source language speech data; The processing module is used to determine the semantic representation information and acoustic representation information of the source language speech data, and perform fusion processing on the semantic representation information and the acoustic representation information to obtain a speech representation sequence; Train a speech translation model based on the speech representation sequence; Wherein, the semantic representation information is used to indicate the semantic features of the source language speech data, and the acoustic representation information is used to indicate the acoustic features of the source language speech data; The processing module is specifically configured to preprocess the source language speech data to obtain the Mel spectrum features and the normalized pitch contour of the source language speech; input the Mel spectrum features into the prosody encoder of the speech representation separation model to obtain prosody features, and input the normalized pitch contour into the pitch encoder of the speech representation separation model to obtain pitch features; determine the pitch features and the prosody features as the acoustic representation information; wherein, the pitch features are used to indicate the intonation information during speech, and the prosody features are used to indicate the fast and slow rhythms of each syllable spoken. The processing module is specifically configured to input the source language speech data into the semantic discretization model to obtain a first speech encoding; convert the first speech encoding into discrete labels, and delete adjacent repeated discrete labels to obtain a discrete unit sequence; determine the discrete unit sequence as the semantic representation information.

5. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the speech translation model training method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the speech translation model training method according to any one of claims 1 to 3.

Images