Speech translation method, device, equipment and computer-readable storage medium

By extracting the tone characteristics of the source speech stream and combining the diffusion model and convolutional neural network, a translated speech stream that matches the tone of the source speech stream is solved, and the problem of poor voice uniformity and interaction in the existing technology is improved. The user experience is improved .

CN118692445BActive Publication Date: 2024-11-12GUANGZHOU QUWAN NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411155650.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-12
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The voice stream generated by existing speech recognition technology is unified in tone and has poor interaction, resulting in poor user experience.

Method used

By obtaining the source speech stream in real time, extracting the tone characteristics of tone, pitch and tone, using diffusion model and convolutional neural network for semantic representation transformation and duration prediction, generate a translated speech stream that matches the tone of the source speech stream.

Benefits of technology

It improves the tone matching and communication interaction of the translated voice stream, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118692445B_ABST
    Figure CN118692445B_ABST
Patent Text Reader

Abstract

The present application discloses a speech translation method, apparatus, device and computer-readable storage medium, which can obtain a source speech stream in real time; extract timbre features for characterizing the tone, pitch and mood of the source speech stream; based on this, it can be used for subsequent simulation of the speaking timbre of a speaker; determine a translated language; perform language conversion on the source speech stream to obtain a semantic representation of the source speech stream represented by the translated language; based on this, semantic conversion of different languages ​​can be achieved; duration prediction can be performed on the semantic representation to obtain a duration prediction result; inject the timbre features into a preset diffusion model, input the duration prediction result and the semantic representation into the latest diffusion model to obtain discrete acoustic features; based on the discrete acoustic features, generate a translated speech stream that matches the timbre of the source speech stream and is in the translated language; it can be seen that the timbre of the translated speech stream generated by the present application can match the speaker who speaks the source speech stream, thereby improving the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of language conversion, and more specifically, to a speech translation method, apparatus, device and computer-readable storage medium. Background Art

[0002] In daily life, when people who do not speak the same language communicate, they often need to use speech recognition technology for speech translation. However, the speech streams generated by existing speech recognition technology often have a uniform tone and poor interactivity, resulting in a poor user experience. Summary of the invention

[0003] In view of this, the present application provides a speech translation method, apparatus, device and computer-readable storage medium to address the shortcomings of the existing speech recognition technology, such as poor user experience.

[0004] In order to achieve the above objectives, the proposed solution is as follows:

[0005] A speech translation method, comprising:

[0006] Get the source voice stream in real time;

[0007] Determine the translation language;

[0008] Performing language conversion on the source speech stream to obtain a semantic representation of the source speech stream expressed in the translated language;

[0009] Extracting timbre features for characterizing the tone, pitch and voice of the source speech stream;

[0010] Performing duration prediction on the semantic representation to obtain a duration prediction result;

[0011] Injecting the timbre feature into a preset diffusion model, inputting the duration prediction result and the semantic representation into the latest diffusion model, and obtaining discrete acoustic features;

[0012] Based on the discrete acoustic features, a translated speech stream is generated, which matches the timbre of the source speech stream and is in a translated language.

[0013] Optionally, performing language conversion on the source voice stream to obtain a semantic representation of the source voice stream expressed in the translated language includes:

[0014] Based on the speech detection technology, the source speech stream is segmented to obtain more than one speech segment;

[0015] Performing speech recognition on each speech segment, determining the source language of the source speech stream, and extracting the source semantic representation expressed in the source language;

[0016] Each source semantic representation is converted into a semantic representation segment expressed in a translated language, and each semantic representation segment constitutes a semantic representation of the source speech stream expressed in the translated language.

[0017] Optionally, the extracting timbre features for characterizing the tone, pitch and tone of the source voice stream includes:

[0018] Obtain a trained acoustic feature extraction model;

[0019] The source speech stream is input into the trained acoustic feature extraction model to obtain the timbre features of the source speech stream.

[0020] Optionally, the training process of the acoustic feature extraction model includes:

[0021] Acquire multiple training speech streams of different speakers, where the same speaker has more than one training speech stream, and each training speech stream corresponds to multiple training languages;

[0022] Inputting each training speech stream of each speaker into the acoustic feature extraction model in turn to obtain the predicted timbre features output by the acoustic feature extraction model;

[0023] Injecting the predicted timbre features into a preset training diffusion model to generate a predicted speech stream consistent with the language of the corresponding training speech stream;

[0024] Calculating the similarity between the predicted speech stream and its corresponding training speech stream;

[0025] Based on the similarity, the parameters of the acoustic feature extraction model are updated until the similarity meets a preset similarity threshold.

[0026] Optionally, performing duration prediction on the semantic representation to obtain a duration prediction result includes:

[0027] The semantic representation is input into a trained convolutional neural network for duration prediction to obtain a duration prediction result.

[0028] Optionally, the training process of the convolutional neural network includes:

[0029] Obtain training semantic representations corresponding to different languages, each training semantic representation is annotated with the corresponding total speech duration;

[0030] Input each training semantic representation into the convolutional neural network in turn, and obtain the training time corresponding to each training semantic representation output by the convolutional neural network;

[0031] Compare the training time with the label of the corresponding training semantic representation and calculate the prediction loss value;

[0032] Based on the predicted loss value, the parameters of the convolutional neural network are adjusted until the predicted loss value reaches a preset stopping condition.

[0033] Optionally, generating a translated speech stream that matches the timbre of the source speech stream and is in a translated language based on the discrete acoustic features includes:

[0034] The discrete acoustic features are input into a discrete acoustic feature-specific vocoder to generate a translated speech stream that matches the timbre of the source speech stream and is in a translated language.

[0035] A speech translation device, comprising:

[0036] An acquisition module is used to acquire the source voice stream in real time;

[0037] A determination module, used for determining a translation language;

[0038] A conversion module, configured to perform language conversion on the source speech stream to obtain a semantic representation of the source speech stream expressed in the translated language;

[0039] An extraction module, used for extracting timbre features for characterizing the tone, pitch and tone of the source speech stream;

[0040] A prediction module, used to perform duration prediction on the semantic representation to obtain a duration prediction result;

[0041] An input module, used to inject the timbre feature into a preset diffusion model, input the duration prediction result and the semantic representation into the latest diffusion model, and obtain discrete acoustic features;

[0042] A generation module is used to generate a translated speech stream that matches the timbre of the source speech stream and is in a translated language based on the discrete acoustic features.

[0043] A speech translation device, comprising a memory and a processor;

[0044] The memory is used to store programs;

[0045] The processor is used to execute the program to implement the various steps of the above-mentioned dialect-to-speech translation method.

[0046] A computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, each step of the above-mentioned speech translation method is implemented.

[0047] It can be seen from the above technical solutions that the speech translation method provided by the present application can obtain the source speech stream in real time; extract the timbre features used to characterize the tone, pitch and tone of the source speech stream; based on this, the timbre features can reflect the timbre-related features such as the tone features, pitch features and tone features of the speaker who speaks the source speech stream, which can be used for subsequent simulation of the speaker's speaking timbre; then, the translated language can be determined; the source speech stream is language-converted to obtain the semantic representation of the source speech stream expressed in the translated language; based on this, the semantic representation of the source speech stream described in the translated language can be generated to realize the semantic conversion of different languages, first converting the source speech stream into a semantic representation to avoid errors in the semantic translation; then After that, the duration of the semantic representation can be predicted to obtain the duration prediction result; the timbre feature is injected into the preset diffusion model, and the duration prediction result and the semantic representation are input into the latest diffusion model to obtain discrete acoustic features; based on the discrete acoustic features, a translated voice stream is generated that matches the timbre of the source voice stream and is in the translated language; based on this, the present application can further ensure the reliability of the generated voice stream through duration prediction, and use the diffusion model to combine the translated semantic representation and the tone features, pitch features and tone features that reflect the speaker, so that the generated translated voice stream can simulate the speaker's tone, pitch and tone while translating, improve communication interactivity, and thus improve the user experience. It can be seen that the timbre of the translated voice stream generated by the present application can match the speaker who speaks the source voice stream, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 A flow chart of a speech translation method disclosed in an embodiment of the present application;

[0050] Figure 2 A structural block diagram of a speech translation device disclosed in an embodiment of the present application;

[0051] Figure 3 The following is a hardware structure block diagram of a speech translation device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] The speech translation method of the present application can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, etc.

[0054] Next, combine Figure 1 The speech translation method of the present application is described in detail, including the following steps:

[0055] Step S1: Acquire the source voice stream in real time.

[0056] Specifically, the user's operation of uploading a voice stream on the interactive interface can be responded to in real time to obtain the source voice stream.

[0057] Step S2: Determine the translation language.

[0058] Specifically, more than one translation language may be determined in response to user needs.

[0059] The translated language is different from the language corresponding to the source speech stream.

[0060] The translated language of the present application is not limited to the official language of each country, that is, the translated language can be the official language of any country or various dialects.

[0061] Step S3: perform language conversion on the source speech stream to obtain a semantic representation of the source speech stream expressed in the translated language.

[0062] Specifically, the speech detection algorithm and the speech semantic recognition algorithm can be called through the modular algorithm service layer to perform semantic recognition on the source speech stream to form a semantic representation, and the semantic representation is expressed in a translated language.

[0063] Step S4: extracting timbre features for characterizing the tone, pitch and voice of the source speech stream.

[0064] Specifically, a preset acoustic feature extraction model may be used to extract timbre features that can be used to simulate the timbre of a source speech stream.

[0065] The timbre features may include tone features, pitch features, tone features, and other feature information that can be used to simulate the timbre of the source speech stream.

[0066] Step S5: perform duration prediction on the semantic representation to obtain a duration prediction result.

[0067] Specifically, the duration of each semantic representation of the source speech stream may be predicted to obtain a duration prediction result corresponding to each semantic representation.

[0068] Step S6: inject the timbre feature into a preset diffusion model, input the duration prediction result and the semantic representation into the latest diffusion model, and obtain discrete acoustic features.

[0069] Specifically, the timbre features can be injected into a diffusion model for generating a speech stream of a corresponding language based on semantic representation.

[0070] Each semantic representation can be input into a diffusion model annotated with the corresponding timbre feature to obtain a discrete acoustic feature output by the diffusion model.

[0071] Among them, discrete acoustic features can be used to indicate the information required for speech generation, such as pronunciation content and pronunciation method.

[0072] The training process of the diffusion model can be as follows:

[0073] Multiple semantic training features annotated with speech training duration can be obtained;

[0074] Input each semantic training feature into the diffusion model in turn to obtain the diffusion speech generation result generated by the diffusion model;

[0075] The semantic recognition of the diffuse speech generation result is performed to obtain the semantic training result;

[0076] The training similarity between the semantic training results and the corresponding semantic training features is calculated, and the parameters of the diffusion model are adjusted according to the training similarity until the training similarity meets the preset conditions.

[0077] Step S7: Based on the discrete acoustic features, generate a translated speech stream that matches the timbre of the source speech stream and is in the translated language.

[0078] Specifically, a translated speech stream may be generated based on discrete acoustic features.

[0079] The similarity between the timbre of the translated voice stream and the timbre of the source voice stream exceeds a preset timbre threshold, and the language corresponding to the translated voice stream is the translated language.

[0080] It can be seen from the above technical solutions that the speech translation method provided by the present application can obtain the source speech stream in real time; extract the timbre features used to characterize the tone, pitch and tone of the source speech stream; based on this, the timbre features can reflect the tone-related features such as the tone features, pitch features and tone features of the speaker who speaks the source speech stream, which can be used for subsequent simulation of the speaker's speaking timbre; then, the translated language can be determined; the source speech stream is language-converted to obtain the semantic representation of the source speech stream expressed in the translated language; based on this, the semantic representation of the source speech stream described in the translated language can be generated to realize the semantic conversion of different languages, and the source speech stream is first converted into a semantic representation to avoid errors in the semantic translation; Then, the duration of the semantic representation can be predicted to obtain a duration prediction result; the timbre feature is injected into a preset diffusion model, and the duration prediction result and the semantic representation are input into the latest diffusion model to obtain discrete acoustic features; based on the discrete acoustic features, a translated voice stream is generated that matches the timbre of the source voice stream and is in the translated language; based on this, the present application can further ensure the reliability of the generated voice stream through duration prediction, and the diffusion model combines the translated semantic representation with the tone features, pitch features and tone features that reflect the speaker, so that the generated translated voice stream can simulate the speaker's tone, pitch and tone while translating, improve communication interactivity, and thus improve the user experience. It can be seen that the timbre of the translated voice stream generated by the present application can match the speaker who speaks the source voice stream, thereby improving the user experience.

[0081] In addition, the present application can receive the source voice stream in real time, and can perform voice stream translation in real time, thereby improving the timeliness of the translation.

[0082] Furthermore, the translated voice stream generated by the present application can be stored in a data storage service together with the corresponding source voice stream, so as to manage and control each voice stream.

[0083] In response to the user's operation of looking up the historical voice stream, the corresponding historical source voice stream and the historical translated voice stream can be searched from the data storage service.

[0084] Furthermore, in response to the user's operation of scoring the translated voice stream on the interactive interface, the translated voice stream can be marked with a score, and the marking situation can be recorded in the message center.

[0085] Each marking situation may include an identifier, a score, and a marking time of the corresponding translated voice stream.

[0086] Furthermore, in response to a selection operation by the user, a translation language may be selected from a plurality of languages.

[0087] Specifically, in response to a selection operation of the user on the interactive interface, any one or more languages ​​can be selected from a plurality of languages ​​as the translation language.

[0088] In some embodiments of the present application, the process of performing language conversion on the source voice stream and obtaining the semantic representation of the source voice stream represented by the translated language in step S3 is described in detail, and the steps are as follows:

[0089] S30. Based on the speech detection technology, the source speech stream is segmented to obtain more than one speech segment.

[0090] Specifically, a speech detection algorithm may be used to segment the source speech stream, and multiple speech segments are obtained after segmentation.

[0091] S31. Perform speech recognition on each speech segment, determine the source language of the source speech stream, and extract source semantic representation expressed in the source language.

[0092] Specifically, a speech recognition algorithm may be used to perform speech recognition on each speech segment and extract the source semantic representation of each speech segment.

[0093] The source semantic representation adopts the source language representation corresponding to the source speech stream.

[0094] S32: Convert each source semantic representation into a semantic representation segment expressed in a translated language, wherein each semantic representation segment constitutes a semantic representation of the source speech stream expressed in the translated language.

[0095] Specifically, each source semantic representation may be text-translated based on the translated language, and after translation, a semantic representation segment expressed in the translated language may be obtained.

[0096] Each semantic representation segment may be sorted according to the order of each source semantic representation, and after sorting, a semantic representation of the source speech stream represented in the translated language is formed.

[0097] It can be seen from the above technical solution that the above embodiment provides an optional way to obtain the semantic representation of the source speech stream expressed in the translated language. Through the above method, a speech stream can be converted into multiple speech segments, and semantic recognition can be performed on each speech segment, and the recognition process of a long speech stream can be converted into the recognition process of short speech segments, which reduces the complexity of speech recognition, and each speech segment can be semantically recognized at the same time, further improving the accuracy of semantic representation and the efficiency of language conversion.

[0098] In some embodiments of the present application, the process of extracting the timbre features for characterizing the tone, pitch and tone of the source voice stream in step S4 is described in detail, and the steps are as follows:

[0099] S40: Obtain a trained acoustic feature extraction model.

[0100] Specifically, a preset trained acoustic feature extraction model may be called.

[0101] S41. Input the source speech stream into the trained acoustic feature extraction model to obtain the timbre features of the source speech stream.

[0102] Specifically, the source speech stream may be input into an acoustic feature extraction model to obtain timbre features indicating the tone, pitch and tone of the source speech stream.

[0103] It can be seen from the above technical solution that this embodiment provides an optional method for obtaining the timbre characteristics of the speaker who emits the source speech stream. Through the above method, a trained acoustic feature extraction model can be used to extract the timbre characteristics, further ensuring the accuracy of the generation of the timbre features of the present application.

[0104] In some embodiments of the present application, the training process of the acoustic feature extraction model is described in detail, and the steps are as follows:

[0105] S400: Acquire multiple training speech streams from different speakers. The same speaker may have more than one training speech stream, and each training speech stream corresponds to multiple training languages.

[0106] Specifically, a plurality of training speech streams may be obtained, each of which is produced by a plurality of speakers.

[0107] The same speaker can correspond to training speech streams of different languages.

[0108] The total length of the training speech stream can vary.

[0109] S401. Input each training speech stream of each speaker into the acoustic feature extraction model in turn to obtain the predicted timbre features output by the acoustic feature extraction model.

[0110] Specifically, each training speech stream can be input into the acoustic feature extraction model one by one to obtain the predicted timbre features output by the acoustic feature extraction model.

[0111] The predicted timbre feature may be a discrete feature.

[0112] S402: Inject the predicted timbre features into a preset training diffusion model to generate a predicted speech stream that is consistent with the language of the corresponding training speech stream.

[0113] Specifically, each output predicted timbre feature may be injected into the training diffusion model in sequence to obtain a predicted speech stream output by the training diffusion model that is consistent with the language of the corresponding training speech stream.

[0114] The training diffusion model may be a diffusion model used for training, which may be the same model as the diffusion model before the timbre features are injected as described above.

[0115] S403: Calculate the similarity between the predicted speech stream and its corresponding training speech stream.

[0116] Specifically, the predicted speech stream may be matched with the corresponding training speech stream for similarity.

[0117] S404: Based on the similarity, update the parameters of the acoustic feature extraction model until the similarity meets a preset similarity threshold.

[0118] Specifically, the parameters of the acoustic feature extraction model may be updated according to the degree of similarity until the degree of similarity meets a preset similarity threshold.

[0119] It can be seen from the above technical solution that this embodiment provides an optional way to train an acoustic feature extraction model. The above method can further improve the matching of the timbre features extracted by the acoustic feature extraction model with the timbre of the source speech stream.

[0120] In some embodiments of the present application, step S5, performing duration prediction on the semantic representation to obtain a duration prediction result is described in detail, and the steps are as follows:

[0121] S50: Input the semantic representation into a trained convolutional neural network to perform duration prediction to obtain a duration prediction result.

[0122] Specifically, each duration prediction result is used to indicate the total duration of the speech corresponding to the semantic representation, so as to avoid speech stream generation errors caused by character duration prediction being too short or too long when performing duration prediction for each character of the semantic representation.

[0123] It can be seen from the above technical solution that this embodiment provides an optional method for obtaining duration prediction results. Through the above method, the total duration of the semantic representation can be predicted, further avoiding errors in the translated speech stream and improving the accuracy of the translated speech stream.

[0124] In some embodiments of the present application, the training process of the convolutional neural network is described in detail, and the steps are as follows:

[0125] S500: Obtain training semantic representations corresponding to different languages, each training semantic representation being annotated with a corresponding total speech duration.

[0126] Specifically, training semantic representations expressed in different languages ​​may be obtained.

[0127] Each training semantic representation may be annotated with the total duration of speech.

[0128] S501. Input each training semantic representation into the convolutional neural network in turn to obtain the training duration corresponding to each training semantic representation output by the convolutional neural network.

[0129] Specifically, each training semantic representation can be input into the convolutional neural network one by one to obtain the training duration of each training semantic representation predicted by the convolutional neural network.

[0130] S502: Compare the training duration with the annotation label of the corresponding training semantic representation, and calculate the prediction loss value.

[0131] Specifically, the difference between each training duration and the total duration of speech of the corresponding training semantic representation can be used as the prediction loss value.

[0132] S503: Based on the predicted loss value, adjust the parameters of the convolutional neural network until the predicted loss value reaches a preset stop condition.

[0133] Specifically, the gradient descent method can be used to adjust the parameters of the convolutional neural network based on the predicted loss value until the predicted loss value reaches a preset stop condition.

[0134] It can be seen from the above technical solution that this embodiment provides an optional method for training a convolutional neural network, and the reliability of duration prediction can be further improved through the above method.

[0135] In some embodiments of the present application, the process of step S7, generating a translated speech stream that matches the timbre of the source speech stream and is in the translated language based on the discrete acoustic features, is described in detail, and the steps are as follows:

[0136] S70, inputting the discrete acoustic features into a discrete acoustic feature-specific vocoder to generate a translated speech stream that matches the timbre of the source speech stream and is in a translated language.

[0137] Specifically, a discrete acoustic feature-specific vocoder may be used to generate a translated speech stream that matches the timbre of the source speech stream and is in the translated language based on the discrete acoustic features.

[0138] It can be seen from the above technical solution that this embodiment provides an optional way to generate a translated speech stream. Through the above method, a discrete acoustic feature dedicated vocoder can be used to generate a translated speech stream, simplifying the difficulty of generating the translated speech stream.

[0139] Next, we will combine Figure 2The speech translation device provided in the present application is introduced in detail. The speech translation device provided below can be compared with the speech translation method provided above.

[0140] See also Figure 2 It can be found that the speech translation device may include:

[0141] An acquisition module 10 is used to acquire a source voice stream in real time;

[0142] A determination module 20, used to determine a translation language;

[0143] A conversion module 30, configured to perform language conversion on the source speech stream to obtain a semantic representation of the source speech stream expressed in the translated language;

[0144] An extraction module 40, for extracting timbre features for characterizing the tone, pitch and tone of the source speech stream;

[0145] A prediction module 50, configured to perform duration prediction on the semantic representation to obtain a duration prediction result;

[0146] An input module 60 is used to inject the timbre feature into a preset diffusion model, input the duration prediction result and the semantic representation into the latest diffusion model, and obtain discrete acoustic features;

[0147] The generating module 70 is used to generate a translated speech stream whose timbre matches that of the source speech stream and whose language is the translated language based on the discrete acoustic features.

[0148] Furthermore, the conversion module may include:

[0149] A first conversion unit, configured to segment the source voice stream based on a voice detection technology to obtain more than one voice segment;

[0150] A second conversion unit is used to perform speech recognition on each speech segment, determine the source language of the source speech stream, and extract the source semantic representation expressed in the source language;

[0151] The third conversion unit is used to convert each source semantic representation into a semantic representation segment expressed in a translated language, and each semantic representation segment constitutes a semantic representation of the source speech stream expressed in the translated language.

[0152] Furthermore, the extraction module may include:

[0153] An acoustic feature extraction model acquisition unit, used to acquire a trained acoustic feature extraction model;

[0154] The timbre feature extraction unit is used to input the source speech stream into the trained acoustic feature extraction model to obtain the timbre features of the source speech stream.

[0155] Further, the acoustic feature extraction model acquisition unit may include:

[0156] A training speech stream acquisition component is used to acquire multiple training speech streams from different speakers. The same speaker may have more than one training speech stream, and each training speech stream corresponds to multiple training languages.

[0157] A predicted timbre feature acquisition component is used to sequentially input each training speech stream of each speaker into the acoustic feature extraction model to obtain the predicted timbre features output by the acoustic feature extraction model;

[0158] A predicted speech stream acquisition component, used for injecting the predicted timbre features into a preset training diffusion model to generate a predicted speech stream consistent with the language of the corresponding training speech stream;

[0159] A similarity calculation component, used to calculate the similarity between the predicted speech stream and its corresponding training speech stream;

[0160] The acoustic feature extraction model updating component is used to update the parameters of the acoustic feature extraction model based on the similarity until the similarity meets a preset similarity threshold.

[0161] Furthermore, the prediction module may include:

[0162] The convolutional neural network utilizing unit is used to input the semantic representation into a trained convolutional neural network for duration prediction to obtain a duration prediction result.

[0163] Further, the convolutional neural network utilizing unit may include:

[0164] A training semantic representation acquisition component is used to acquire training semantic representations corresponding to different languages, each training semantic representation is annotated with the corresponding total speech duration;

[0165] A training duration prediction component is used to input each training semantic representation into the convolutional neural network in turn to obtain the training duration corresponding to each training semantic representation output by the convolutional neural network;

[0166] The prediction loss value calculation component is used to compare the training time with the annotation label of the corresponding training semantic representation to calculate the prediction loss value;

[0167] The convolutional neural network adjustment component is used to adjust the parameters of the convolutional neural network based on the predicted loss value until the predicted loss value reaches a preset stop condition.

[0168] Furthermore, the generation module may include:

[0169] The translated speech stream generating unit is used to input the discrete acoustic features into a discrete acoustic feature-specific vocoder to generate a translated speech stream that matches the timbre of the source speech stream and is in a translated language.

[0170] The speech translation apparatus provided in the embodiment of the present application can be applied to speech translation equipment, such as PC terminals, cloud platforms, servers and server clusters. Figure 3 The hardware structure diagram of the speech translation device is shown in FIG. Figure 3 ,The hardware structure of the speech translation device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0171] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0172] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0173] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;

[0174] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to:

[0175] Get the source voice stream in real time;

[0176] Determine the translation language;

[0177] Performing language conversion on the source speech stream to obtain a semantic representation of the source speech stream expressed in the translated language;

[0178] Extracting timbre features for characterizing the tone, pitch and voice of the source speech stream;

[0179] Performing duration prediction on the semantic representation to obtain a duration prediction result;

[0180] Injecting the timbre feature into a preset diffusion model, inputting the duration prediction result and the semantic representation into the latest diffusion model, and obtaining discrete acoustic features;

[0181] Based on the discrete acoustic features, a translated speech stream is generated, which matches the timbre of the source speech stream and is in a translated language.

[0182] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0183] The embodiment of the present application further provides a computer-readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0184] Get the source voice stream in real time;

[0185] Determine the translation language;

[0186] Performing language conversion on the source speech stream to obtain a semantic representation of the source speech stream expressed in the translated language;

[0187] Extracting timbre features for characterizing the tone, pitch and voice of the source speech stream;

[0188] Performing duration prediction on the semantic representation to obtain a duration prediction result;

[0189] Injecting the timbre feature into a preset diffusion model, inputting the duration prediction result and the semantic representation into the latest diffusion model, and obtaining discrete acoustic features;

[0190] Based on the discrete acoustic features, a translated speech stream is generated, which matches the timbre of the source speech stream and is in a translated language.

[0191] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0192] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0193] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0194] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. The various embodiments of the present application can be combined with each other. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech translation method, It is characterized in that include: Get the source voice stream in real time; Determine the translation language; Performing language conversion on the source speech stream to obtain a semantic representation of the source speech stream expressed in the translated language; Obtain a trained acoustic feature extraction model; Inputting the source speech stream into the trained acoustic feature extraction model to obtain the timbre features of the source speech stream; Performing duration prediction on the semantic representation to obtain a duration prediction result; Injecting the timbre feature into a preset diffusion model, inputting the duration prediction result and the semantic representation into the diffusion model injected with the timbre feature, and obtaining a discrete acoustic feature; Based on the discrete acoustic features, generating a translated speech stream that matches the timbre of the source speech stream and is in a translated language; The training process of the acoustic feature extraction model includes: Acquire multiple training speech streams of different speakers, where the same speaker has more than one training speech stream, and each training speech stream corresponds to multiple training languages; Inputting each training speech stream of each speaker into the acoustic feature extraction model in turn to obtain the predicted timbre features output by the acoustic feature extraction model; Injecting the predicted timbre features into a preset training diffusion model to generate a predicted speech stream consistent with the language of the corresponding training speech stream; Calculating the similarity between the predicted speech stream and its corresponding training speech stream; Based on the similarity, the parameters of the acoustic feature extraction model are updated until the similarity meets a preset similarity threshold.

2. The speech translation method according to claim 1, It is characterized in that Performing language conversion on the source speech stream to obtain a semantic representation of the source speech stream expressed in the translated language includes: Based on the speech detection technology, the source speech stream is segmented to obtain more than one speech segment; Performing speech recognition on each speech segment, determining the source language of the source speech stream, and extracting the source semantic representation expressed in the source language; Each source semantic representation is converted into a semantic representation segment expressed in a translated language, and each semantic representation segment constitutes a semantic representation of the source speech stream expressed in the translated language.

3. The speech translation method according to claim 1, It is characterized in that Performing duration prediction on the semantic representation to obtain a duration prediction result includes: The semantic representation is input into a trained convolutional neural network for duration prediction to obtain a duration prediction result.

4. The speech translation method according to claim 3, It is characterized in that The training process of a convolutional neural network includes: Obtain training semantic representations corresponding to different languages, each training semantic representation is annotated with the corresponding total speech duration; Input each training semantic representation into the convolutional neural network in turn, and obtain the training time corresponding to each training semantic representation output by the convolutional neural network; Compare the training time with the label of the corresponding training semantic representation and calculate the prediction loss value; Based on the predicted loss value, the parameters of the convolutional neural network are adjusted until the predicted loss value reaches a preset stopping condition.

5. The speech translation method according to any one of claims 1 to 4, It is characterized in that Based on the discrete acoustic features, generating a translated speech stream that matches the timbre of the source speech stream and is in a translated language, comprising: The discrete acoustic features are input into a discrete acoustic feature-specific vocoder to generate a translated speech stream that matches the timbre of the source speech stream and is in a translated language.

6. A speech translation device, It is characterized in that include: An acquisition module is used to acquire the source voice stream in real time; A determination module, used for determining a translation language; A conversion module, configured to perform language conversion on the source speech stream to obtain a semantic representation of the source speech stream expressed in the translated language; An extraction module, used to obtain a trained acoustic feature extraction model; Inputting the source speech stream into the trained acoustic feature extraction model to obtain the timbre features of the source speech stream; A prediction module, used to perform duration prediction on the semantic representation to obtain a duration prediction result; An input module, used for injecting the timbre feature into a preset diffusion model, inputting the duration prediction result and the semantic representation into the diffusion model injected with the timbre feature, and obtaining a discrete acoustic feature; A generating module, configured to generate a translated speech stream that matches the timbre of the source speech stream and is in a translated language based on the discrete acoustic features; The extraction model includes: A training speech stream acquisition component is used to acquire multiple training speech streams from different speakers. The same speaker may have more than one training speech stream, and each training speech stream corresponds to multiple training languages. A predicted timbre feature acquisition component is used to sequentially input each training speech stream of each speaker into the acoustic feature extraction model to obtain the predicted timbre features output by the acoustic feature extraction model; A predicted speech stream acquisition component, used for injecting the predicted timbre features into a preset training diffusion model to generate a predicted speech stream consistent with the language of the corresponding training speech stream; A similarity calculation component, used to calculate the similarity between the predicted speech stream and its corresponding training speech stream; The acoustic feature extraction model updating component is used to update the parameters of the acoustic feature extraction model based on the similarity until the similarity meets a preset similarity threshold.

7. A speech translation device, It is characterized in that including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the speech translation method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, each step of the speech translation method according to any one of claims 1 to 5 is implemented.