Speech translation method, system and related device

Abstract

The invention discloses a speech translation method, system and related device, and the method comprises the steps: determining an audio feature and a text conversion feature corresponding to a to-be-translated audio, and an acoustic feature matched with a target object based on the to-be-translated audio of the target object; obtaining link information matched with a translation task, and coding based on the link information, the audio features, the text conversion features and the acoustic features to obtain target conversion features matched with the to-be-translated audio; and decoding the target conversion feature to obtain a translated audio corresponding to the audio to be translated. In this way, the accuracy of speech translation can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech translation method, system and related devices. Background Art

[0002] With the continuous development of artificial intelligence, voice translation is being applied in more and more scenarios. For example, in large-scale conferences or lectures, there is a language barrier between speakers and participants, which makes it difficult for participants to accurately understand what the speakers are saying. Current voice translation methods mainly rely on neural network models to translate word by word based on the input audio stream. Due to the differences in expressions in different languages, neural network models are prone to asynchronous sentences during translation, resulting in a poor user experience.

[0003] In view of this, how to improve the accuracy of audio broadcasting has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a speech translation method, system and related devices, which can improve the accuracy of speech translation.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: providing a speech translation method, comprising: based on the audio to be translated of the target object, determining the audio features, text conversion features corresponding to the audio to be translated and the acoustic features matching the target object; obtaining link information matching the translation task, and encoding the target conversion features matching the audio to be translated based on the link information, the audio features, the text conversion features and the acoustic features; decoding the target conversion features to obtain the translated audio corresponding to the audio to be translated.

[0006] To solve the above technical problems, another technical solution adopted by the present application is: to provide a speech translation system, including: an acquisition module, used to determine the audio features, text conversion features and acoustic features matching the target object corresponding to the audio to be translated based on the audio to be translated of the target object; a processing module, used to obtain link information matching the translation task, and based on the link information, the audio features, the text conversion features and the acoustic features, encode the target conversion features matching the audio to be translated; a translation module, used to decode the target conversion features to obtain the translated audio corresponding to the audio to be translated.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising: a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the method mentioned in the above technical solution.

[0008] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the method mentioned in the above technical solution is implemented.

[0009] The beneficial effects of the present application are as follows: Different from the prior art, the speech translation method proposed in the present application simplifies the translation task by acquiring link information matching the translation task, thereby reducing the difficulty of translating the audio to be translated. In addition, link information, audio features, text conversion features and acoustic features are combined to obtain target conversion features according to multiple types of feature encoding, thereby improving the expressiveness of the target conversion features and helping to improve the accuracy of translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a flowchart of an implementation method of the speech translation method of the present application; Figure 2 yes Figure 1 Step S101 corresponds to a flow chart of another implementation method; Figure 3 yes Figure 1 Step S102 corresponds to a flow chart of another implementation method; Figure 4 yes Figure 3 Step S302 corresponds to a flow chart of another implementation method; Figure 5 yes Figure 1 Step S103 corresponds to a flow chart of another implementation method; Figure 6 It is a structural diagram of an implementation method of the speech translation system of the present application; Figure 7 It is a structural schematic diagram of an embodiment of the electronic device of the present application; Figure 8 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different embodiments can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] See also Figure 1 , Figure 1 1 is a flow chart of an implementation method of a speech translation method of the present application, the method comprising: S101: Based on the audio to be translated of the target object, determine the audio features corresponding to the audio to be translated, the text conversion features and the acoustic features matching the target object.

[0013] In one embodiment, in order to translate the audio expressed by the target object, the audio expressed by the target object is used as the audio to be translated. According to the audio to be translated, the corresponding audio features, text conversion features and acoustic features matching the target object are determined.

[0014] Specifically, the audio features are obtained by extracting audio features from the audio to be translated. The text conversion features are obtained by converting the audio to be translated into a converted text. The acoustic features are obtained by extracting acoustic features from the audio to be translated, and the acoustic features include at least one of timbre information, speech rate information, and prosody information that match the target object.

[0015] In one implementation scenario, a trained audio recognition model and a trained acoustic feature extraction model are obtained. The audio to be translated is input into the audio recognition model to obtain audio features corresponding to the audio to be translated, and corresponding text conversion features are obtained according to the audio feature recognition. Furthermore, the audio to be translated is input into the acoustic feature extraction model to extract acoustic features.

[0016] In another embodiment, in order to improve the accuracy of translation, after obtaining the audio to be translated of the target object, the audio to be translated is subjected to noise reduction processing to remove environmental noise. The audio to be translated after noise removal is processed to obtain corresponding audio features, text conversion features and acoustic features.

[0017] S102: Acquire link information matching the translation task, and encode target conversion features matching the audio to be translated based on the link information, audio features, text conversion features, and acoustic features.

[0018] In one embodiment, a translation task corresponding to the audio to be translated is determined, and the translation task is used to represent the conversion of the audio to be translated into a translated audio matching the target language. The target language may be determined according to the application scenario, or may be instructed by relevant personnel.

[0019] Furthermore, based on decomposing the translation task, matching link information is obtained. According to the link information, audio features, text conversion features and acoustic features, target conversion features for synthesizing translation audio are encoded. Among them, the above-mentioned target conversion features include acoustic features matching the target object and translation text information.

[0020] S103: Decode the target conversion feature to obtain the translated audio corresponding to the audio to be translated.

[0021] In one implementation, the obtained target conversion feature is decoded to generate a translated audio corresponding to the audio to be translated.

[0022] Specifically, in response to the target conversion feature including an acoustic feature matching the target object, the generated translation audio also has an acoustic feature matching the target object to achieve simultaneous interpretation.

[0023] The speech translation method proposed in this application simplifies the translation task by acquiring link information matching the translation task, thereby reducing the difficulty of translating the audio to be translated. In addition, link information, audio features, text conversion features and acoustic features are combined to obtain target conversion features according to multiple types of feature encoding, thereby improving the expressiveness of the target conversion features and helping to improve the accuracy of translation.

[0024] See also Figure 2 , Figure 2 yes Figure 1 Step S101 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S101 includes: S201: Based on the audio to be translated, corresponding audio features and text conversion features are obtained.

[0025] In one implementation, audio features corresponding to the audio to be translated are obtained, the audio features are recognized, and text conversion features are obtained.

[0026] In one implementation scenario, a trained audio recognition model is obtained, and the audio to be translated is input into a feature encoding network in the audio recognition model to obtain audio features corresponding to the audio to be translated output by the feature encoding network. The audio features are recognized to obtain corresponding text conversion features, which correspond to the conversion text corresponding to the audio to be translated, and the text conversion features are used to generate the above conversion text. The structure of the above audio recognition model can refer to the existing neural network structure, which will not be elaborated in detail here.

[0027] S202: extracting an audio segment from the audio to be translated, and acquiring acoustic features matching the target object based on the audio segment.

[0028] In one embodiment, the start acquisition time corresponding to the audio to be translated is obtained. Based on the start acquisition time, an audio segment with a target duration is obtained. By extracting the audio segment from the audio to be translated, it is avoided to directly use the complete audio to be translated to extract acoustic features, which saves computing consumption and helps to improve the efficiency of translation. The audio segment is input into the acoustic feature extraction model to extract acoustic features matching the target object.

[0029] Specifically, the time when the audio to be translated is collected is taken as the starting collection time. From the starting collection time, an audio segment with a target duration is intercepted from the collected audio to be translated. The target duration can be obtained by estimation or by reverse calculation by relevant technical personnel after multiple experiments.

[0030] In another embodiment, the audio segment may also be randomly extracted from the audio to be translated, and its corresponding duration is the target duration.

[0031] It should be noted that, in actual applications, the execution order of step S201 and step S202 may also be other, for example, executing step S201 and step S202 at the same time; or, executing step S202 first, and then executing step S201.

[0032] See also Figure 3 , Figure 3 yes Figure 1 Step S102 in the flowchart corresponds to another implementation method. Specifically, the link information includes multiple translation subtasks, and the implementation process of step S102 includes: S301: Decomposing the translation task using an intelligent analysis model to generate a plurality of corresponding translation subtasks.

[0033] In one embodiment, in order to improve the accuracy of the target conversion feature, the translation task is decomposed using an intelligent analysis model to generate multiple translation subtasks, and the multiple translation subtasks constitute the link information mentioned in step S102.

[0034] Specifically, an intelligent analysis model is used to simulate the human thinking and reasoning process, and the translation task is decomposed into multiple corresponding translation sub-tasks to reduce the difficulty of translating the audio to be translated.

[0035] In one implementation scenario, the intelligent analysis model is a large language model with better data analysis capabilities. By inputting the translation task into the intelligent analysis model, the intelligent analysis model simulates human thinking to generate multiple translation subtasks to construct link information.

[0036] In a specific application scenario, the above-mentioned large language model may include but is not limited to deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTM), and generative pre-trained Transformer models, etc. No specific restrictions are made on the specific structure and specific deployment of the large language model. In addition, it should be noted that the specific structure and specific deployment of the intelligent analysis model mentioned in other implementations of this application can refer to this implementation.

[0037] S302: Input the translation subtask, audio features, text conversion features and acoustic features into the intelligent analysis model, and use the intelligent analysis model to obtain target conversion features that match the audio to be translated.

[0038] In one embodiment, the translation subtask, audio features, text conversion features and acoustic features are input into the intelligent analysis model. According to the correlation between different translation subtasks, the reply text corresponding to the current translation subtask is generated by combining the audio features, text conversion features, acoustic features and the reply text corresponding to the generated translation subtask, until the target conversion feature matching the audio to be translated is obtained. Among them, the above-mentioned target conversion feature includes the translation text information and the acoustic information of the target object, the translation text information matches the translation text corresponding to the conversion text, and the conversion text is determined based on the text conversion feature.

[0039] In a specific application scenario, in response to the fact that the language corresponding to the audio to be translated expressed by the target object is Chinese, and the target language for translation is English, the implementation process of step S301 and step S302 includes: using the intelligent analysis model to generate multiple translation subtasks, which are "extracting text conversion information", "converting the conversion text information into translation text information matching the target language", "converting the translation text information into reference conversion features", and "converting the reference conversion features into target conversion features including acoustic information".

[0040] In another embodiment, in order to improve the accuracy of the target conversion features obtained by encoding the intelligent analysis model, a plurality of first training samples are used to fine-tune the intelligent analysis model to obtain a trained intelligent analysis model. The first training sample is matched with the corresponding target conversion features. During the fine-tuning training process, the intelligent analysis model is prompted in advance to predict the corresponding conversion text based on the input first training sample, and then the intelligent analysis model is prompted to predict the translation text corresponding to the conversion text, and output the predicted conversion features, which include the translation text information and acoustic information corresponding to the first training sample. At least some parameters in the intelligent analysis model are adjusted according to the predicted conversion features and the target conversion features until the preset convergence conditions are met to obtain the fine-tuned intelligent analysis model.

[0041] In one implementation scenario, in combination with a translation scenario, a first training sample related to the scenario is obtained, and the LoRA model (Low-Rank Adaptation of Large Language Models) is used to fine-tune the intelligent analysis model to obtain a fine-tuned intelligent analysis model. For example, when the translation scenario is related to the earth environment, multiple first training samples related to the earth environment are obtained, and the obtained first training samples are used to fine-tune the intelligent analysis model.

[0042] The above solution generates link information by using an intelligent analysis model to simulate human thinking, thereby reducing the difficulty of obtaining target conversion features in the future. In addition, the intelligent analysis model is used to combine link information, audio features, text conversion features and acoustic features to generate target conversion features, which helps to enhance the expressiveness of target conversion features, thereby improving the accuracy of subsequent speech translation.

[0043] See also Figure 4 , Figure 4 yes Figure 3 Step S302 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S302 includes: S401: Based on audio features, text conversion features and acoustic features, an intelligent analysis model is used to generate reply information matching each translation subtask.

[0044] In one embodiment, the translation subtask, audio features, text conversion features and acoustic features are input into an intelligent analysis model, so that the intelligent analysis model is used to analyze the above audio features, text conversion features and acoustic features, and outputs reply information matching each translation subtask.

[0045] In a specific application scenario, multiple translation subtasks generated in response to the use of an intelligent analysis model include "extracting text conversion information", "obtaining translation text information matching the target language", and "extracting acoustic information of the target object". For the translation subtask "extracting conversion text information", the intelligent analysis model is used to combine audio features and text conversion features and perform further extraction to obtain text conversion information. For the translation subtask "obtaining translation text information matching the target language", the intelligent analysis model is used to obtain translation text information based on text conversion features, and the translation text information is a feature vector representation of the translation text corresponding to the audio to be translated. For the translation subtask "extracting acoustic information of the target object", the intelligent analysis model is used to further extract acoustic features to obtain acoustic information.

[0046] S402: Generate target conversion features based on all reply information, wherein the target conversion features include translation text information and acoustic information of the target object, the translation text information matches the translation text corresponding to the conversion text, and the conversion text is determined based on the text conversion features.

[0047] In one embodiment, an intelligent analysis model is used to jointly analyze and summarize all the response information obtained, so as to finally output a target conversion feature containing the translated text information and the acoustic information.

[0048] Specifically, after obtaining all the reply information generated by the intelligent analysis model, the intelligent analysis model is prompted to generate target conversion features for synthesizing translated audio based on all the reply information, that is, the intelligent analysis model generates target conversion features by combining the translated text information and acoustic information.

[0049] See also Figure 5 , Figure 5 yes Figure 1 Step S103 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S103 includes: S501: Input the target conversion features into the trained codebook compilation model to generate target decoding features.

[0050] In one implementation, a trained codebook coding model is obtained, a target conversion feature is input into the codebook coding model, a corresponding codebook feature is generated, and the codebook feature is used as a target decoding feature.

[0051] In one application scenario, the trained codebook coding model converts the target conversion features into target decoding features for synthesizing translated audio through RVQ (Residual Vector Quantization) technology.

[0052] In another embodiment, in order to improve the accuracy of translation, the trained codebook compilation model also has the function of predicting codebook features. After the target conversion feature is input into the codebook compilation model, the codebook compilation model is used to output the initial codebook feature that matches the target conversion feature. Based on the initial codebook feature, multiple predicted codebook features predicted by the codebook compilation model are obtained. Among them, the initial codebook feature and the predicted codebook feature both contain text information corresponding to the complete translation text and acoustic information corresponding to the target object, and different predicted codebook features have different sampling rates. The initial codebook feature and the predicted codebook feature are used as the target decoding feature.

[0053] Specifically, the trained codebook compilation model is used to generate the initial codebook features of digital representation according to the target conversion features. Based on the obtained initial codebook features, the predicted codebook features matching multiple different sampling rates are predicted, and the initial codebook features and all the predicted codebook features are used as the target decoding features. By using the initial codebook features to predict multiple different predicted codebook features, the expressive power of the target decoding features is improved, which helps to improve the accuracy of subsequent translation.

[0054] In one implementation scenario, the codebook coding model is trained using a plurality of second training samples, and the second training samples include target training features and corresponding target training audio. The target training features are input into the codebook coding model so that the codebook coding model outputs training decoding features including a plurality of codebook features. According to the training decoding features, the predicted audio is synthesized. The training loss of the model is calculated based on the predicted audio and the target training audio, and the parameters of the codebook coding model are adjusted using the training loss until the preset convergence conditions are met. The codebook coding model is trained using the second training samples so that the trained codebook coding model can output codebook features corresponding to a plurality of sampling rates as target decoding features, and the translated audio is synthesized based on the target decoding features.

[0055] In a specific application scenario, the structure of the codebook coding model may refer to the existing nonlinear autoregressive model structure.

[0056] S502: Decode the target decoding feature to obtain the translated audio corresponding to the audio to be translated.

[0057] In one embodiment, the generated target decoding features are decoded to synthesize corresponding translated audio.

[0058] Specifically, the target decoding features are input into the decoder to obtain the translated audio output by the decoder.

[0059] In another embodiment, the present application also proposes a speech translation model, which is used to implement the speech translation method mentioned in any of the above embodiments. The speech translation model includes a pre-trained audio recognition model, a pre-trained acoustic feature extraction model, a fine-tuned intelligent analysis model, a pre-trained codebook compilation model and a pre-trained decoder. The audio recognition model is used to obtain the audio features and text conversion features corresponding to the audio to be translated; the acoustic feature extraction model is used to extract acoustic features that match the target object; the intelligent analysis model is used to construct link information and encode the target conversion features; the codebook compilation model is used to generate the target decoding features; the decoder is used to decode the target decoding features.

[0060] In one implementation, in order to improve the accuracy of speech translation performed by a speech translation model, after the speech translation model is constructed, joint training is performed on the speech translation model to improve the stability of the speech translation model.

[0061] See also Figure 6 , Figure 6 1 is a schematic diagram of the structure of an embodiment of the speech translation system of the present application. The speech translation system comprises an acquisition module 10, a processing module 20 and a translation module 30 which are coupled to each other.

[0062] Specifically, the acquisition module 10 is used to determine the audio features corresponding to the audio to be translated, the text conversion features and the acoustic features matched by the target object based on the audio to be translated of the target object.

[0063] The processing module 20 is used to obtain link information matching the translation task, and encode the target conversion features matching the audio to be translated based on the link information, audio features, text conversion features and acoustic features.

[0064] The translation module 30 is used to decode the target conversion feature to obtain the translated audio corresponding to the audio to be translated.

[0065] In one embodiment, the link information includes multiple translation subtasks, and the processing module 20 obtains the link information matching the translation task, and encodes the target conversion features matching the audio to be translated based on the link information, audio features, text conversion features and acoustic features, including: using an intelligent analysis model to decompose the translation task to generate multiple corresponding translation subtasks; inputting the translation subtasks, audio features, text conversion features and acoustic features into the intelligent analysis model, and using the intelligent analysis model to obtain the target conversion features matching the audio to be translated.

[0066] In one embodiment, the processing module 20 inputs the translation subtask, audio features, text conversion features and acoustic features into the intelligent analysis model, and uses the intelligent analysis model to obtain target conversion features that match the audio to be translated, including: based on the audio features, text conversion features and acoustic features, using the intelligent analysis model to generate reply information matching each translation subtask; based on all reply information, generating target conversion features; wherein the target conversion features include translation text information and acoustic information of the target object, the translation text information matches the translation text corresponding to the conversion text, and the conversion text is determined based on the text conversion features.

[0067] In one embodiment, the translation module 30 decodes the target conversion feature to obtain the translated audio corresponding to the audio to be translated, including: inputting the target conversion feature into the trained codebook compilation model to generate a target decoding feature; decoding the target decoding feature to obtain the translated audio corresponding to the audio to be translated.

[0068] In one embodiment, the translation module 30 inputs the target conversion feature into the trained codebook compilation model to generate a target decoding feature, including: using the codebook compilation model to output an initial codebook feature that matches the target conversion feature; based on the initial codebook feature, obtaining multiple predicted codebook features predicted by the codebook compilation model; wherein different predicted codebook features have different sampling rates; and using the initial codebook feature and the predicted codebook feature as the target decoding feature.

[0069] In one embodiment, the acquisition module 10 determines the audio features, text conversion features corresponding to the audio to be translated, and acoustic features matching the target object based on the audio to be translated of the target object, including: acquiring the corresponding audio features and text conversion features based on the audio to be translated; and extracting an audio segment from the audio to be translated, and acquiring the acoustic features matching the target object based on the audio segment.

[0070] In one embodiment, the acquisition module 10 acquires corresponding audio features and text conversion features based on the audio to be translated, including: acquiring audio features corresponding to the audio to be translated, identifying the audio features, and obtaining text conversion features.

[0071] The acquisition module 10 extracts an audio segment from the audio to be translated, including: acquiring a starting acquisition time corresponding to the audio to be translated; and acquiring an audio segment with a target duration based on the starting acquisition time.

[0072] See also Figure 7 , Figure 7: is a structural diagram of an embodiment of an electronic device of the present application. The electronic device comprises: a memory 40 and a processor 50 coupled to each other. The memory 40 stores program instructions, and the processor 50 is used to execute the program instructions to implement the method mentioned in any of the above embodiments. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 50 can also be called a CPU (Center Processing Unit). The processor 50 may be an integrated circuit chip with signal processing capabilities. The processor 50 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 50 may be implemented by an integrated circuit chip.

[0073] See also Figure 8 , Figure 8 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 60 stores program instructions 70 that can be executed by a processor. When the program instructions 70 are executed by the processor, the method mentioned in any of the above embodiments is implemented.

[0074] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0076] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0077] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0078] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A speech translation method, characterized in that: include: Based on the audio to be translated of the target object, determining the audio features corresponding to the audio to be translated, the text conversion features and the acoustic features matching the target object; Acquire link information matching the translation task, and encode target conversion features matching the audio to be translated based on the link information, the audio features, the text conversion features, and the acoustic features; The target conversion feature is decoded to obtain a translated audio corresponding to the audio to be translated.

2. The method according to claim 1, characterized in that The link information includes a plurality of translation subtasks, and the link information matching the translation task is obtained, and based on the link information, the audio feature, the text conversion feature and the acoustic feature, a target conversion feature matching the audio to be translated is encoded, including: Decomposing the translation task by using an intelligent analysis model to generate a plurality of corresponding translation subtasks; The translation subtask, the audio feature, the text conversion feature and the acoustic feature are input into the intelligent analysis model, and the target conversion feature matching the audio to be translated is obtained by using the intelligent analysis model.

3. The method according to claim 2, characterized in that The step of inputting the translation subtask, the audio feature, the text conversion feature, and the acoustic feature into the intelligent analysis model, and using the intelligent analysis model to obtain the target conversion feature that matches the audio to be translated includes: Based on the audio features, the text conversion features and the acoustic features, using the intelligent analysis model to generate reply information matching each of the translation subtasks; Based on all the reply information, the target conversion feature is generated; wherein the target conversion feature includes translation text information and acoustic information of the target object, the translation text information matches the translation text corresponding to the conversion text, and the conversion text is determined based on the text conversion feature.

4. The method according to claim 1, characterized in that: The decoding of the target conversion feature to obtain the translated audio corresponding to the audio to be translated includes: Inputting the target conversion features into the trained codebook compilation model to generate target decoding features; The target decoding feature is decoded to obtain the translated audio corresponding to the audio to be translated.

5. The method according to claim 4, characterized in that The step of inputting the target conversion feature into the trained codebook compilation model to generate the target decoding feature includes: Outputting an initial codebook feature matching the target conversion feature using the codebook coding model; Based on the initial codebook features, a plurality of predicted codebook features predicted by the codebook compilation model are obtained; wherein different predicted codebook features have different sampling rates; The initial codebook features and the predicted codebook features are used as the target decoding features.

6. The method according to claim 1, characterized in that The step of determining the audio features corresponding to the audio to be translated, the text conversion features, and the acoustic features matching the target object based on the audio to be translated of the target object includes: Based on the audio to be translated, obtaining the corresponding audio features and the text conversion features; and, An audio segment is extracted from the audio to be translated, and based on the audio segment, the acoustic feature matching the target object is acquired.

7. The method according to claim 6, characterized in that The obtaining the corresponding audio features and text conversion features based on the audio to be translated includes: Acquire the audio feature corresponding to the audio to be translated, identify the audio feature, and obtain a text conversion feature; The step of extracting an audio segment from the audio to be translated comprises: Obtaining the starting acquisition time corresponding to the audio to be translated; Based on the start acquisition time, the audio segment with an audio duration equal to a target duration is obtained.

8. A speech translation system, characterized in that: include: An acquisition module, configured to determine, based on the audio to be translated of the target object, audio features corresponding to the audio to be translated, text conversion features, and acoustic features matching the target object; A processing module, configured to obtain link information matching the translation task, and encode a target conversion feature matching the audio to be translated based on the link information, the audio feature, the text conversion feature and the acoustic feature; The translation module is used to decode the target conversion feature to obtain the translated audio corresponding to the audio to be translated.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images