Speech translation method, device, equipment and product

Abstract

The invention provides a speech translation method, device, equipment and product, and is applied to the technical field of artificial intelligence. The speech translation method comprises the steps that speech to be translated and associated data of the speech to be translated are acquired, the associated data comprise associated images and / or associated videos, the associated images comprise image information of at least part of objects described by the speech to be translated, and the associated videos display lip actions when a speaking object expresses the speech to be translated; and based on the to-be-translated speech and the associated data, translating the to-be-translated speech through the multi-modal speech translation model to obtain a translation result of the to-be-translated speech. Therefore, in combination with the images and / or videos related to the voice, end-to-end voice translation is carried out through the multi-mode voice translation model, and the voice translation quality is effectively improved.

Description

Technical Field

[0001] The present application is applied to the field of artificial intelligence technology, and in particular relates to a speech translation method, apparatus, device and product. Background Art

[0002] Voice translation technology is used in a wide variety of products and scenarios, including social media apps, translation machines, and other products, as well as in trade exchanges, business matching, and international travel. Therefore, high-quality, low-latency voice translation technology is attracting significant attention and is in high market demand.

[0003] In related technologies, speech translation is achieved in stages. For example, translating speech from a source language into a target language involves first using speech recognition technology to transcribe the source language speech into text in the source language. Next, a machine translation system translates the source language text into text in the target language. Finally, speech synthesis technology generates and broadcasts the corresponding speech in the target language.

[0004] However, the above translation process has the problem of error accumulation, which leads to the decline of speech translation quality. Summary of the Invention

[0005] In order to solve the above problems, the present application proposes a speech translation method, apparatus, device and product, which can improve the quality of speech translation.

[0006] In a first aspect, the present application provides a speech translation method, comprising: obtaining speech to be translated and associated data of the speech to be translated, the associated data including associated images and / or associated videos, the associated images containing image information of at least part of an object described by the speech to be translated, and the associated videos displaying lip movements of a speaker when expressing the speech to be translated; and translating the speech to be translated using a multimodal speech translation model based on the speech to be translated and the associated data to obtain a translation result of the speech to be translated.

[0007] In one possible implementation, the speech translation model includes a multimodal feature extraction module and a large language model. The method of translating the speech to be translated using the multimodal speech translation model based on the speech to be translated and the associated data to obtain a translation result of the speech to be translated includes: extracting features from the speech to be translated and the associated data using the multimodal feature extraction module to obtain a multimodal feature representation; generating prompt information for the large language model based on the multimodal feature representation; and generating the translation result at least using the large language model based on the prompt information.

[0008] In one possible implementation, the multimodal feature extraction module includes a speech encoder, an adapter corresponding to the speech encoder, an image encoder, an adapter corresponding to the image encoder, a video encoder, and an adapter corresponding to the video encoder. The multimodal feature extraction module performs feature extraction on the speech to be translated and the associated data to obtain a multimodal feature representation, including: performing feature encoding and feature conversion on the speech to be translated by the speech encoder and the adapter corresponding to the speech encoder to obtain a speech feature representation; performing feature encoding and feature conversion on the associated image by the image encoder and the adapter corresponding to the image encoder to obtain an image feature representation; and / or performing feature encoding and feature conversion on the associated video by the video encoder and the adapter corresponding to the video encoder to obtain a video feature representation; the multimodal features include the speech feature representation and the associated feature representation, and the associated features include the image feature representation and / or the video feature representation.

[0009] In one possible implementation, the speech translation model includes a combination module, and generating prompt information of the large language model based on the multimodal feature representation includes: inputting the multimodal feature representation into the combination module, performing cross-modal interaction between the multimodal features in the combination module to obtain a target feature representation; and generating the prompt information based on the target feature representation.

[0010] In a possible implementation, generating the prompt information according to the target feature representation includes: combining translation task information with the target feature representation to obtain the prompt information, wherein the translation task information indicates performing speech translation.

[0011] In one possible implementation, the translation result includes translated text, and generating the translation result at least through the large language model according to the prompt information includes: generating the translated text according to the prompt information in the large language model; and / or, the translation result includes translated speech, and the speech translation model also includes a speech generation model, and generating the translation result at least through the large language model according to the prompt information includes: generating the translated speech based on the prompt information through the large language model and the speech generation model, wherein the input data of the speech generation model includes hidden layer features generated by the large language model based on the prompt information.

[0012] In one possible implementation, the process of determining the input data of the speech generation model includes: predicting hidden layer features belonging to the same semantic scope among the hidden layer features generated by the large language model based on the prompt information; and determining that the input data includes the hidden layer features belonging to the same semantic scope.

[0013] In one possible implementation, predicting hidden layer features belonging to the same semantic scope among the hidden layer features generated by the large language model based on the prompt information includes: identifying hidden layer features corresponding to semantically interrupting characters among the hidden layer features generated by the large language model based on the prompt information; and determining hidden layer features located between the hidden layer features corresponding to the semantically interrupting characters as the hidden layer features belonging to the same semantic scope.

[0014] In one possible implementation, the input data further includes translation-assisting features of the speech to be translated, where the translation-assisting features are extracted from the associated video via a video encoder and / or are extracted from the speech to be translated via a speech encoder; the translation-assisting features include emotional features and / or speaking object features.

[0015] In one possible implementation, the speech generation model includes an acoustic feature prediction module and an audio synthesis module. After determining the input data, the process of generating the translated speech includes: inputting the input data into the acoustic feature prediction module, and generating acoustic features based on the input data in the acoustic feature prediction module; inputting the acoustic features into the audio synthesis module, and performing audio synthesis based on the acoustic features in the audio synthesis module to obtain synthesized audio; and generating the translated speech based on the synthesized audio.

[0016] In one possible implementation, the training process of the speech translation model includes: obtaining first training data, where the first training data includes a first training speech, an associated image of the first training speech, an associated video of the first training speech, and a real translation result of the first training speech, where the associated image of the first training speech contains image information of at least part of the object described by the first training speech, and the associated video of the first training speech shows the lip movements of the speaker when expressing the first training speech; and performing speech translation training on the speech translation model based on the first training data.

[0017] In one possible implementation, the speech translation model includes a large language model. Before training the speech translation model based on the first training data, the method further includes: obtaining second training data, where the second training data includes a second training speech and a real translation result of the second training speech; and training the large language model based on the second training data.

[0018] In one possible implementation, before training the speech translation model based on the first training data, the method further includes: obtaining third training data, where the third training data includes a third training speech and a real translation result of the third training speech, and the third training data also includes an associated image of the third training speech or an associated video of the third training data; and training the speech translation model based on the third training data.

[0019] In a second aspect, the present application provides a speech translation device, comprising: an acquisition unit, configured to acquire speech to be translated and associated data of the speech to be translated, the associated data comprising associated images and / or associated videos, the associated images comprising image information of at least part of an object described by the speech to be translated, and the associated videos displaying lip movements of a speaking subject when expressing the speech to be translated; and a translation unit, configured to translate the speech to be translated based on the speech to be translated and the associated data using a multimodal speech translation model to obtain a translation result of the speech to be translated.

[0020] In a third aspect of the present application, an electronic device is provided, comprising a memory and a processor; the memory is connected to the processor and is used to store a program; the processor is used to implement the speech translation method as described in the first aspect of the present application or any possible implementation method of the first aspect of the present application by running the program in the memory.

[0021] In a fourth aspect, the present application provides a chip comprising a processor and a data interface, wherein the processor reads and runs a program stored in a memory through the data interface to execute the speech translation method as described in the first aspect of the present application or any possible implementation of the first aspect of the present application.

[0022] A fifth aspect of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the computer program implements the speech translation method as described in the first aspect of the present application or any possible implementation of the first aspect of the present application.

[0023] In a sixth aspect, the present application provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the speech translation method as described in the first aspect of the present application or any possible implementation of the first aspect of the present application.

[0024] According to a speech translation method, apparatus, equipment and product proposed in the present application, first, the speech to be translated and the associated data of the speech to be translated are obtained, the associated data including associated images and / or associated videos, the associated images contain image information of at least part of the object described by the speech to be translated, and the associated videos show the lip movements of the speaking object when expressing the speech to be translated. It can be seen that for the speech to be translated, the associated images and associated videos are data of other modalities, and the content displayed by the associated images and associated videos is related to the speech to be translated. Then, based on the speech to be translated and the associated data, the speech to be translated is translated through a multimodal speech translation model to obtain a translation result of the speech to be translated. Combined with the above-mentioned characteristics of the associated images and associated videos, it can be seen that the present application realizes end-to-end speech translation based on multimodal data, effectively improving the quality of speech translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0026] Figure 1 A schematic diagram of an implementation environment according to an embodiment of the present application;

[0027] Figure 2 The process diagram of the speech translation method provided according to the embodiment of the present application is as follows Figure 1 ;

[0028] Figure 3 The process diagram of the speech translation method provided according to the embodiment of the present application is as follows Figure 2 ;

[0029] Figure 4 Example of a multimodal speech translation model structure Figure 1 ;

[0030] Figure 5 Example of a multimodal speech translation model structure Figure 2 ;

[0031] Figure 6 Example of a multimodal speech translation model structure Figure 3 ;

[0032] Figure 7 Example of a multimodal speech translation model structure Figure 4 ;

[0033] Figure 8This is an example flow chart of the training process of a multimodal speech translation model;

[0034] Figure 9 A schematic diagram of the structure of a speech translation device provided according to an embodiment of the present application;

[0035] Figure 10 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] First, the speech-to-speech translation technology and existing technical problems are described.

[0038] In one approach, a speech recognition system first recognizes the collected speech content as text in the original language. A machine translation system then translates the original text into text in the target language. Finally, text-to-speech (TTS) technology is used to synthesize the target text into speech in the target language. This approach involves a multi-stage process: speech recognition from the original language speech to text, translation from the original language text to text in the target language, and speech synthesis from text in the target language to speech in the target language. This approach is also known as a strong cascade approach. In this strong cascade approach, there are gaps in the coordination between the various stages, and there is also the problem of error accumulation.

[0039] In another approach, a speech translation system directly translates speech in the source language into text in the target language; using text-to-speech (TTS) technology, the target language text is synthesized into speech in the target language. This approach is also a multi-stage process, but it combines the speech recognition and machine translation stages of the strong cascade approach into a single speech translation stage, directly translating speech into text. This is why it's called a weak cascade approach. While the weak cascade approach mitigates some of the shortcomings of the strong cascade approach, there are still gaps in the coordination between the various stages, and the problem of error accumulation persists.

[0040] Among other issues, the problem of error accumulation leads to a decline in translation quality. For example, speech recognition performance deteriorates in scenarios involving rare and difficult-to-use words, high noise levels, accents, and dialects, significantly reducing the quality of downstream machine translation. There are gaps in the coordination between various stages, and information loss is significant. This increases the latency of speech translation. Furthermore, speech synthesis technology cannot effectively integrate information from the input speech, making it easy for the output speech to not match the timbre of the original speaker. The speed of output at each stage can also lead to pauses and a lack of naturalness in the translated speech.

[0041] Another approach involves directly translating the input speech into the target speech without any intermediate text conversion. This approach is known as end-to-end translation. This approach reduces the gaps and error accumulation in the intermediate process, reduces translation latency, and improves the accuracy and naturalness of the output. However, this approach lacks data support, and its effectiveness remains to be seen.

[0042] The embodiments of the present application propose a speech translation method, apparatus, device and product, which obtains the speech to be translated and the associated data of the speech to be translated, wherein the associated data includes associated images and / or associated videos, the associated images include image information of at least part of the object described by the speech to be translated, and the associated videos show the lip movements of the speaking object when expressing the speech to be translated; based on the speech to be translated and the associated data, the speech to be translated is translated through a multimodal speech translation model to obtain a translation result of the speech to be translated. Thus, based on the multimodal data and the multimodal speech translation model, end-to-end speech translation is achieved, and various problems in the cascade mode (such as the gap problem of multi-stage coordination, error accumulation problem, information loss problem between multiple stages, unnatural translation speech problem, etc.) are solved in an end-to-end manner, and the speech translation quality of the end-to-end mode is improved through multimodal data.

[0043] Exemplary Implementation Environment

[0044] Please refer to Figure 1 , Figure 1 Schematic diagram of an implementation environment according to an embodiment of the present application. The implementation environment includes a processing device 110, on which a multimodal speech translation model is deployed. On the processing device 110, the speech to be translated can be translated by the multimodal speech translation model based on the speech to be translated and its associated data (the associated data includes associated images and / or associated videos), ultimately obtaining a translation result.

[0045] Among them, the implementation environment scenario can be a local voice translation scenario or an online voice translation scenario. In the local voice translation scenario, the processing device 110 can be a user's terminal device, such as a mobile phone, computer, smart wearable device, smart pen, car terminal, etc.; in the online voice translation scenario, the processing device 110 can be a server, cloud device, etc. Figure 1 Take the processing device 120 as an example, which is a server.

[0046] Exemplary Methods

[0047] See also Figure 2 In an exemplary embodiment, a speech translation method is provided, the speech translation method comprising the following steps:

[0048] S201: Acquire the speech to be translated and its associated data, where the associated data includes an associated image and / or an associated video. The associated image includes image information of at least a portion of the object described by the speech to be translated, and the associated video displays the lip movements of the speaker when expressing the speech to be translated.

[0049] The number of associated images can be one or more, and the associated images contain image information of at least a portion of the object described by the speech to be translated, such as at least one object, at least two objects, etc. The objects here can be, for example, people, animals, plants, virtual characters, buildings, weather, etc. The number of associated videos can be one or more, and the speaking subject refers to the subject who speaks the speech to be translated, and can be a real or virtual person.

[0050] In one example, the associated video includes a lip video of a speaker, where the video frames of the lip video show the lip movements of the speaker when expressing the speech to be translated.

[0051] In another example, the associated video includes a facial video of a speaker, and the video frames of the facial video show the speaker's facial expression when expressing the language to be translated, and the facial expression includes lip movements.

[0052] In actual scenarios, users can choose to provide associated images or associated videos for the translation of the speech to be translated, making the provision of multimodal data more flexible and reducing the pressure of data provision.

[0053] In this embodiment, the speech to be translated and its associated data can be obtained from a user input via an interactive page displayed on a terminal. Alternatively, the speech to be translated and its associated data can be received from another device. Alternatively, the speech to be translated and its associated data can be obtained from a translation request message or from a translation task list.

[0054] In one example, when the associated data cannot be obtained (for example, the user is inconvenient to provide the associated data or the associated data cannot be obtained through other means), or when the associated data is lost during transmission or a data error occurs, the associated data may not be obtained or used. Based on the speech to be translated, the speech to be translated may be translated through a multimodal speech translation model. At this time, the multimodal speech translation model acts as a single-modal speech translation model.

[0055] S202 : Based on the speech to be translated and the associated data, the speech to be translated is translated using a multimodal speech translation model to obtain a translation result of the speech to be translated.

[0056] Among them, the multimodal speech translation model is an artificial intelligence model based on deep learning technology, which has the ability to perform end-to-end speech translation based on multimodal data.

[0057] Among them, the translation result of the speech to be translated may include the translated text of the speech to be translated and / or the translated speech of the speech to be translated, so as to realize end-to-end translation of speech to text and end-to-end translation of speech to speech, and improve the translation quality of speech to text and speech to speech.

[0058] The language of the speech to be translated and the language of the translation result can be different languages. The language of the speech to be translated is the original language, and the language of the translation result is the target language, so as to achieve end-to-end speech translation from the original language to the target language.

[0059] In this embodiment, the speech to be translated and the associated data can be input into a multimodal speech translation model. In the multimodal speech translation model, the speech to be translated is translated by combining the feature information of the speech to be translated and the feature information of the associated data to obtain a translation result of the speech to be translated.

[0060] In one example, before obtaining the speech to be translated and the data associated with the speech to be translated, a speech translation request is received. If the speech translation request indicates that the speech to be translated in the source language be translated into text in the target language, the translation result includes the translated text; if the speech translation request indicates that the speech to be translated in the source language be translated into speech in the target language, the translation result includes the translated speech.

[0061] In the embodiments of the present application, the content displayed by the associated images and associated videos is related to the speech to be translated. The associated images and / or associated videos are combined and the speech to be translated is translated using a multimodal speech translation model. This not only achieves end-to-end speech translation, but also solves the problem of strong and weak association forms. Furthermore, it provides multimodal data for end-to-end speech translation, improving the quality of end-to-end speech translation. Therefore, the quality of speech translation is effectively improved, the delay of speech translation is reduced, and the efficiency of speech translation is improved.

[0062] In some embodiments, the speech translation model includes a multimodal feature extraction module and a large language model. Corresponding embodiments are given below with reference to the accompanying drawings.

[0063] See also Figure 3 In another exemplary embodiment, a speech translation method is provided, the speech translation method comprising the following steps:

[0064] S301: Acquire the speech to be translated and its associated data, where the associated data includes an associated image and / or an associated video. The associated image includes image information of at least a portion of the object described by the speech to be translated, and the associated video displays the lip movements of the speaker when expressing the speech to be translated.

[0065] The implementation principle and technical effects of S301 may refer to the aforementioned embodiments and will not be described in detail.

[0066] S302 , performing feature extraction on the speech to be translated and the associated data through a multimodal feature extraction module in a multimodal speech translation model to obtain a multimodal feature representation.

[0067] Multimodal feature representations can include speech feature representations and context feature representations. Speech feature representations are obtained by extracting features from the speech to be translated, while context feature representations are obtained by extracting features from contextual data. Feature representations can be understood as embedded representations or embedded vectors.

[0068] In this embodiment, the speech to be translated and the associated data can be input into a multimodal feature extraction module. Within the multimodal feature extraction module, feature extraction can be performed on the speech to be translated to obtain a speech feature representation, and feature extraction can be performed on the associated data to obtain an associated feature representation. Because the associated data includes associated images and / or associated videos, the multimodal feature extraction module can be used to perform feature extraction on the images associated with the speech to be translated to obtain an image feature representation, and / or the multimodal feature extraction module can be used to perform feature extraction on the videos associated with the speech to be translated to obtain a video feature representation. Thus, an associated feature representation comprising an image feature representation and / or a video feature representation can be obtained.

[0069] In one example, the multimodal feature extraction module includes multiple encoders, and the multiple encoders perform feature encoding on the speech to be translated and the associated data respectively to obtain a multimodal feature representation.

[0070] In another example, the multimodal feature extraction module also includes adapters (also called adapters, which are components used to adjust feature representations) corresponding to multiple encoders, so that the feature representations of different modalities output by the encoders are converted into feature representations of text modality through the adapters, thereby facilitating the large language model to perform speech translation based on the feature representations of text modality.

[0071] In one possible implementation, the multimodal feature extraction module includes a speech encoder, an adapter corresponding to the speech encoder, an image encoder, an adapter corresponding to the image encoder, a video encoder, and an adapter corresponding to the video encoder. S302 includes: S3021, performing feature encoding and feature conversion on the speech to be translated using the speech encoder and the adapter corresponding to the speech encoder to obtain a speech feature representation; S3022, performing feature encoding and feature conversion on the associated image using the image encoder and the adapter corresponding to the image encoder to obtain an image feature representation; and / or performing feature encoding and feature conversion on the associated video using the video encoder and the adapter corresponding to the video encoder to obtain a video feature representation. Thus, the speech encoder, the adapter corresponding to the speech encoder, the image encoder, the adapter corresponding to the image encoder, the video encoder, and the adapter corresponding to the video encoder perform feature extraction on data of different modalities, namely, the speech to be translated, the associated image, and the associated video, respectively, thereby improving the accuracy of feature extraction for multimodal data.

[0072] The multimodal extraction module includes a speech encoder, an adapter corresponding to the speech encoder, an image encoder, an adapter corresponding to the image encoder, a video encoder, and an adapter corresponding to the video encoder, enabling the multimodal extraction module to extract features from speech, images, and videos. When the associated data includes speech to be translated and associated images, feature extraction can be performed using the speech encoder, the adapter corresponding to the speech encoder, the image encoder, and the adapter corresponding to the image encoder. When the associated data includes speech to be translated and associated videos, feature extraction can be performed using the speech encoder, the adapter corresponding to the speech encoder, the video encoder, and the adapter corresponding to the video encoder. When the associated data includes speech to be translated, associated images, and associated videos, feature extraction can be performed using the speech encoder, the adapter corresponding to the speech encoder, the image encoder, the adapter corresponding to the image encoder, the video encoder, and the adapter corresponding to the video encoder. In this way, multimodal feature extraction can be flexibly performed for different input situations.

[0073] In this implementation, the speech to be translated can be input into a speech encoder, where the speech encoder performs feature encoding on the speech to be translated, obtaining an encoded feature representation of the speech modality. The encoded feature representation of the speech modality is then input into an adapter corresponding to the speech encoder, which converts the encoded feature representation of the speech modality into a text modality to obtain a speech feature representation. If the associated data includes an associated image, the associated image can be input into an image encoder, where the image encoder performs feature encoding on the associated image to obtain an encoded feature representation of the image modality. The encoded feature representation of the image modality is then input into an adapter corresponding to the image encoder, which converts the encoded feature representation of the image modality into a text modality to obtain an image feature representation. If the associated data includes an associated video, the associated video can be input into a video encoder, where the video encoder performs feature encoding on the associated video to obtain an encoded feature representation of the video modality. The encoded feature representation of the video modality is then input into an adapter corresponding to the video encoder, which converts the encoded feature representation of the video modality into a text modality to obtain a video feature representation.

[0074] S303: Generate prompt information of the large language model based on the multimodal feature representation.

[0075] The prompt information is used to guide the large language model to generate the expected output text content.

[0076] In one example, translation task information can be combined with multimodal feature representation to generate feature information, where the translation task information indicates speech translation. For example, if the translation task information is "translate English into Chinese," the large language model is instructed to translate from English to Chinese. Combining the translation task with the multimodal feature representation instructs the large language model to translate from Chinese to English based on the multimodal feature representation.

[0077] For example, Figure 4 Example of a multimodal speech translation model structure Figure 1 .like Figure 4 As shown, the multimodal speech translation model includes a speech encoder, an adapter corresponding to the speech encoder, an image encoder, an adapter corresponding to the image encoder, a video encoder, an adapter corresponding to the video encoder, and a large language model. In actual application, the speech to be translated is input into the speech encoder, and passes through the speech encoder and the adapter corresponding to the speech encoder to obtain a speech feature representation; the associated image is input into the image encoder, and passes through the image encoder and the adapter corresponding to the image encoder to obtain an image feature representation; the associated video is input into the video encoder, and passes through the video encoder and the adapter corresponding to the video encoder to obtain a video feature representation. The "translate English into Chinese" function, the speech feature representation, the image feature representation, and the video feature representation can be combined to obtain prompt information, which is then input into the large language model to obtain the translation result.

[0078] In another example, the speech translation model includes a combination module, and S303 includes: inputting the multimodal feature representation into the combination module, performing cross-modal interaction between the multimodal features in the combination module to obtain a target feature representation; and generating prompt information according to the target feature representation.

[0079] The combination module is a component used for cross-modal interaction of multimodal features and can be implemented as a neural network. During the training of a multimodal speech translation model, the combination module actively learns the interactive capabilities of multimodal features. Specifically, it learns the interaction between the feature representations of speech, image, and video modalities.

[0080] In this embodiment, when the associated feature representation includes an image feature representation, the speech feature representation and the image feature representation can be input into a combination module, and a cross-modal interaction between the speech feature representation and the image feature representation is performed in the combination module to obtain a target feature representation. When the associated feature representation includes a video feature representation, the speech feature representation and the video feature representation can be input into a combination module, and a cross-modal interaction between the speech feature representation and the video feature representation is performed in the combination module to obtain a target feature representation. When the associated feature representation includes an image feature representation and a video feature representation, the speech feature representation, the image feature representation and the video feature representation can be input into a combination module, and a cross-modal interaction between the speech feature representation, the image feature representation and the video feature representation is performed in the combination module to obtain a target feature representation. Thus, through the cross-modal interaction of multimodal feature representations, the target feature representation contains the association between the multimodal feature representations, thereby improving the accuracy of speech translation.

[0081] Furthermore, prompt information is generated based on the target feature representation, including combining the translation task information with the target feature representation to obtain prompt information, wherein the translation task information instructs the large language model to translate the speech to be translated based on the target feature representation, thereby guiding the large language model to generate the desired output. The process of combining the translation task information with the target feature representation to obtain prompt information can be referred to as the process of combining the translation task information with the multimodal feature representation to obtain prompt information, and will not be further described.

[0082] For example, Figure 5 Example of a multimodal speech translation model structure Figure 2 . Figure 5 The multimodal speech translation model shown is Figure 4 The multimodal speech translation model shown has the same parts, and the contents of these same parts can be referred to Figure 4 The description of , no longer repeat. Figure 4Based on the multimodal speech translation model shown, Figure 5 The multimodal speech translation model also includes a combining module, which can input the speech feature representation, image feature representation and video feature representation into the combining module to obtain the target feature representation, combine "translate English into Chinese" with the target feature representation to obtain prompt information, and input the prompt information into the large language model to obtain the translation result.

[0083] S304: Generate a translation result based on the prompt information at least using a large language model in the multimodal speech translation model.

[0084] In this embodiment, the prompt information may be input into the large language model, and the translation result of the speech to be translated may be generated in the large language model based on the prompt information.

[0085] The translation results may include translated text and / or translated speech, examples of which are provided below:

[0086] In one example, the translation result includes a translated text, and S304 may include: generating the translated text according to the prompt information in the large language model, thereby achieving end-to-end translation from speech to text.

[0087] In this example, the prompt information can be input into the large language model. In the large language model, a translation text of the speech to be translated is generated based on the prompt information. The translation text is the output data of the large language model.

[0088] In another example, the translation result includes translated speech, and the speech translation model also includes a speech generation model. S304 may include: generating translated speech based on the prompt information using the large language model and the speech generation model, wherein the input data of the speech generation model includes hidden layer features generated by the large language model based on the prompt information. This achieves end-to-end speech-to-speech translation.

[0089] Among them, the hidden layer features are feature representations of the hidden layer output of the large language model.

[0090] Optionally, the hidden layer features are feature representations output by the last hidden layer of the large language model. Compared to other hidden layers, the feature representations output by the last hidden layer of the large language model are processed by more hidden layers and are more accurate, which helps improve speech recognition accuracy.

[0091] In this example, prompt information can be input into a large language model. In the large language model, feature processing can be performed based on the prompt information to obtain hidden layer features of the hidden layer output of the large language model. At least based on the hidden layer features, input data for the speech generation model is generated, and the input data is input into the speech generation model for speech generation to obtain translated speech. In traditional speech synthesis methods, speech synthesis is performed based on the translated text of the speech to be translated, or based on the discrete units of the speech to be translated (the basic units obtained by discretizing continuous speech signals or text information). This example uses the hidden layer features of the large language model to replace the translated text and discrete units, which can reduce information loss in the intermediate process and improve the stability and reliability of speech synthesis.

[0092] The above two examples may be executed simultaneously or only one of them may be executed.

[0093] In one possible implementation, the prompt information can be processed by a large language model to obtain hidden layer features generated by the large language model based on the prompt information. The hidden layer features can be input into the classification layer of the large language model, and the output tokens of the large language model can be obtained through the classification layer. Combined with the output tokens of the large language model, the translated text corresponding to the speech to be translated can be obtained. Based on the hidden layer features, input data for the speech generation model can be obtained. This input data is then input into the speech generation model, and the speech generation model generates the translated speech corresponding to the speech to be translated based on the input data. Thus, text and speech are generated based on the hidden layer features of the large speech model, achieving end-to-end speech-to-speech and speech-to-text translation, thereby improving the quality of speech translation.

[0094] In this implementation process, the large language model can obtain hidden layer features in multiple iterations through multiple iterations. In the i-th iteration, the hidden layer features generated in the i-th iteration can be input into the classification layer to obtain the output word unit of the i-th iteration. In this way, the output word units of multiple iterations are combined to obtain the translated text.

[0095] The input data for the speech generation model can refer to the input data of the classification layer mentioned above. Other methods can also be used to achieve better speech synthesis effects. The following provides corresponding examples of other methods:

[0096] In one example, the process of determining input data of a speech generation model includes: predicting hidden layer features belonging to the same semantic scope among hidden layer features generated by a large language model based on prompt information; and determining that the input data includes hidden layer features belonging to the same semantic scope.

[0097] The hidden layer features generated by the large language model based on the prompt information include the hidden layer features generated by the large language model based on the prompt information in multiple iterations.

[0098] The same semantic scope can be divided based on semantic continuity. The same semantic scope corresponds to a continuous semantic segment. For example, within a sentence, the grammatical characteristics can be used to determine which words constitute a continuous semantic segment, or the punctuation marks can be used to determine which words constitute a continuous semantic segment. Hidden layer features belonging to the same semantic scope meet the following two characteristics: they are hidden layer features generated by adjacent iterative processes and are semantically continuous.

[0099] In this example, among the hidden layer features generated by multiple iterations of the large language model, hidden layer features belonging to the same semantic range are predicted, and it is determined that the input data includes hidden layer features belonging to the same semantic range, so that at least one input data can be obtained. Specifically, among the hidden layer features generated by multiple iterations of the large language model, the hidden layer features generated from the i-th iteration to the j-th iteration belong to the same semantic range, and it can be determined that the k-th input data of the speech generation model includes the hidden layer features generated from the i-th iteration to the j-th iteration, where i is greater than or equal to 1, j is greater than or equal to i, and j is less than or equal to M, and k is greater than or equal to 1, and M is the total number of iterations of the large language model.

[0100] For example, among the hidden layer features generated by multiple iterations of a large language model: the hidden layer features generated from the first iteration to the hidden layer features generated from the second iteration belong to the same semantic range, and it can be determined that the first input data of the speech generation model includes the hidden layer features generated from the first iteration to the hidden layer features generated from the second iteration; the hidden layer features generated from the third iteration to the hidden layer features generated from the fifth iteration belong to the same semantic range, and it can be determined that the second input data of the speech generation model includes the hidden layer features generated from the third iteration to the hidden layer features generated from the fifth iteration.

[0101] Compared with inputting the hidden layer features obtained from multiple iterations into the speech generation model one by one independently, this scheme dynamically determines the hidden layer features input into the speech generation model together from the hidden layer features obtained from multiple iterations according to the semantic scope, thereby dynamically capturing the semantic scope of the hidden layer features, improving semantic continuity to a certain extent, and improving the speech generation quality.

[0102] Optionally, predicting hidden features belonging to the same semantic range from hidden features generated by the large language model based on the prompt information includes: identifying hidden features corresponding to semantically disruptive characters from the hidden features generated by the large language model based on the prompt information; and determining hidden features located between hidden features corresponding to semantically disruptive characters as hidden features belonging to the same semantic range. Thus, by identifying hidden features corresponding to semantically disruptive characters, the accuracy of semantic range division of hidden features is improved.

[0103] The semantic interruption characters may be special characters inserted into the training data used in the training process of the multimodal speech translation model. Specifically, they may be inserted into the actual translation text corresponding to the training speech contained in the training data. Through the model training process, the speech translation model may learn the ability to recognize the hidden layer features of the semantic interruption characters.

[0104] In this optional method, among the hidden layer features generated by the large language model based on the prompt information, that is, among the hidden layer features generated by multiple iterations of the large language model, the hidden layer features corresponding to the semantic interruption characters are identified; and the hidden layer features located between the hidden layer features corresponding to the semantic interruption characters are determined to be hidden layer features belonging to the same semantic scope. For example: if the hidden layer feature corresponding to the first semantic interruption character is the hidden layer feature generated by the third iteration, then the hidden layer features generated by the first iteration and the hidden layer features generated by the second iteration can be determined to be hidden layer features belonging to the same semantic scope; if the hidden layer feature corresponding to the second semantic interruption character is the hidden layer feature generated by the sixth iteration, then the hidden layer features generated by the fourth iteration and the hidden layer features generated by the fifth iteration can be determined to be hidden layer features belonging to the same semantic scope.

[0105] Optionally, the speech generation model includes an acoustic feature prediction module and an audio synthesis module. After determining the input data for the speech generation model, the translated speech generation process includes: inputting the input data from the speech generation model into the acoustic feature prediction module, where the acoustic feature prediction module generates acoustic features based on the input data; inputting the acoustic features into the audio synthesis module, where the audio synthesis module performs audio synthesis based on the acoustic features to obtain synthesized audio; and generating the translated speech based on the synthesized audio. This achieves audio synthesis based on acoustic features. Compared to audio synthesis based on text or discrete units, audio synthesis based on acoustic features enables end-to-end speech-to-speech translation, reduces information loss and error accumulation, and improves speech translation quality.

[0106] It can be seen that the input data of the acoustic feature prediction module is feature data (including the hidden layer features of the large language model), and the output data is acoustic features. The audio prediction module can be a neural network for predicting acoustic features based on the hidden layer features of the large language model.

[0107] In this optional method, since there may be multiple input data for speech generation, there may also be multiple synthesized audios generated. The multiple synthesized audios may be spliced ​​together to obtain the translated speech.

[0108] Furthermore, the acoustic features may be mel-spectrogram features. Mel-spectrogram features have advantages in speech synthesis, such as being consistent with human hearing characteristics, requiring minimal computational effort, and being robust to noise. Audio synthesis based on mel-spectrogram features can improve the quality of synthesized audio.

[0109] Furthermore, the audio synthesis module may be a vocoder.

[0110] For example, Figure 6 Example of a multimodal speech translation model structure Figure 3 . Figure 6 The multimodal speech translation model shown is Figure 4 、 Figure 5 There are the same parts, these same parts can refer to Figure 4 and Figure 5 The above description is omitted here. Figure 6 As shown, in addition to the large language model, multiple encoders and adapters corresponding to the multiple encoders, the multimodal speech translation model also includes a speech generation model, which includes an acoustic feature prediction module and a speech synthesis module; through multiple iterations, the hidden layer of the large speech model outputs multiple hidden layer features; the hidden layer features corresponding to the semantic interruption characters are identified among the multiple hidden layer features, and the hidden layer features located between the hidden layer features corresponding to the semantic interruption characters are determined to belong to the same semantic range. The hidden layer features belonging to the same speech range are input into the acoustic feature prediction module as the same input data; the acoustic feature prediction module predicts acoustic features based on the input data, and inputs the predicted acoustic features into the speech synthesis module to obtain translated speech; the multiple hidden layer features can be input into the classification layer of the large speech model to obtain translated text.

[0111] In addition to the hidden layer features of the large language model, the input data of the speech generation model can also include other features, especially features such as the emotion and timbre of the speech to be translated, so as to improve the accuracy of the translated speech generated by the speech generation model in terms of emotion, timbre, etc.

[0112] In one example, the input data of the speech generation model also includes translation-auxiliary features of the speech to be translated. The translation-auxiliary features of the speech to be translated are extracted from the associated video by a video encoder, and / or the translation-auxiliary features of the speech to be translated are extracted from the speech to be translated by a speech encoder. The translation-auxiliary features include emotional features and / or speaking object features. Thus, by extracting emotional features and / or speaking object features from the associated video and / or speech to be translated, and inputting the emotional features and / or speaking object features as part of the speech generation model, the accuracy of the translated speech generated by the speech generation model in terms of emotion, sound characteristics, etc. is improved.

[0113] Among them, the speaker characteristics reflect the speaker's age, personality, timbre, range, intonation and other characteristics related to voice characteristics.

[0114] The video encoder and / or the speech encoder may be trained in advance so that the video encoder and / or the speech encoder have the ability to extract emotional features and / or speaker features.

[0115] In this example, a first emotion feature and / or a first speaking object feature are extracted from an associated video by a video encoder, and / or a second emotion feature and / or a second speaking object feature are extracted from the speech to be translated by a speech encoder; input data for a speech generation model can be obtained by combining at least one of the first emotion feature, the first speaking object feature, the second emotion feature and / or the second speaking object feature, and hidden layer features belonging to the same speech range.

[0116] Optionally, the first emotion feature and the first speaker feature are feature representations output by different network layers of the video encoder. The second emotion feature and the second speaker feature are feature representations output by different network layers of the speech encoder. For the video encoder and speech encoder, the feature representations output by different network layers are feature representations at different scales. Some feature representations at certain scales can more clearly represent emotion features, while others can more clearly represent speaker features. Therefore, the emotion features and speaker features are obtained from different network layers.

[0117] For example, Figure 7 Example of a multimodal speech translation model structure Figure 4 . Figure 7 The multimodal speech translation model shown is Figures 4 to 6 There are identical parts, these identical parts can refer to Figures 4 to 6 The above description is omitted here. Figure 7As shown, the emotional features and speaker features obtained by extracting features from the associated video through the video encoder, the emotional features and speaker features obtained by extracting features from the to-be-translated speech through the speech encoder, and the hidden layer features belonging to the same semantic range output by the large language model are input into the speech generation model together to generate the translated speech.

[0118] In an embodiment of the present application, a multimodal feature extraction module is used to extract features from the speech to be translated and the associated data to obtain a multimodal feature representation; based on the multimodal feature representation, prompt information of a large language model is generated; and based on the prompt information, a translation result is generated at least by the large language model. Thus, by combining the multimodal feature information and leveraging the large model capabilities of the large language model, the quality of speech translation is improved. In particular, the hidden layer features of the large language model can be further used in the speech generation process of the speech generation model, and the emotional features and speaker object features can also be used in the speech generation process of the speech generation model, thereby improving the quality of speech-to-speech translation from multiple aspects.

[0119] Below, corresponding embodiments are provided for the training process of a multimodal speech translation model.

[0120] See also Figure 8 In an exemplary embodiment, a flowchart illustrating a training process of a multimodal speech translation model is provided. The training process may include:

[0121] S801. Obtain first training data. The first training data includes a first training speech, an associated image of the first training speech, an associated video of the first training speech, and a true translation result of the first training speech. The associated image of the first training speech includes image information of at least part of an object described by the first training speech, and the associated video of the first training speech displays lip movements of the speaker when expressing the first training speech.

[0122] S802: Perform speech translation training on a multimodal speech translation model based on first training data.

[0123] The multimodal speech translation model may refer to the description of the aforementioned embodiment.

[0124] In this embodiment, the first training data is training data including at least three modalities: speech modality, video modality and image modality. Through the multimodal first training data, supervised speech translation training is performed on the multimodal speech translation model, so that the speech translation model learns the ability to perform end-to-end speech translation based on multimodal data, thereby improving the quality of speech translation.

[0125] When the multimodal speech translation model includes a speech encoder, an adapter corresponding to the speech encoder, an image encoder, an adapter corresponding to the image encoder, a video encoder, an adapter corresponding to the video encoder, and a large language model, the speech encoder and the adapter corresponding to the speech encoder can perform feature encoding and feature conversion on the first training speech to obtain a speech feature representation corresponding to the first training speech; the image encoder and the adapter corresponding to the image encoder can perform feature encoding and feature conversion on an image associated with the first training speech to obtain an image feature representation corresponding to the associated image; the video encoder and the adapter corresponding to the video encoder can perform feature encoding and feature conversion on a video associated with the first training speech to obtain a video feature representation corresponding to the associated video; training prompt information is generated based on the speech feature representation, the image feature representation, the video feature representation, and the actual translation result of the first training speech; based on the training prompt information, the first training speech is speech translated using the large language model to obtain a predicted translation result of the first training speech; a training error is determined based on the actual translation result of the first training speech and the predicted translation result of the first training speech; and parameters of the multimodal speech translation model are adjusted based on the training error.

[0126] Optionally, before training the multimodal speech translation model, which includes a large language model, based on the first training data, the method further includes: obtaining second training data, the second training data including second training speech and actual translation results of the second training speech; and training the large language model based on the second training data. Therefore, given that multimodal training data is difficult to collect and relatively small in volume, while single-modal training data is easier to collect, the large language model can be trained first using the massive amount of second training data to improve its speech translation capabilities.

[0127] The actual translation results of the second training speech include the actual translated text of the second training speech. Speech-to-text translation training data collection is relatively easy, facilitating the acquisition of massive amounts of secondary training data. Large language models possess powerful comprehensive capabilities, including reasoning, writing, conversational, coding, and long text processing. However, their translation functions lack data coverage and adaptation, leaving room for improvement in translation quality. Therefore, through massive amounts of secondary training data, the underlying translation capabilities of large language models are strengthened, thereby enhancing the speech translation capabilities of multimodal speech translation models.

[0128] Optionally, before training the speech translation model based on the first training data, the method further includes: obtaining third training data, the third training data including a third training speech and a real translation result of the third training speech, and the third training data also including an associated image of the third training speech or an associated video of the third training data; and training the speech translation model based on the third training data. As can be seen, the third training data includes two modalities. Compared to the first training data, the third training data is easier to collect and a larger amount of third training data can be obtained. This larger amount of third training data is used to train the speech translation model's ability to perform speech translation based on data from both modalities, thereby improving the speech translation model's speech translation capabilities.

[0129] When the multimodal speech translation model includes a speech encoder, an adapter corresponding to the speech encoder, an image encoder, an adapter corresponding to the image encoder, a video encoder, an adapter corresponding to the video encoder, and a large language model, the speech encoder, the adapter corresponding to the speech encoder, the image encoder, the adapter corresponding to the image encoder, and the large language model can be trained using a third training speech and an associated image of the third training speech; alternatively, the speech encoder, the adapter corresponding to the speech encoder, the video encoder, the adapter corresponding to the video encoder, and the large language model can be trained using a third training speech and an associated video of the third training speech.

[0130] Optionally, before training the large language model based on the second training data, a speech encoder can be obtained from the trained speech recognition model, and the speech encoder can be used in the multimodal speech translation model; an image encoder can be obtained from the trained image recognition model, and the image encoder can be used in the multimodal speech translation model; a video encoder can be obtained from the trained video recognition model, and the video encoder can be used in the multimodal speech translation model. Thus, the trained speech encoder, image encoder and video encoder can be used in the multimodal speech translation model. The subsequent training process only needs to fine-tune these encoders to obtain an encoder suitable for the multimodal speech translation process, thereby reducing the training pressure of the multimodal speech translation model.

[0131] By referring to the encoder acquisition process described above, the adapters corresponding to the speech encoder, the image encoder, and the video encoder can be obtained from the trained neural network model. After fine-tuning in the subsequent training process, these adapters can be applied to the multimodal speech translation process, reducing the training pressure of the multimodal speech translation model.

[0132] Optionally, when the real translation result includes real translated text and real translated speech, a multimodal speech translation model may be first trained based on first training data including the first training speech, its associated images, its associated videos, and the real translated text of the first training speech. The multimodal speech translation model may then be trained based on the first training data including the first training speech, its associated images, its associated videos, and the real translated speech of the first training speech. This staged training can improve the model training effect.

[0133] Exemplary devices

[0134] Correspondingly, an embodiment of the present application also provides a speech translation device.

[0135] See also Figure 9 In an exemplary embodiment, a speech translation device 900 is provided, which includes an acquisition unit 901 and a translation unit 902.

[0136] An acquisition unit 901 is configured to acquire the speech to be translated and associated data of the speech to be translated, wherein the associated data includes associated images and / or associated videos, wherein the associated images include image information of at least a portion of the object described by the speech to be translated, and the associated videos display the lip movements of the speaker when expressing the speech to be translated;

[0137] The translation unit 902 is configured to translate the speech to be translated based on the speech to be translated and the associated data using a multimodal speech translation model to obtain a translation result of the speech to be translated.

[0138] In some embodiments, the speech translation model includes a multimodal feature extraction module and a large language model. The translation unit 902 is specifically used to: perform feature extraction on the speech to be translated and related data through the multimodal feature extraction module to obtain a multimodal feature representation; generate prompt information of the large language model based on the multimodal feature representation; and generate a translation result based on the prompt information at least through the large language model.

[0139] In some embodiments, the multimodal feature extraction module includes a speech encoder, an adapter corresponding to the speech encoder, an image encoder, an adapter corresponding to the image encoder, a video encoder, and an adapter corresponding to the video encoder. The translation unit 902 is specifically used to: perform feature encoding and feature conversion on the speech to be translated through the speech encoder and the adapter corresponding to the speech encoder to obtain a speech feature representation; perform feature encoding and feature conversion on the associated image through the image encoder and the adapter corresponding to the image encoder to obtain an image feature representation, and / or, perform feature encoding and feature conversion on the associated video through the video encoder and the adapter corresponding to the video encoder to obtain a video feature representation; the multimodal features include speech feature representation and associated feature representation, and the associated features include image feature representation and / or video feature representation.

[0140] In some embodiments, the speech translation model includes a combination module, and the translation unit 902 is specifically used to: input the multimodal feature representation into the combination module, perform cross-modal interaction between the multimodal features in the combination module to obtain the target feature representation; and generate prompt information based on the target feature representation.

[0141] In some embodiments, the translation unit 902 is specifically configured to combine the translation task information with the target feature representation to obtain prompt information, where the translation task information indicates to perform speech translation.

[0142] In some embodiments, the translation result includes translated text, and the translation unit 902 is specifically configured to generate the translated text in the large language model based on the prompt information. Alternatively, the translation result includes translated speech, and the speech translation model further includes a speech generation model. The translation unit 902 is specifically configured to generate the translated speech based on the prompt information using the large language model and the speech generation model, wherein the input data of the speech generation model includes hidden layer features generated by the large language model based on the prompt information.

[0143] In some embodiments, the process of determining the input data of the speech generation model includes: predicting hidden layer features belonging to the same semantic scope among the hidden layer features generated by the large language model based on prompt information; and determining that the input data includes hidden layer features belonging to the same semantic scope.

[0144] In some embodiments, predicting hidden layer features belonging to the same semantic scope among hidden layer features generated by the large language model based on prompt information includes: identifying hidden layer features corresponding to semantic interruption characters among the hidden layer features generated by the large language model based on prompt information; and determining hidden layer features located between hidden layer features corresponding to semantic interruption characters as hidden layer features belonging to the same semantic scope.

[0145] In some embodiments, the input data further includes translation-assisting features of the speech to be translated, where the translation-assisting features are extracted from the associated video by a video encoder, and / or the translation-assisting features are extracted from the speech to be translated by a speech encoder; the translation-assisting features include emotional features and / or speaking object features.

[0146] In some embodiments, the speech generation model includes an acoustic feature prediction module and an audio synthesis module. After determining the input data, the process of generating the translated speech includes: inputting the input data into the acoustic feature prediction module, generating acoustic features based on the input data in the acoustic feature prediction module; inputting the acoustic features into the audio synthesis module, performing audio synthesis based on the acoustic features in the audio synthesis module to obtain synthesized audio; and generating the translated speech based on the synthesized audio.

[0147] In some embodiments, the training process of the speech translation model includes: obtaining first training data, the first training data including a first training speech, an associated image of the first training speech, an associated video of the first training speech, and a real translation result of the first training speech, the associated image of the first training speech contains image information of at least part of the object described by the first training speech, and the associated video of the first training speech shows the lip movements of the speaking object when expressing the first training speech; based on the first training data, performing speech translation training on the speech translation model.

[0148] In some embodiments, the speech translation model includes a large language model. Before training the speech translation model based on the first training data, the method further includes: obtaining second training data, the second training data including a second training speech and a real translation result of the second training speech; and training the large language model based on the second training data.

[0149] In some embodiments, before training the speech translation model based on the first training data, the method further includes: obtaining third training data, the third training data including a third training speech and a real translation result of the third training speech, the third training data also including an associated image of the third training speech or an associated video of the third training data; and training the speech translation model based on the third training data.

[0150] The speech translation device 900 provided in this embodiment is based on the same concept as the speech translation method provided in the aforementioned embodiments of this application. It can execute the speech translation method provided in any of the aforementioned embodiments of this application and has the corresponding functional modules and beneficial effects. For technical details not fully described in this embodiment, please refer to the specific processing content of the speech translation method provided in the aforementioned embodiments of this application and will not be repeated here.

[0151] The functions implemented by each unit in the above device can be implemented by the same or different processors, which is not limited in the embodiments of the present application.

[0152] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and the memory stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device. The processor can be a general-purpose processor, such as a CPU or a microprocessor, and the memory can be a memory within the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits. The functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units can be realized by designing the logical relationships between the components within the circuit. For another example, in another implementation, the hardware circuit can be implemented by a PLD. For example, an FPGA can include a large number of logic gate circuits. The connection relationships between the logic gate circuits are configured through a configuration file to realize the functions of some or all of the above units. All units of the above devices can be implemented entirely in the form of a processor calling software, or entirely in the form of hardware circuits, or partially in the form of a processor calling software, with the remaining parts implemented in the form of hardware circuits.

[0153] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and execute instructions, such as a CPU, a microprocessor, a GPU, or a DSP. In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0154] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0155] In addition, the various units in the above apparatus may be fully or partially integrated together, or may be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the various units of the apparatus. The at least one processor may be of different types, such as a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0156] Exemplary electronic devices

[0157] Another embodiment of the present application also provides an electronic device. Figure 10 As shown, the electronic device may include: a memory 1000 and a processor 1010; wherein the memory 1000 is connected to the processor 1010 and is used to store programs; the processor 1010 is used to implement the speech translation method disclosed in any of the above embodiments by running the program stored in the memory 1000.

[0158] Specifically, the electronic device may further include: a bus, a communication interface 1020 , an input device 1030 and an output device 1040 .

[0159] The processor 1010, the memory 1000, the communication interface 1020, the input device 1030 and the output device 1040 are interconnected via a bus.

[0160] A bus may include a pathway that transfers information between components of a computer system.

[0161] Processor 1010 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.

[0162] The processor 1010 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0163] The memory 1000 stores a program for executing the technical solution of the present application, and may also store an operating system and other key services. Specifically, the program may include program code, and the program code may include computer operating instructions. More specifically, the memory 1000 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, disk storage, flash, etc.

[0164] The input device 1030 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0165] Output device 1040 may include devices that allow information to be output to a user, such as a display screen, printer, speakers, etc.

[0166] The communication interface 1020 may include any transceiver or similar device for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0167] The processor 1010 executes the program stored in the memory 1000 and calls other devices, and can be used to implement each step of any speech translation method provided in the above embodiments of the present application.

[0168] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute any one of the speech translation methods provided in the above embodiments. The specific processing process and its beneficial effects can be found in the embodiment introduction of the above-mentioned speech translation method.

[0169] Exemplary computer program products and storage media

[0170] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the speech translation method according to various embodiments of the present application described in any of the above-mentioned embodiments of this specification.

[0171] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0172] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is used by a processor to execute the steps of the speech translation method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0173] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0174] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0175] The steps in the methods of each embodiment of the present application can be adjusted in sequence, merged, and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0176] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be merged, divided, and deleted according to actual needs.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or submodules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0178] The modules or submodules described as separate components may or may not be physically separate, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules may be selected to achieve the purpose of this embodiment according to actual needs.

[0179] In addition, each functional module or submodule in each embodiment of the present application may be integrated into a processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into a single module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or software functional modules or submodules.

[0180] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software units executed by a processor, or a combination of the two. The software units may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0182] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0183] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A speech translation method, characterized in that: include: Acquiring the speech to be translated and associated data of the speech to be translated, the associated data including an associated image and / or an associated video, the associated image containing image information of at least a portion of the object described by the speech to be translated, and the associated video showing lip movements of the speaker when expressing the speech to be translated; Based on the speech to be translated and the associated data, the speech to be translated is translated by a multimodal speech translation model to obtain a translation result of the speech to be translated.

2. The speech translation method according to claim 1, wherein The speech translation model includes a multimodal feature extraction module and a large language model. Based on the speech to be translated and the associated data, the speech to be translated is translated by the multimodal speech translation model to obtain a translation result of the speech to be translated, including: Performing feature extraction on the speech to be translated and the associated data by the multimodal feature extraction module to obtain a multimodal feature representation; generating prompt information of the large language model according to the multimodal feature representation; According to the prompt information, the translation result is generated at least by using the large language model.

3. The speech translation method according to claim 2, wherein: The multimodal feature extraction module includes a speech encoder, an adapter corresponding to the speech encoder, an image encoder, an adapter corresponding to the image encoder, a video encoder, and an adapter corresponding to the video encoder. The multimodal feature extraction module extracts features from the speech to be translated and the associated data to obtain a multimodal feature representation, including: Performing feature encoding and feature conversion on the speech to be translated by the speech encoder and an adapter corresponding to the speech encoder to obtain a speech feature representation; performing feature encoding and feature conversion on the associated image by using the image encoder and an adapter corresponding to the image encoder to obtain an image feature representation, and / or performing feature encoding and feature conversion on the associated video by using the video encoder and an adapter corresponding to the video encoder to obtain a video feature representation; The multimodal features include the speech feature representation and an associated feature representation, and the associated features include the image feature representation and / or the video feature representation.

4. The speech translation method according to claim 2, wherein: The speech translation model includes a combining module, which generates prompt information of the large language model based on the multimodal feature representation, including: Inputting the multimodal feature representation into the combining module, performing cross-modal interaction between the multimodal features in the combining module, and obtaining a target feature representation; The prompt information is generated according to the target feature representation.

5. The speech translation method according to claim 4, wherein: Generating the prompt information according to the target feature representation includes: The prompt information is obtained by combining translation task information with the target feature representation, wherein the translation task information indicates to perform speech translation.

6. The speech translation method according to any one of claims 2 to 5, characterized in that: The translation result includes a translated text, and generating the translation result at least by the large language model according to the prompt information includes: In the large language model, generating the translation text according to the prompt information; And / or, the translation result includes translated speech, the speech translation model further includes a speech generation model, and generating the translation result at least by the large language model according to the prompt information includes: Based on the prompt information, the translated speech is generated by the large language model and the speech generation model, wherein input data of the speech generation model includes hidden layer features generated by the large language model based on the prompt information.

7. The speech translation method according to claim 6, wherein: The process of determining the input data of the speech generation model includes: Predicting hidden layer features belonging to the same semantic range among the hidden layer features generated by the large language model based on the prompt information; It is determined that the input data includes the hidden layer features belonging to the same semantic scope.

8. The speech translation method according to claim 7, wherein: The step of predicting hidden layer features belonging to the same semantic range among the hidden layer features generated by the large language model based on the prompt information includes: Identifying hidden layer features corresponding to semantic interruption characters from hidden layer features generated by the large language model based on the prompt information; The hidden layer features located between the hidden layer features corresponding to the semantic interruption characters are determined as the hidden layer features belonging to the same semantic range.

9. The speech translation method according to claim 6, wherein: The input data further includes translation-assisting features of the speech to be translated, wherein the translation-assisting features are extracted from the associated video by a video encoder, and / or the translation-assisting features are extracted from the speech to be translated by a speech encoder; The translation-assisting features include sentiment features and / or speaking object features.

10. The speech translation method according to claim 6, wherein: The speech generation model includes an acoustic feature prediction module and an audio synthesis module. After determining the input data, the generation process of the translated speech includes: Inputting the input data into the acoustic feature prediction module, wherein the acoustic feature prediction module generates acoustic features based on the input data; Inputting the acoustic features into the audio synthesis module, wherein the audio synthesis module performs audio synthesis based on the acoustic features to obtain synthesized audio; The translated speech is generated according to the synthesized audio.

11. The speech translation method according to any one of claims 1 to 5, characterized in that: The training process of the speech translation model includes: Obtaining first training data, where the first training data includes a first training speech, an image associated with the first training speech, a video associated with the first training speech, and a real translation result of the first training speech, wherein the image associated with the first training speech includes image information of at least a portion of an object described by the first training speech, and the video associated with the first training speech shows lip movements of the speaker when expressing the first training speech; Based on the first training data, speech translation training is performed on the speech translation model.

12. The speech translation method according to claim 11, wherein: The speech translation model includes a large language model. Before training the speech translation model based on the first training data, the method further includes: Acquire second training data, where the second training data includes a second training speech and a true translation result of the second training speech; The large language model is trained according to the second training data.

13. The speech translation method according to claim 11, wherein: Before training the speech translation model based on the first training data, the method further includes: Acquire third training data, where the third training data includes a third training speech and a true translation result of the third training speech, and the third training data also includes an associated image of the third training speech or an associated video of the third training data; The speech translation model is trained according to the third training data.

14. A speech translation device, characterized in that: include: an acquisition unit, configured to acquire the speech to be translated and associated data of the speech to be translated, wherein the associated data includes an associated image and / or an associated video, wherein the associated image includes image information of at least a portion of the object described by the speech to be translated, and the associated video displays lip movements of the speaker when expressing the speech to be translated; The translation unit is configured to translate the speech to be translated based on the speech to be translated and the associated data by using a multimodal speech translation model to obtain a translation result of the speech to be translated.

15. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the speech translation method according to any one of claims 1 to 13 by running the program in the memory.

16. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the speech translation method according to any one of claims 1 to 13.

Images