Speech translation method and related apparatus, device and storage medium

Abstract

The application discloses a speech translation method and related device, equipment and storage medium, wherein the speech translation method comprises: extracting speech features of to-be-translated speech and text features of prompt text; wherein the prompt text is used to indicate translation from a source language of the to-be-translated speech to a target language; obtaining mapped features of the speech features after feature dimension mapping, inputting the text features and the mapped features into a large model, and obtaining first output features in a processing process of the large model; wherein the mapped features and the first output features have the same feature dimension; obtaining first fusion features based on fusion of the first output features and the mapped features; obtaining target text output by the large model continuing to process the first fusion features; wherein the target text is translated text of the to-be-translated speech translated into the target language. The above scheme can reduce the consumption resources and use complexity of multi-language speech translation as much as possible, and improve translation accuracy.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a speech translation method and related apparatus, devices and storage media. Background Technology

[0002] Speech translation (ST) aims to convert spoken language into text in the target language. Speech translation is applied in all aspects of life and work, such as outbound tourism, international conferences, and business entertainment.

[0003] Currently, most existing speech translation strategies require separate modeling of the mapping from each source language to the target language. These methods are not only resource-intensive but also increase the complexity for users. Furthermore, for languages ​​with insufficient corpora, speech translation systems trained through separate modeling may suffer from insufficient translation accuracy. Therefore, how to minimize the resource consumption and complexity of multilingual speech translation while improving translation accuracy has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a speech translation method and related apparatus, equipment, and storage medium that can minimize the resource consumption and complexity of multilingual speech translation and improve translation accuracy.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a speech translation method, comprising: extracting speech features of the speech to be translated and extracting text features of a prompt text; wherein the prompt text is used to indicate the translation from the source language of the speech to be translated into the target language; obtaining a mapping feature after mapping the speech features through feature dimensions, and inputting the text features and the mapping feature into a large model, and obtaining a first output feature during the processing of the large model; wherein the mapping feature and the first output feature have the same feature dimension; obtaining a first fusion feature based on the fusion of the first output feature and the mapping feature; obtaining the target text output by the large model further processing the first fusion feature; wherein the target text is the translated text of the speech to be translated into the target language.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a speech translation device, comprising: a feature extraction module, a feature mapping module, a first processing module, a feature fusion module, and a second processing module. The feature extraction module is used to extract speech features of the speech to be translated and to extract text features of a prompt text; wherein the prompt text is used to instruct the speech to be translated from a source language to a target language. The feature mapping module is used to obtain mapped features of the speech features after feature dimension mapping. The first processing module is used to input the text features and the mapped features into a large model, and to obtain a first output feature during the processing of the large model; wherein the mapped features and the first output features have the same feature dimension. The feature fusion module is used to obtain a first fused feature based on the fusion of the first output features and the mapped features. The second processing module is used to obtain the target text output by the large model after further processing of the first fused feature; wherein the target text is the translated text of the speech to be translated into the target language.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device including a memory and a processor coupled to each other. The memory stores program instructions, and the processor executes the program instructions to implement the speech translation method described in the first aspect.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the speech translation method of the first aspect described above.

[0009] The above scheme extracts the speech features of the speech to be translated and the text features of the prompt text, which is used to indicate the translation from the source language to the target language. It then obtains the mapped features of the speech features after feature dimension mapping, and inputs the text features and mapped features into a large model. It also obtains the first output feature of the large model during processing, where the mapped feature and the first output feature have the same feature dimension. This yields a first fusion feature based on the fusion of the first output feature and the mapped feature. The large model then further processes the first fusion feature to obtain the target text output, which is the translated text from the speech to the target language. On the one hand, by fully utilizing the large model's general multilingual translation capabilities and aligning the speech features with the large model input through feature mapping, and by adding a fusion operation between the first output feature and the mapped feature in the large model, the large model can fully utilize semantic knowledge to generate the target text in the target language without separate modeling. This minimizes the resource consumption and translation accuracy of multilingual speech translation. On the other hand, for the user, only the speech to be translated and the prompt text indicating the translation from the source language to the target language need to be input, effectively reducing the complexity of use. Therefore, it can minimize the resource consumption and complexity of multilingual speech translation, and improve translation accuracy. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the speech translation method of this application;

[0011] Figure 2 This is a schematic diagram of the framework of an embodiment of a speech translation model;

[0012] Figure 3 This is a schematic diagram of the framework of an embodiment of a converged network;

[0013] Figure 4 This is a flowchart illustrating an embodiment of the training method for the speech translation model of this application;

[0014] Figure 5 This is a schematic diagram of the framework of an embodiment of the speech translation device of this application;

[0015] Figure 6 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0016] Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0017] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0018] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures, interfaces, and technologies are presented to provide a thorough understanding of the present application.

[0019] The terms "system" and "network" in this document are often used interchangeably herein. The term "and / or" in this document is merely an association relationship describing associated objects, indicating that three relationships may exist, for example, A and / or B may represent: the presence of A alone, the simultaneous presence of A and B, and the presence of B alone. In addition, the slash " / " in this document generally indicates that the associated objects before and after are in an "or" relationship. Furthermore, "plurality" in this document means two or more than two.

[0020] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of an embodiment of the speech translation method of the present application.

[0021] Specifically, the following steps may be included:

[0022] Step S11: Extract the speech feature of the to-be-translated speech, and extract the text feature of the prompt text.

[0023] In the embodiments of the present disclosure, the prompt text is used to instruct translation from the source language of the to-be-translated speech to the target language. Exemplarily, taking the to-be-translated speech "good morning" as an example, the prompt text may be "translate Chinese speech into English", that is, the source language is Chinese and the target language is English at this time; or, taking the to-be-translated speech "good morning" as an example, the prompt text may be "translate English audio into Chinese". Other situations can be deduced by analogy, and no more examples will be given one by one here. In particular, the prompt text and the to-be-translated speech may belong to the same language (i.e., the source language), so as to improve the consistency between the input multimodal data (i.e., speech and text), thereby helping to improve the accuracy of speech translation. Certainly, the prompt text may also not belong to the same language as the to-be-translated speech, which is not limited herein. In addition, the prompt text may be specifically described in the form of "translate [source language] speech into [target language]" (i.e., "translate [source] audio into [target]"), so as to provide relevant information of the source language and the target language. Certainly, the above description form is only a possible implementation manner in the actual application process, and does not limit the description form of the prompt text accordingly.

[0024] In one implementation scenario, the speech to be translated can be acquired in real time. For example, in translation scenarios such as international conferences, speech data can be acquired in real time and used as the speech to be translated. Alternatively, the speech to be translated can also be acquired non-real time. For example, in translation scenarios such as language learning, speech data can be obtained from the Internet or other means and used as the speech to be translated. This disclosure does not specifically limit the application scenarios of speech translation.

[0025] In one implementation scenario, the acoustic features (e.g., MFCC, FBank, etc.) of each audio frame of the speech to be translated can be extracted as the speech features of the speech to be translated. For example, taking a speech to be translated containing N audio frames as an example, its speech features can be represented as H = {h1, h2, ..., h...} N}, where h i This represents the speech features of the i-th audio frame.

[0026] In another implementation scenario, unlike the aforementioned methods of extracting speech features, to further improve the accuracy of speech translation, an encoding network pre-trained with multilingual sample speech can be obtained. This encoding network is then used to encode the speech to be translated, yielding its speech features. For example, the encoding network can include, but is not limited to, mHuBERT, etc., and the network structure is not limited here. It should be noted that mHuBERT is a multilingual version of HuBERT. HuBERT is a self-supervised model that predicts offline labels for masked audio segments for model learning through K-Means clustering applied to the model's intermediate representations. Specifically, it combines one-dimensional convolutional layers and a Transformer encoder to encode speech into continuous intermediate representations, which are then further converted into speech representations by K-Means. Furthermore, in this embodiment, the parameters of the encoding network can be set to be trainable to quickly align the output speech representations with the input of a large model. The above method obtains the encoding network pre-trained with multilingual sample speech, and encodes the speech to be translated based on the encoding network to obtain the speech features of the speech to be translated. This can further improve the general processing capability for multiple languages ​​and help improve the accuracy of speech translation.

[0027] In one implementation scenario, the text features of the prompt text can be extracted using word vector tools such as embedding and word2vec, which are not limited here. For ease of description, taking a prompt text containing M characters as an example, the text features can be represented as T={t1,t2,…,t…} M}

[0028] Step S12: Obtain the mapped features of the speech features after feature dimension mapping, and input the text features and mapped features into the large model, and obtain the first output features in the processing of the large model.

[0029] In this embodiment, the mapped features and the first output features have the same feature dimension. That is, the mapped features and the speech features essentially have the same or similar feature information; the main difference lies in bridging the difference between the input parameters of the large model and the mapped features, where the feature dimension of the mapped features is consistent with the feature dimension of the large model. For example, to facilitate feature mapping, a mapping network can be used to map the speech features to obtain the mapped features. It should be noted that the mapping network may include, but is not limited to, convolutional layers, fully connected layers, etc., and this embodiment does not limit the network structure of the mapping network. For ease of description, let's take the example of the speech to be translated containing N audio frames, where the speech features H = {h1, h2, ..., h...} N The mapping feature of} can be represented as Z = {z1, z2, ..., z} N}

[0030] In one implementation scenario, the Large Language Model (LLM) can include, but is not limited to, GPT, LLaMA, etc., without further limitation. Taking LLaMA as an example, the large model can include several network layers, which can be referred to as LLaMA layers for ease of description. Other cases can be deduced similarly, and will not be listed here. That is, the mapped features Z of the speech features H and the text features T can be used as inputs to the large model to obtain the first output features in the large model processing. For ease of description, the network layer that outputs the first output feature in the large model processing can be called the first network layer. Specifically, the first network layer can be any network layer before the last network layer in the large model. For ease of description, the first output feature can be denoted as X. llm .

[0031] In one implementation scenario, the first network layer can be the first network layer of a large model, the second network layer, the third network layer, etc., without limitation. Taking a large model containing L network layers as an example, the first network layer can be any network layer from the 1st to the (L-1)th network layer, without limitation.

[0032] Step S13: Obtain the first fused feature based on the fusion of the first output feature and the mapping feature.

[0033] In one implementation scenario, to reduce algorithm complexity, the first output feature and the mapped feature can be directly fused through concatenation, addition, weighting, averaging, and other fusion operations to achieve feature fusion and obtain the first fused feature. It should be noted that fusing the first output feature and the mapped feature effectively integrates semantic and speech feature information, which helps improve speech translation accuracy.

[0034] In one implementation scenario, as another possible approach, the first output feature and the mapping feature can be processed using an attention mechanism to obtain a second output feature with the same feature dimension as the first output feature. The first and second output features are then fused to obtain a second fused feature. This second fused feature is then subjected to feedforward processing to obtain a third output feature. Finally, the second and third fused features can be fused to obtain a first fused feature. This method, through the attention mechanism, fusion operation, and feedforward processing, can better extract semantic information from speech, thus contributing to further improvements in speech translation accuracy.

[0035] In a specific implementation scenario, the key and value features of the attention mechanism can be obtained based on the mapping features, and the query features of the attention mechanism can be obtained based on the first output features. Therefore, the second output feature can be obtained based on the query feature, key feature, and value feature. For ease of description, the key feature can be denoted as k, the value feature as v, and the query feature as q. The second output feature can then be represented as attention(q,k,v). The specific processing steps can be found in the details of the cross-attention mechanism, and will not be elaborated upon here.

[0036] In a specific implementation scenario, to facilitate data processing, after obtaining the second output feature, it can be activated first. For example, activation functions such as tanh can be used, without limitation. Based on this, the first output feature and the activated second output feature are then fused to obtain the second fused feature Y. pre :

[0037] Y pre =X llm +tanh(attention(q,k,v))……(1)

[0038] In a specific implementation scenario, after performing feedforward processing on the second fused feature, activation processing can be performed first, followed by fusion with the second fused feature to obtain the first output feature Y:

[0039] Y = Y pre +tanh(ffn(Y pre ))……(2)

[0040] In the above formula (2), tanh denotes the activation function, ffn denotes the feed-forward processing, and ffn(Y pre ) denotes the third output feature.

[0041] It should be noted that the above exemplified manner is merely a possible implementation of the fusion operation in the actual application process, and the fusion operation of the first output feature and the mapped feature is not limited to the implementation described in the embodiments of the present disclosure.

[0042] Step S14: Obtain the target text output by the large model after continuing to process the first fused feature.

[0043] In the embodiments of the present disclosure, the target text is a translated text obtained by translating the to-be-translated speech into a target language. Still taking the to-be-translated speech "good morning" as an example, when the prompt text is "translate Chinese speech into English", the target text may be the translated text "good morning" obtained by translating the above to-be-translated speech into English; or, taking the to-be-translated speech "good morning" as an example, when the prompt text is "translate English audio into Chinese", the target text may be the translated text "good morning" obtained by translating the above to-be-translated speech into Chinese. Other situations can be deduced by analogy, and no enumeration will be given here. In addition, after the first fused feature is obtained, the large model may continue the processing starting from the position where the large model outputs the first output feature in the previous steps.

[0044] In one implementation scenario, the first fused feature may specifically be processed by a second network layer located after the aforementioned first network layer in the large model. Taking the first network layer as the i-th network layer in the large model as an example, the second network layer is the network layers from the (i+1)-th network layer to the L-th network layer in the large model. Therefore, after the first fused feature is obtained, the first fused feature may be input to the (i+1)-th network layer in the large model, and processed sequentially by this network layer and each subsequent network layer, so as to obtain the target text output by the large model.

[0045] In another implementation scenario, as another possible approach, to extract semantic information from speech at different scales and better utilize the rich multilingual semantic knowledge in the large model to generate translated text in the target language, it is also possible to detect whether the number of feature fusion iterations is less than a preset number. Of course, the preset number can be set according to application needs. For example, when high translation accuracy is required, the preset number can be set slightly higher, while when translation accuracy is relatively less critical, the preset number can be set slightly lower; no limitation is made here. Based on this, in response to the feature fusion iterations being less than the preset number, the step of obtaining the first output features in the large model processing can be repeatedly executed until the feature fusion iterations are no less than the preset number, thus obtaining the final target text output by the large model. The above method, by detecting whether the feature fusion iterations are less than the preset number and re-executing the series of steps when they are less than the preset number, can extract semantic information from speech at different scales to better utilize the rich multilingual semantic knowledge in the large model to generate translated text in the target language.

[0046] In a specific implementation scenario, taking a large model containing L network layers, with the i-th layer as the first network layer, if the preset number of iterations is set to 2, after obtaining the first fused feature, we can check if the feature fusion count is less than the preset number. Since the feature fusion count is 1, it is less than the preset number. Therefore, we can select the network layer following the i-th network layer in the large model (i.e., the (i+1)-th network layer) as the new first network layer, and re-obtain the first output feature of the first network layer in the large model (i.e., the new first network layer). It should be noted that, for the new first network layer, its input is the latest first fused feature. Based on this, we can then fuse the first output feature of the (i+1)-th network layer with the mapping feature to obtain a new first fused feature. Since the feature fusion count is 2, which is not less than the preset number, we can obtain the second network layer after the (i+2)-th network layer (i.e., the (i+2)-th to the L-th network layers) and process them sequentially to finally output the target text. Other cases can be deduced similarly, and will not be listed here.

[0047] As one possible implementation, to improve the efficiency of speech translation, the translated text can also be obtained by a speech translation model processing the speech to be translated and the prompt text. Specifically, the speech translation model can include a large model, an encoding network, a mapping network, and a fusion network. During training, the network parameters of the large model are fixed, i.e., the network parameters of the large model are frozen. This helps to fully utilize the general multilingual understanding capabilities of the large model and reduces the model training complexity. Furthermore, the encoding network can be sequentially connected to the mapping network, the fusion network can be placed between the network layers of the large model, and the mapping network also outputs to both the first network layer and the fusion network. Please refer to [reference needed]. Figure 2 , Figure 2 This is a schematic diagram of the framework of an embodiment of a speech translation model. For example... Figure 2 As shown, Figure 2 The diagram shows a large model with four network layers. The speech to be translated is processed by an encoding network to extract speech features, which are then mapped by a mapping network to have the same feature dimensions as the output features of the network layers in the large model. These mapped features, along with the text features of the prompt text, are input into the large model. The first network layer is the second network layer in the large model. Its first output feature is fused with the mapped feature to obtain the first fused feature, which is then input into the next network layer (the third network layer) to obtain a new first output feature. This new first fused feature is then fused with the mapped feature again to obtain a new first fused feature, which is then input into the next network layer (the fourth network layer) to decode the target text. It should be noted that self-decoding or other decoding methods can be used during the decoding process; no specific method is specified here. Furthermore, when using self-decoding, each decoding operation yields the probability value of each preset character in a preset vocabulary. Finally, strategies such as beam search can be used to obtain the target text. For details on beam search techniques, please refer to the technical specifications; they will not be elaborated upon here. Of course, the number of network layers in the large model can also be other values, such as 5, 6, 7, etc., and is not limited here. Furthermore, the number of fusion networks can also be other values, such as 3, 4, 5, etc., also not limited here, as long as the fusion networks are located between network layers. In the model structure... Figure 2 In other cases not shown, the specific process of speech translation can be deduced similarly and will not be elaborated upon here. Please refer to further reading. Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of a converged network. For example... Figure 3As shown, the fusion network can specifically include an attention mechanism layer (e.g., cross-attention mechanism), an activation layer (e.g., tanh), and the input of the attention mechanism layer is residually connected to the output of the activation layer. It also sequentially connects to a feedforward layer and an activation layer, with the input of the feedforward layer residually connected to the output of the activation layer. It should be noted that the input of the attention mechanism layer includes three components: key features and value features obtained through mapping feature transformation, and query features obtained through the first output feature transformation. The specific processing flow of the fusion network can be found in the aforementioned descriptions and will not be repeated here.

[0048] The above scheme extracts the speech features of the speech to be translated and the text features of the prompt text, which is used to indicate the translation from the source language to the target language. It then obtains the mapped features of the speech features after feature dimension mapping, and inputs the text features and mapped features into a large model. It also obtains the first output feature of the large model during processing, where the mapped feature and the first output feature have the same feature dimension. This yields a first fusion feature based on the fusion of the first output feature and the mapped feature. The large model then further processes the first fusion feature to obtain the target text output, which is the translated text from the speech to the target language. On the one hand, by fully utilizing the large model's general multilingual translation capabilities and aligning the speech features with the large model input through feature mapping, and by adding a fusion operation between the first output feature and the mapped feature in the large model, the large model can fully utilize semantic knowledge to generate the target text in the target language without separate modeling. This minimizes the resource consumption and translation accuracy of multilingual speech translation. On the other hand, for the user, only the speech to be translated and the prompt text indicating the translation from the source language to the target language need to be input, effectively reducing the complexity of use. Therefore, it can minimize the resource consumption and complexity of multilingual speech translation, and improve translation accuracy.

[0049] Please see Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of the training method for the speech translation model of this application. Specifically, it may include the following steps:

[0050] Step S41: Obtain sample audio, sample prompt text, and sample target text.

[0051] In this embodiment of the disclosure, the sample prompt text is used to indicate the translation from the source language of the sample speech to the target language, and the sample target text is the actual translated text of the sample speech into the target language. For ease of description, it can be represented as {X}. i ,T i .C i}, where X i Let T represent the i-th sample speech. iC represents the sample prompt text for the i-th sample speech. i This represents the target text of the i-th sample speech translated into the target language. Furthermore, regarding the description of the sample prompt text, please refer to the relevant descriptions in the foregoing disclosed embodiments, which will not be repeated here.

[0052] Step S42: Extract the sample speech features of the sample speech and extract the sample text features of the sample prompt text.

[0053] Specifically, please refer to the following: Figure 2 The sample speech features can be extracted by the encoding network in the speech translation model. It should be noted that, to improve the translation accuracy of the speech translation model, the encoding network can be pre-trained with multilingual sample speech. Furthermore, the extraction process of the sample text features of the sample prompt text can be found in the aforementioned descriptions of text feature extraction in the disclosed embodiments, and will not be repeated here. For ease of description, sample speech X... i The sample speech features can be denoted as H i Correspondingly, the sample prompt text T i The sample text features can be denoted as S i .

[0054] Step S43: Obtain the sample mapping features after the sample speech features are mapped by the feature dimension, and input the sample text features and sample mapping features into the large model in the speech translation model, and obtain the first sample output features in the processing of the large model.

[0055] In this embodiment of the disclosure, the sample mapping features and the first sample output features have the same feature dimension. Please continue to refer to the following... Figure 2 The sample mapping features can be obtained by mapping the sample speech features to the mapping network in the speech translation model. For details, please refer to the relevant descriptions of the mapping features in the aforementioned public embodiments, which will not be repeated here. Furthermore, the first sample output features can be obtained from the output of the first network layer in the large model. For the specific meaning of the first network layer, please refer to the relevant descriptions in the aforementioned public embodiments, which will also not be repeated here. For ease of description, the mapping network f... proj (·) Transfer the sample speech features H i Perform mapping to obtain the sample mapping features Z. i And the sample mapping feature Z i and sample text features S i As a common input (such as concatenation), it forms the sample command representation P. i .

[0056] Step S44: Obtain the first sample fusion feature based on the fusion of the first sample output feature and the sample mapping feature.

[0057] Specifically, please refer to the following: Figure 2 The first sample fusion feature can be obtained by the fusion network in the speech translation model by fusing the output feature of the first sample with the sample mapping feature. For details, please refer to the relevant description of the fusion network in the aforementioned public embodiments, which will not be repeated here.

[0058] Step S45: Obtain the predicted target text from the first sample fusion feature output of the large model for further processing.

[0059] In this embodiment, the predicted target text is the predicted translated text in the target language obtained by translating the sample speech into a speech translation model. It should be noted that the first sample fusion features can be processed sequentially by a second network layer following the aforementioned first network layer. The specific meaning of the second network layer can be found in the relevant descriptions in the aforementioned embodiments, and will not be repeated here. Based on this, the predicted target text is finally decoded; or, as... Figure 2 As shown, the large model can contain multiple fusion networks. That is, during the sequential processing through the second network layer, at least one feature fusion can be interspersed. For details, please refer to the description of "detecting whether the number of feature fusions is not less than a preset number" in the aforementioned disclosed embodiments. The main difference is that in this disclosed embodiment, the preset number is the number of fusion networks interspersed in the large model. The rest of the process is largely similar to the aforementioned disclosed embodiments and will not be repeated here. For ease of description, the sample command can be represented as P. i Input a large LLM model, and during this period, map the sample to the feature Z. i By inputting the fusion network CML, which is interspersed among the network layers within a large model, the predicted target text Y can be obtained. i :

[0060] Y i =f LLM+CML (P i Z i )……(3)

[0061] Step S46: Adjust the network parameters of the speech translation model based on the difference between the sample target text and the predicted target text.

[0062] As one possible implementation, the cross-entropy loss function can be used to measure the difference between the sample target text and the predicted target text, thus obtaining the loss value L of the speech translation model:

[0063]

[0064] In the above formula (4), γ represents the learnable network parameters in the speech translation model, namely the network parameters of the encoding network, mapping feature, and fusion network. I represents the total number of samples in the training batch, and Q represents the text length of the i-th predicted target text. For the specific calculation principle, please refer to the technical details of the cross-entropy loss function, which will not be elaborated here.

[0065] The above-described scheme, during the training of the speech translation model, leverages the model's inherent multilingual translation capabilities. It aligns speech features with the model's input through feature mapping and incorporates a fusion operation between the first output feature and the mapped features. This allows the model to fully utilize semantic knowledge to generate target text in the target language without requiring separate modeling, thus minimizing resource consumption and improving translation accuracy in multilingual speech translation. Furthermore, for the user, only the speech to be translated and the accompanying text indicating the translation from the source language to the target language are required, effectively reducing complexity. Therefore, this approach minimizes resource consumption and complexity in multilingual speech translation while improving translation accuracy.

[0066] Please see Figure 5 , Figure 5 This is a schematic diagram of the framework of an embodiment of the speech translation device 50 of this application. The speech translation device 50 includes: a feature extraction module 51, a feature mapping module 52, a first processing module 53, a feature fusion module 54, and a second processing module 55. The feature extraction module 51 is used to extract the speech features of the speech to be translated and extract the text features of the prompt text; wherein, the prompt text is used to indicate the translation from the source language of the speech to be translated into the target language. The feature mapping module 52 is used to obtain the mapped features after the speech features are mapped by feature dimensions. The first processing module 53 is used to input the text features and the mapped features into a large model, and to obtain the first output features during the processing of the large model; wherein, the mapped features and the first output features have the same feature dimensions. The feature fusion module 54 is used to obtain the first fused features based on the fusion of the first output features and the mapped features. The second processing module 55 is used to obtain the target text output by the large model after further processing of the first fused features; wherein, the target text is the translated text of the speech to be translated into the target language.

[0067] The aforementioned scheme, through the speech translation device 50, fully utilizes the general multilingual translation capabilities of the large model itself. It aligns speech features with the input of the large model via feature mapping and adds a fusion operation between the first output feature and the mapped feature within the large model. This allows the large model to fully leverage semantic knowledge to generate target text in the target language without requiring separate modeling, thus minimizing resource consumption and translation accuracy in multilingual speech translation. Furthermore, for the user, only the speech to be translated and the prompt text indicating the translation from the source language to the target language are required, effectively reducing usage complexity. Therefore, it minimizes resource consumption and usage complexity in multilingual speech translation while improving translation accuracy.

[0068] In some disclosed embodiments, the feature fusion module 54 includes an attention submodule for processing a first output feature and a mapped feature based on an attention mechanism to obtain a second output feature with the same feature dimension as the first output feature; the feature fusion module 54 includes a first fusion submodule for fusing the first output feature and the second output feature to obtain a second fused feature; the feature fusion module 54 includes a feedforward processing submodule for performing feedforward processing based on the second fused feature to obtain a third output feature; and the feature fusion module 54 includes a second fusion submodule for fusing the second fused feature and the third output feature to obtain a first fused feature.

[0069] In some disclosed embodiments, the attention submodule includes a feature acquisition unit, used to obtain key features and value features of the attention mechanism based on the mapping features, and to obtain query features of the attention mechanism based on the first output features; the attention submodule includes a feature output unit, used to obtain a second output feature based on the query features, key features and value features.

[0070] In some disclosed embodiments, the speech translation device 50 further includes a count detection module for detecting whether the number of feature fusions is less than a preset number; the second processing module 55 is specifically used to, in response to the number of feature fusions being less than the preset number, repeatedly execute the step of obtaining the first output feature in the large model processing process until the number of feature fusions is not less than the preset number, and obtain the target text output based on the last network layer in the large model.

[0071] In some disclosed embodiments, the feature extraction module 51 includes a network acquisition submodule for acquiring an encoding network pre-trained with multilingual sample speech; the feature extraction module 51 also includes a speech encoding submodule for encoding the speech to be translated based on the encoding network to obtain the speech features of the speech to be translated.

[0072] In some publicly disclosed embodiments, the translated text of the speech to be translated is obtained by a speech translation model processing the speech to be translated and the prompt text. The speech translation model includes a large model, an encoding network, a mapping network, and a fusion network. During the training process, the network parameters of the large model are fixed.

[0073] In some disclosed embodiments, the encoding network and the mapping network are connected sequentially, the fusion network is located between the network layers of the large model, and the mapping network also outputs to the first network layer and the fusion network respectively.

[0074] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the electronic device 60 of this application. The electronic device 60 includes a memory 61 and a processor 62. The memory 61 stores program instructions, and the processor 62 is used to execute the program instructions to implement the steps in any of the above-described speech translation method embodiments. Specifically, the electronic device 60 may include, but is not limited to, mobile phones, tablet computers, translators, laptops, learning machines, office laptops, servers, etc., and is not limited thereto.

[0075] Specifically, processor 62 controls itself and memory 61 to implement the steps in any of the above-described speech translation method embodiments. Processor 62 can also be referred to as a CPU (Central Processing Unit). Processor 62 may be an integrated circuit chip with signal processing capabilities. Processor 62 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 62 can be implemented using integrated circuit chips.

[0076] The above-described scheme, in its electronic device 60, fully utilizes the multilingual translation capabilities of the large model itself, aligns speech features with the input of the large model through feature mapping, and adds a fusion operation between the first output feature and the mapped feature in the large model. This allows the large model to fully utilize semantic knowledge to generate target text in the target language without requiring separate modeling, thereby minimizing the resource consumption and translation accuracy of multilingual speech translation. Furthermore, for the user, only the speech to be translated and the prompt text indicating the translation from the source language to the target language are needed, effectively reducing the complexity of use. Therefore, it can minimize the resource consumption and complexity of multilingual speech translation while improving translation accuracy.

[0077] Please see Figure 7 , Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium 70 of this application. The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor. The program instructions 71 are used to implement the steps in any of the above-described embodiments of the speech translation method.

[0078] The above-described scheme, through the computer-readable storage medium 70, fully utilizes the large model's inherent multilingual translation capabilities and aligns speech features with the large model's input via feature mapping. Furthermore, by adding a fusion operation between the first output feature and the mapped feature within the large model, the model can fully leverage semantic knowledge to generate target text in the target language without requiring separate modeling. This minimizes the resource consumption and translation accuracy of multilingual speech translation. On the other hand, for the user, only the speech to be translated and the prompt text indicating the translation from the source language to the target language are required, effectively reducing usage complexity. Therefore, this scheme minimizes the resource consumption and usage complexity of multilingual speech translation while improving translation accuracy.

[0079] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0080] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0083] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0084] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A speech translation method, characterized in that, include: Extract the speech features of the speech to be translated and extract the text features of the prompt text; wherein, the prompt text is used to indicate the translation from the source language of the speech to be translated into the target language; The speech features are obtained by mapping the speech features to the feature dimension, and the text features and the mapping features are input into a large model, and the first output feature is obtained during the processing of the large model; wherein the mapping feature and the first output feature have the same feature dimension. Obtain a first fused feature based on the fusion of the first output feature and the mapping feature; The large model is then processed to obtain the target text output by the first fusion feature; wherein, the target text is the translated text of the speech to be translated into the target language.

2. The method according to claim 1, characterized in that, The step of obtaining the first fused feature based on the first output feature and the mapping feature includes: The first output feature and the mapping feature are processed based on an attention mechanism to obtain a second output feature with the same feature dimension as the first output feature. The second fused feature is obtained by fusing the first output feature and the second output feature. Based on the second fusion feature, a feedforward process is performed to obtain the third output feature; The first fusion feature is obtained by fusing the second fusion feature and the third output feature.

3. The method according to claim 2, characterized in that, The process of processing the output feature and the mapping feature based on the attention mechanism to obtain a second output feature with the same feature dimension as the output feature includes: Based on the mapping features, the key features and value features of the attention mechanism are obtained, and based on the first output features, the query features of the attention mechanism are obtained. The second output feature is obtained based on the query feature, the key feature, and the value feature.

4. The method according to claim 1, characterized in that, After obtaining the first fused feature based on the first output feature and the mapping feature, and before obtaining the target text output by the large model further processing the first fused feature, the method further includes: Whether the number of feature fusion attempts is less than the preset number; The step of obtaining the target text output by the large model for further processing of the first fused feature includes: In response to the fact that the number of feature fusions is less than the preset number, the step of obtaining the first output feature in the large model processing is executed repeatedly until the number of feature fusions is not less than the preset number, and the target text finally output by the large model is obtained.

5. The method according to claim 1, characterized in that, The extraction of speech features from the speech to be translated includes: Obtain the encoding network pre-trained from multilingual sample speech; The speech to be translated is encoded based on the encoding network to obtain the speech features of the speech to be translated.

6. The method according to any one of claims 1 to 5, characterized in that, The translated text of the speech to be translated is obtained by the speech translation model processing the speech to be translated and the prompt text. The speech translation model includes the large model, the encoding network, the mapping network and the fusion network, and the network parameters of the large model are fixed during the training process of the speech translation model.

7. The method according to claim 6, characterized in that, The encoding network and the mapping network are connected sequentially, the fusion network is located between the network layers of the large model, and the mapping network also outputs to the first network layer and the fusion network respectively.

8. A voice translation device, characterized in that, include: The feature extraction module is used to extract the speech features of the speech to be translated and the text features of the prompt text; wherein, the prompt text is used to indicate the translation from the source language of the speech to be translated into the target language; The feature mapping module is used to obtain the mapped features of the speech features after feature dimension mapping; The first processing module is used to input the text features and the mapping features into a large model, and to obtain the first output features during the processing of the large model; wherein the mapping features and the first output features have the same feature dimension; The feature fusion module is used to obtain a first fused feature based on the fusion of the first output feature and the mapped feature; The second processing module is used to obtain the target text output by the large model for further processing of the first fusion feature; wherein, the target text is the translated text of the speech to be translated into the target language.

9. An electronic device, characterized in that, The method includes a memory and a processor coupled to each other, the memory storing program instructions, and the processor executing the program instructions to implement the speech translation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The system stores program instructions that can be executed by a processor, the program instructions being used to implement the speech translation method according to any one of claims 1 to 7.

Images