Voice processing method and electronic device

Abstract

The present disclosure relates to a speech processing method, apparatus, electronic device, computer readable storage medium and computer program product. The method comprises: obtaining a trained speech translation model, wherein the trained speech translation model comprises an acoustic encoder and a text encoder, wherein the acoustic encoder and / or the text encoder comprises an intermediate CTC module between adjacent first and second layers, the intermediate CTC module being configured to determine an input of the second layer based on an output of the first layer and a word embedding matrix; and inputting a source language speech to be processed into the trained speech translation model to obtain a corresponding target language text. In this way, the speech translation model is able to solve the independent assumption problem inherent in CTC by introducing an intermediate CTC module between two adjacent layers in the acoustic encoder and / or the text encoder, which integrates the predictive perceptual coding into the encoding information, and thus is able to improve the performance of the speech translation processing.

Description

Technical Field

[0001] This disclosure generally relates to the field of computers, and more specifically to speech processing methods and electronic devices. Background Technology

[0002] Speech translation (ST) aims to translate spoken language into text in a target language. It is widely used in various scenarios, such as conference speech translation, video subtitle translation, and augmented reality (AR) translation. Models for speech translation can be built based on technologies such as machine learning and deep learning.

[0003] Traditional speech translation systems consist of two cascaded models: a speech recognition model and a machine translation model. The speech recognition model performs Automatic Speech Recognition (ASR) to convert source language speech into a transcribed text. The machine translation model performs Machine Translation (MT) to translate the transcribed text into the target language. However, cascading these two models results in slow speech translation speeds and significant translation delays.

[0004] End-to-end models can convert source language speech into target language text using a single model. They employ an autoregressive approach and are therefore also known as autoregressive translation models. Compared to cascaded models, end-to-end models significantly reduce translation latency. However, further research is needed to further improve the performance and speed of speech translation. Summary of the Invention

[0005] According to an example embodiment of this disclosure, a speech processing scheme is provided that can solve the independent assumption problem, thereby improving the performance of speech translation processing.

[0006] In a first aspect of this disclosure, a speech processing method is provided, comprising: acquiring a trained speech translation model, wherein the trained speech translation model includes an acoustic encoder and a text encoder, wherein the acoustic encoder and / or the text encoder includes an intermediate CTC module between adjacent first and second layers, the intermediate CTC module being used to determine the input of the second layer based on the output of the first layer and a word embedding matrix; and inputting the source language speech to be processed into the trained speech translation model to obtain the corresponding target language text.

[0007] In a second aspect of this disclosure, an electronic device is provided, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method described in the first aspect of this disclosure when executed by the at least one processing unit.

[0008] In a third aspect of this disclosure, a speech processing apparatus is provided, comprising: a model acquisition unit configured to acquire a trained speech translation model, wherein the trained speech translation model includes an acoustic encoder and a text encoder, wherein the acoustic encoder and / or the text encoder includes an intermediate CTC module between adjacent first and second layers, the intermediate CTC module being used to determine the input of the second layer based on the output of the first layer and a word embedding matrix; and an output determination unit configured to input source language speech to be processed into the trained speech translation model to obtain corresponding target language text.

[0009] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having machine-executable instructions stored thereon, which, when executed by a device, cause the device to perform the method described in the first aspect of this disclosure.

[0010] In a fifth aspect of this disclosure, a computer program product is provided, including computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method described in the first aspect of this disclosure.

[0011] In a sixth aspect of this disclosure, an electronic device is provided, comprising: processing circuitry configured to perform the method described in the first aspect of this disclosure.

[0012] The summary section is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0014] Figure 1A A schematic diagram of the model structure of a speech translation model according to some embodiments of the present disclosure is shown;

[0015] Figure 1BA schematic diagram of the structure of an acoustic encoder and a text encoder for a speech translation model according to some embodiments of the present disclosure is shown;

[0016] Figure 2 A schematic flowchart illustrating the model training process according to some embodiments of the present disclosure is shown;

[0017] Figure 3 A schematic diagram of a CTC module according to some embodiments of the present disclosure is shown;

[0018] Figure 4 A schematic diagram of a CTC module for a text encoder according to some embodiments of the present disclosure is shown;

[0019] Figure 5 A schematic diagram of the structure of layers in a text encoder according to some embodiments of the present disclosure is shown;

[0020] Figure 6 A flowchart illustrating the model usage process according to some embodiments of this disclosure is shown;

[0021] Figure 7 Block diagrams of example apparatuses according to some embodiments of the present disclosure are shown; and

[0022] Figure 8 A block diagram of an example device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] As mentioned earlier, speech translation has been widely used in various scenarios. To meet the performance and speed requirements of speech translation in different scenarios, an end-to-end model using autoregressive generation has been proposed to realize the conversion from source language speech to target language text.

[0025] Recently, end-to-end models employing non-autoregressive generation have been proposed, enabling parallel processing to further improve speech translation speed. However, the training process for end-to-end speech translation models is inherently challenging, and reaching convergence is not easy. The non-autoregressive generation of objectives further exacerbates the difficulty of model learning; therefore, designing appropriate training strategies is crucial.

[0026] End-to-end models employing non-autoregressive generation have achieved excellent results in speech recognition and text translation tasks. These models are based on Connectionist Temporal Classification (CTC). CTC is suitable for generation tasks where the model input length is greater than the model output length and allows for variable-length generation.

[0027] However, for speech translation, text generated based on CTC suffers from at least the problem of conditional independence assumption. Conditional independence is an inherent property of CTC and the source of its ability to perform non-autoregressive generation. However, this assumption ignores internal dependencies with the context, leading to poor generation performance.

[0028] Furthermore, text generated based on CTC suffers from the monotonicity assumption problem. The inherent monotonicity assumption of CTC does not hold true for translation tasks, and while text translation models use CTC for generation at the decoder end, the reordering problem can be mitigated through encoder-decoder attention modules, but this increases structural complexity. One consideration is to adopt an encoder-only architecture. Although self-attention networks already possess some reordering capabilities, decoder-less structures still struggle to handle the monotonicity assumption problem, especially in long-distance languages.

[0029] Furthermore, convergence difficulties exist. End-to-end speech translation models already bear a significant burden of cross-modal and cross-linguistic modeling, and non-autoregressive generation further increases the difficulty of model training.

[0030] To address the aforementioned problems and other potential issues, embodiments of this disclosure provide a speech translation model. This speech translation model includes an acoustic encoder and a text encoder, with the acoustic encoder and / or text encoder comprising an intermediate CTC module between adjacent first and second layers. This intermediate CTC module is used to determine the input to the second layer based on the output of the first layer and the word embedding matrix. The intermediate CTC module can be used to integrate predictive-aware coding into the encoded information, thus resolving the inherent independence assumption problem of CTC. Therefore, it can further improve the performance of speech translation processing while maintaining the processing speed of CTC.

[0031] The speech translation model in the embodiments of this disclosure may also be referred to as a CTC-based non-autoregressive speech translation (NAST) model, or a CTC-based end-to-end NAST model, or other names, which are not limited in this disclosure. As an example, the speech translation model in the embodiments of this disclosure may be represented as CTC-NAST. Furthermore, CTC-NAST can be generated in parallel, thereby ensuring a high speedup ratio.

[0032] Figure 1A A schematic diagram of the model structure of a speech translation model 100 according to some embodiments of the present disclosure is shown. The speech translation model 100 includes an acoustic encoder 110 and a text encoder 120, which predict transcribed text 102 and translated text 103 via CTC, respectively.

[0033] In this manner, embodiments of the present disclosure construct a suitable architecture for non-autoregressive speech translation that includes only encoders and not decoders, specifically modeled by two stacked encoders (i.e., acoustic encoder 110 and text encoder 120).

[0034] like Figure 1A As shown, for the source language speech, its speech features 101 can be determined. Through the acoustic encoder 110 and the source CTC 112, the transcribed text 102 can be obtained. Subsequently, through the text encoder 120 and the target CTC 122, the translated text 103 can be obtained.

[0035] In some examples, learning the transcribed text can be used as an auxiliary loss during training (e.g.) Figure 1A The CTC loss in the model helps the model converge better.

[0036] Understandably, the input to the speech translation model 100 can be source language speech, and the output can be target language text (i.e., Figure 1A The translated text shown is 103. That is, the input format of the speech translation model 100 is speech, and its language is the source language. The output format of the speech translation model 100 is text, and its language is the target language, wherein the target language is different from the source language. In embodiments of this disclosure, the terms "language" and "language type" can be used interchangeably in some scenarios, and this disclosure does not limit this.

[0037] Figure 1B A schematic diagram of the structure 150 of an acoustic encoder 110 and a text encoder 120 of a speech translation model according to some embodiments of the present disclosure is shown. Figure 1B As shown, the speech translation model includes an acoustic encoder 110 and a text encoder 120, wherein the acoustic encoder 110 includes multiple layers (such as...). Figure 1B The middle layer is M1), and the text encoder 120 includes multiple layers (such as M1 layer). Figure 1B (M2 layer is in the middle).

[0038] Optionally, the number of layers in the acoustic encoder 110 may be equal to or unequal to the number of layers in the text encoder 120. That is, M1 and M2 may be equal or unequal. As an example, it can be assumed that M1=M2=12. For example, the acoustic encoder 110 includes layers 1 to 12, and the text encoder 120 includes layers 1 to 12.

[0039] However, it should be understood that in other embodiments, the number of layers (M1) of the acoustic encoder 110 and the number of layers (M2) of the text encoder 120 may not be equal, or even if the number of layers is equal, it may be other values, such as 10 layers, 16 layers, etc., and this disclosure does not limit this.

[0040] The acoustic encoder 110 includes at least one intermediate CTC module. For example... Figure 1B As shown, intermediate CTC modules 165 are illustrated in layers 161 (layer m) and 162 (layer m+1) of the acoustic encoder 110. The text encoder 120 includes at least one intermediate CTC module. Figure 1B As shown, the intermediate CTC module 175 is shown in layers 171 (layer n) and 172 (layer n+1) of the text encoder 120.

[0041] It should be noted that, although in Figure 1B The acoustic encoder 110 shown in the diagram has only one intermediate CTC module, but in real-world scenarios, the acoustic encoder 110 may include more than one intermediate CTC module. For example, an intermediate CTC module may be included between every two layers of the acoustic encoder 110, such as the acoustic encoder 110 including M1-1 intermediate CTC modules. For instance, a source intermediate CTC module may be placed between layer 6 and layer 7 of the acoustic encoder 110, and another source intermediate CTC module may be placed between layer 9 and layer 10 of the acoustic encoder 110, for example, the acoustic encoder 110 includes two intermediate CTC modules.

[0042] It should be noted that, although in Figure 1B The text encoder 120 shown in the diagram has only one intermediate CTC module, but in real-world scenarios, the text encoder 120 may include more than one intermediate CTC module. For example, an intermediate CTC module can be included between every two layers of the text encoder 120; for instance, the text encoder 120 may include M²-1 intermediate CTC modules. For example, a target intermediate CTC module can be placed between layers 6 and 7 of the text encoder 120, and another target intermediate CTC module can be placed between layers 9 and 10 of the text encoder 120. For example, the acoustic encoder 110 includes two intermediate CTC modules.

[0043] It is understood that the number of intermediate CTC modules included in the acoustic encoder 110 may be equal to or different from the number of intermediate CTC modules included in the text encoder 120. For example, the acoustic encoder 110 may include one intermediate CTC module, while the text encoder 120 may include two or more intermediate CTC modules.

[0044] Understandably, the location of the intermediate CTC module included in the acoustic encoder 110 may be the same as, partially the same as, or completely different from the location of the intermediate CTC module included in the text encoder 120. For example, the acoustic encoder 110 may include one intermediate CTC module located between layer 6 and layer 7 of the acoustic encoder 110; the text encoder 120 may include two intermediate CTC modules, located between layer 5 and layer 6 and between layer 9 and layer 10 of the text encoder 120, respectively.

[0045] In embodiments of this disclosure, the source intermediate CTC module 165 and the target intermediate CTC module 175 may have the same structure or may have different structures. For example, the source intermediate CTC module 165 and the target intermediate CTC module 175 may have the same combination as described below. Figure 3 The structure is shown. For example, the source intermediate CTC module 165 can have the following combination. Figure 3 The structure shown, and the target intermediate CTC module 175 can have different combinations as described below. Figure 4 The structure shown.

[0046] In some embodiments of this disclosure, the intermediate CTC module 165 in the acoustic encoder 110 may be referred to as the source intermediate CTC module, while the intermediate CTC module 175 in the text encoder 120 may be referred to as the target intermediate CTC module, and collectively referred to as the inter-CTC module.

[0047] In embodiments of this disclosure, an intermediate CTC module is positioned between two adjacent layers (located within the same encoder, such as acoustic encoder 110 or text encoder 120) and can be used to determine the input of a subsequent layer based on the output of the preceding layer. For example, if an intermediate CTC module is positioned between layer 6 and layer 7, the input of layer 7 can be determined based on the output of layer 6. A more detailed description of the intermediate CTC module can be found below in conjunction with... Figures 3 to 4 Some of the embodiments.

[0048] Figure 2 A schematic flowchart of a model training process 200 according to some embodiments of the present disclosure is shown. In box 210, a training dataset is constructed, which may include multiple data items, each including a source language speech sample and a corresponding target language text. In box 220, a trained speech translation model is generated based on the training dataset.

[0049] Optionally, in some embodiments, each data item may include a source language speech sample, a source language text sample, and a target language text sample. The source language and the target language are different languages. For example, the source language is English and the target language is French. For example, the source language is Spanish and the target language is German. It should be noted that the examples of source and target languages ​​herein are merely illustrative and should not be construed as limiting the embodiments of this disclosure.

[0050] As described above, the speech translation model includes an acoustic encoder and a text encoder, the acoustic encoder comprising multiple layers, and the text encoder comprising multiple layers. Exemplarily, the acoustic encoder includes an intermediate CTC module between at least two adjacent layers. Exemplarily, the text encoder includes an intermediate CTC module between at least two adjacent layers.

[0051] In some embodiments, whether for an acoustic encoder or a text encoder, it can be assumed that an intermediate CTC module is provided between the L1 layer and the L2 layer (where L2 = L1 + 1). This intermediate CTC module enables the introduction of predictive perceptual coding methods to address the aforementioned conditional independence assumption problem.

[0052] If the output of the L1 layer is represented as Then, the intermediate CTC module (prediction-aware coding method) can weight the word embedding matrix through the CTC prediction distribution and integrate the predicted content into the output of the L1 layer, as shown in the following equation (1):

[0053] (1)

[0054] In equation (1), Represents the word embedding matrix, This indicates the output of the intermediate CTC module for layer L1. The obtained CTC prediction distribution, This indicates the path predicted by CTC. This represents the integrated result. In this way, the integrated result can be used as the input to the L2 layer.

[0055] In embodiments of this disclosure, the word embedding matrix is ​​shared among the various intermediate CTC modules. As previously described, the acoustic encoder 110 may include one or more intermediate CTC modules, and the text encoder 120 may include the same or different numbers of one or more intermediate CTC modules. Therefore, the word embedding matrix used by each intermediate CTC module can be the same. Specifically, in the initial stage of training, the word embedding matrix can be generated through random initialization; and during intermediate iterations of training, the word embedding matrix can be iteratively updated.

[0056] For example, the intermediate output of the encoder (acoustic encoder 110 and / or text encoder 120) can be understood as encoded information, such as the output of the L1 layer, which is also referred to as encoded information. The product of the CTC predictive coding distribution of the L1 layer output and the word embedding matrix can be understood as predictive coding information, or it can be referred to as the bias associated with the output of the L1 layer, the offset associated with the output of the L1 layer, or other names, which are not limited in this disclosure.

[0057] Figure 3 A schematic diagram of a CTC module 300 according to some embodiments of the present disclosure is shown. For example... Figure 3 As shown, the output 301 of the L1 layer can be regularized by layer 320 and then the CTC prediction distribution 314 can be obtained by softmax 330.

[0058] By multiplying the word embedding matrix 312 by the CTC prediction distribution 314 (310), the predicted content 302, which can be called the bias 302, can be obtained. Further, by integrating the bias 302 with the output 301 of the L1 layer (340), the input 303 of the next layer (L2 layer) can be obtained. Optionally, the result of layer regularization (320) of the output 301 of the L1 layer can be integrated with the bias 302, thus ensuring a consistent integration scope.

[0059] Optionally, bias 302 can be understood as the initial prediction result, thus enabling the initial prediction result to be integrated during training. Optionally, integration 340 can be implemented as summation, weighted summation, or similar operations, which are not limited in this disclosure.

[0060] Understandably, for the sake of simplicity, in Figure 3 The CTC prediction distribution 314 shown only displays the maximum values ​​in each column, such as 0.8, 0.7, 0.6, and 0.7, without showing the remaining values.

[0061] In this way, including an intermediate CTC module in the acoustic encoder 110 can effectively convert acoustic representations into more explicit textual information. It is understood that the intermediate CTC module (i.e., predictive perceptual coding) can integrate transcribed text prediction information and translated text prediction information in the acoustic encoder 110 and text encoder 120, respectively.

[0062] It is understood that the intermediate CTC module depends on the accuracy of the CTC prediction. For example, it can be assumed that the CTC prediction is as accurate as possible, but this prerequisite is not necessarily true. If the accuracy of the CTC prediction is low, the integration operation of the intermediate CTC module may introduce erroneous information. To this end, embodiments of this disclosure may include a first loss function associated with the intermediate CTC module in the acoustic encoder and a second loss function associated with the intermediate CTC module in the text encoder in the loss function during training. This optimizes the loss of the intermediate CTC. For example, the intermediate loss function can be expressed as Equation (2) below:

[0063] (2)

[0064] In equation (2), Indicates the prediction result. This represents the true result. Accordingly, the first loss function and the second loss function can be expressed as follows: and , where M represents the number of intermediate CTC modules in the acoustic encoder and N represents the number of intermediate CTC modules in the text encoder.

[0065] In some examples, the loss function used for training can be expressed as equation (3) below:

[0066] (3)

[0068] In equation (3), Indicates as Figure 1A The CTC loss shown represents, as Figure 1A The XCTC loss is shown. , , and All are weights.

[0069] In some embodiments of this disclosure, during the training of the intermediate CTC module of the text encoder 120, at least one of the multiple incorrectly predicted positions in the bias can be replaced with the correct output according to the replacement rate to obtain the replaced bias. This can improve the convergence speed and thus improve the training efficiency.

[0070] Figure 4 A schematic diagram of a CTC module 400 for a text encoder according to some embodiments of the present disclosure is shown. Figure 4As shown, assuming the output 401 of the L1 layer of the text encoder can be regularized by layer 420, the CTC prediction distribution 414 can be obtained by softmax 430. By multiplying the word embedding matrix 412 with the CTC prediction distribution 414 by 410, the bias 402 can be obtained.

[0071] On the other hand, a reference distribution 452 can be obtained through a smoothing operation based on the true output (obtained from data items in the training dataset). The correct content 454 can be obtained by multiplying the word embedding matrix 412 by the reference distribution 452 by 450.

[0072] The content in bias 402 can be partially replaced with the correct content according to the replacement rate, resulting in a replaced bias 403. In some examples, bias 402 can be compared with the correct content 454 to identify one or more locations of prediction errors. The number of locations to be replaced can be determined based on the replacement rate. The locations to be replaced can be sampled from one or more locations of prediction errors through random sampling. The content at the locations to be replaced in bias 402 can then be replaced with the correct content at the corresponding locations in correct content 454, as shown below. Figure 4 This yields the replaced bias 403.

[0073] Furthermore, by integrating the replaced bias 403 with the output 401 of the L1 layer (the result after layer regularization 320) 440, the input 404 of the next layer (the L2 layer) can be obtained. Optionally, the integration 440 can be implemented as a summation, a weighted summation, or a similar operation, which is not limited in this disclosure.

[0074] Understandably, for the sake of simplicity, in Figure 4 The CTC prediction distribution 414 and reference distribution 452 shown only show the maximum value in each column, without showing the remaining values.

[0075] Through the aforementioned substitutions, the embodiments of this disclosure employ a course-based learning strategy to provide the model with a smooth learning path, thereby addressing the aforementioned convergence difficulty problem. Specifically, the course-based learning strategy employed in the embodiments of this disclosure can be referred to as course-based learning hybridization.

[0076] Understandably, in the early stages of model training, replacing some content with correct information reduces the amount of data that needs to be learned, allowing the model to quickly acquire certain capabilities. As training progresses, the difficulty can be gradually increased to enable the model to achieve its final learning objective.

[0077] Specifically, some of the predicted content in the intermediate CTC module (i.e., prediction-aware coding) of the text encoder 120 can be replaced with the correct content, thereby mitigating the error propagation problem caused by the poor performance of XCTC.

[0078] Understandably, while this approach makes model training easier, it can lead to inconsistencies between the training and subsequent inference phases. For example, during training, the loss on the training set decreases rapidly due to the presence of known correct content; however, during inference, the absence of correct content may result in poor model output.

[0079] Since the speech translation model 100 in the embodiments of this disclosure is based on CTC, and CTC is designed for tasks where the input length is greater than the output length, there may be multiple "empty" positions in the predicted content. Therefore, a scheme that dynamically determines the replacement rate based on the current prediction accuracy may not be feasible and may not substantially help with convergence difficulties. For example, reducing the replacement rate may lead to a failure to provide effective information, while an excessively high replacement rate may easily result in a serious inconsistency between training and inference.

[0080] In some embodiments of this disclosure, multiple locations in the bias where predictions are incorrect can be replaced according to a replacement rate. That is, replacement can be restricted to only the locations where predictions are incorrect. In this way, a larger replacement rate can be used throughout the training process.

[0081] In comparison, the approach that replaces all content can be called the basic method. Understandably, in the early stages of training, since most predictions are inaccurate, this strategy (i.e., replacing based on incorrect content) is essentially no different from the basic method; almost all positions are mixed with predicted content at a fixed replacement rate. As the model's capabilities improve, the basic method still replaces most of the predicted content, while this strategy only randomly samples from the incorrectly predicted positions, leaving the correctly predicted positions untouched. Therefore, for training, the provided hints remain unchanged, still ensuring good convergence. Simultaneously, the number of replacement positions is significantly reduced; compared to the inference stage, only a small portion is replaced with the correct answer, thus effectively mitigating the inconsistency problem.

[0082] In this way, in the embodiments of this disclosure, during the training process, the positions of the predicted errors are replaced with the correct content according to the replacement rate, which can improve the convergence speed and thus solve the problem of convergence difficulty.

[0083] Figure 5A schematic diagram of the structure 500 of a layer of a text encoder according to some embodiments of the present disclosure is shown. Each layer of the text encoder may include a self-attention module and a feed-forward network (FFN).

[0084] Reference Figure 5 The text encoder layer 510 includes a self-attention module 512 and an FFN 514. The text encoder layer 520 includes a self-attention module 522 and an FFN 524. Furthermore, at least one layer of the text encoder (such as...) Figure 5 The layer shown (520) also includes a cross-layer attention module (523).

[0085] Specifically, the cross-layer attention module 523 is inserted between the self-attention module 522 and the FFN 524. The cross-layer attention module 523 also employs an attention mechanism, and its inputs include the output of the self-attention module 522 and the reference layer (e.g., ...). Figure 5 The output of layer 510 in the middle, and the reference layer (such as Figure 5 The output of layer 510 in the reference layer. For example, the reference layer does not include a cross-layer attention module.

[0086] In some embodiments of this disclosure, one of the multiple layers of the text encoder may be designated as the reference layer, such as layer K, for example, K=4 or other values, which are not limited in this disclosure. In some examples, at least one layer following the reference layer includes a cross-layer attention module. For example, each layer following the reference layer (such as layers K+1 to M2) may include a cross-layer attention module.

[0087] Thus, including both self-attention modules and cross-layer attention modules in at least one layer of the text encoder enables complementarity.

[0088] For example, suppose the self-attention module is represented as SA(q, k, v), and the cross-layer attention module is represented as CLA(q, k, v), where q represents the query, k represents the key, and v represents the value. Then the output of layer J can be obtained using the following formula:

[0089] (4)

[0090] (5)

[0091] (6)

[0092] In equations (4)-(6), This represents the output of layer K. The output of the self-attention module 522 in layer J is represented. The output of the cross-layer attention module 523 in layer J is represented. This indicates the output of layer J.

[0093] In this way, the cross-layer attention module can provide a way to compensate for missing attention by capturing information directly from the bottom reference layer. In some examples, text encoders that include cross-layer attention modules can achieve similar performance to architectures that include both encoders and decoders.

[0094] Additionally or optionally, for a specific layer (e.g., layer J) that includes a cross-layer attention module, the self-attention module in that specific layer can be discarded based on randomly generated probabilities. Specifically, at some iteration stage of the training process, for that specific layer, the cross-layer attention module can be trained without training the self-attention module. This allows information (such as information from the reference layer) to be captured solely through the cross-layer attention module, thereby enhancing the learning capability. The process of discarding the self-attention module can be based on drop-net techniques, for example, by representing the randomly generated probabilities as... ,in It represents a closed interval from 0 to 1.

[0095] Based on the above and Figures 1 to 1 Figure 5 The present disclosure provides a speech translation model that can be generated through a training process, as described in some embodiments. Specifically, this speech translation model addresses the inherent independence assumption problem of CTC by introducing an intermediate CTC module between two adjacent layers in the acoustic encoder and text encoder, integrating predictive perceptual encoding into the encoded information. By introducing a cross-layer attention module in the layer following the reference layer of the text encoder, the lack of attention provided by the self-attention module is compensated for, capturing information from the bottom reference layer and thus resolving the monotonicity assumption problem. During training, the predictive encoding of the intermediate CTC module in the text encoder can replace incorrect content with correct content according to the replacement rate, accelerating the training process and addressing convergence difficulties.

[0096] Figure 6 An example flowchart of example usage process 600 according to some embodiments of the present disclosure is shown. At block 610, a trained speech translation model is obtained, wherein the trained speech translation model includes an acoustic encoder and a text encoder, wherein the acoustic encoder and / or text encoder includes an intermediate CTC module between adjacent first and second layers, the intermediate CTC module being used to determine the input to the second layer based on the output of the first layer and the word embedding matrix. At block 620, the source language speech to be processed is input into the trained speech translation model to obtain the corresponding target language text.

[0097] In embodiments of this disclosure, the trained speech translation model acquired at box 610 may have the following characteristics: Figure 1A The model structure shown can have acoustic encoders and / or text encoders as follows: Figure 1B The structure shown is such that the layers of the text encoder can have, for example, the structure shown. Figure 5 The structure shown. The trained speech translation model obtained at box 610 can be achieved through the combination of the above. Figures 2 to 5 The process described above will not be repeated here for the sake of brevity.

[0098] Optionally, the acoustic encoder comprises multiple layers, each including a self-attention module and an FFN. Optionally, the text encoder comprises multiple layers, each including a self-attention module and an FFN, and layers above the reference layer (excluding the reference layer) may optionally include cross-layer attention modules.

[0099] Suppose that a specific layer of the text encoder (located above the reference layer) includes a cross-layer attention module, then the three inputs of this cross-layer attention module are: the output of the self-attention module of the specific layer, the output of the reference layer, and the output of the reference layer.

[0100] As previously mentioned, the acoustic encoder and the text encoder each include one or more intermediate CTC modules, for example, one intermediate CTC module between layers 6 and 7 of the acoustic encoder and one between layers 9 and 10 of the text encoder. Taking the intermediate CTC module located between layers 6 and 7 as an example, the CTC distribution of the output of layer 6 can be determined; based on this CTC distribution and the word embedding matrix (e.g., the product of the two), the bias associated with the output of layer 6 can be determined; subsequently, the output of layer 6 can be integrated with this bias (e.g., summed) to determine the input of layer 7.

[0101] Specifically, at position 620, the speech features of the source language speech to be processed can be determined, for example, through feature extraction methods; these speech features can then be used as input to layer 1 of the acoustic encoder. The output of the text encoder, such as the output of layer M2 of the text encoder, can be determined as the output of the speech translation model, i.e., the corresponding target language text.

[0102] In this way, embodiments of the present disclosure can determine the target language text corresponding to the source language speech through a trained speech translation model. This determination process is fast and performs well. For example, Table 1 below shows the comparison results between the present solution and some existing solutions, including the CTC basic solution (CTC), the CTC improvement solution (Orthros-CTC), and the further improvement solution (Orthros-CMLM) proposed by Inaguma et al.

[0103] Specifically, the comparison results were performed on the open-source Multilingual Speech Translation Corpus (MuST-C), with English as the source language. The target languages ​​included in Table 1 are German (De), Spanish (Es), French (Fr), Italian (It), and Dutch (Nl). The numerical values ​​shown in Table 1 represent Bilingual Evaluation Understudy (BLEU), which are percentage values. BLEU is an automatic evaluation metric for machine-translated text, which can be understood as a score of accuracy. The meanings of BLEU values ​​can be categorized as follows: <10 indicates almost useless translation; 10-19 indicates the main points are difficult to understand; 20-29 indicates the main points are clear, but there are obvious grammatical errors; 30-40 indicates a good quality translation; 40-50 indicates a high-quality translation; and >50 indicates a very high-quality, fluent translation.

[0104] Table 1

[0105]

[0106] As can be seen from Table 1, the BLEU obtained using this scheme is superior to that obtained using existing schemes. Furthermore, this scheme can also be applied to other target languages, such as Portuguese (Pt), Romanian (Ro), Russian (Ru), and Japanese (Ja). Since existing schemes do not list BLEU results for these languages, only the BLEU obtained using this scheme is shown in Table 2 below.

[0107] Table 2

[0108]

[0109] As can be seen, the CTC-NAST model proposed in this disclosure has a relatively simple architecture, can achieve full parallelism, and can achieve excellent performance on both close language pairs with similar word order (such as English-German) and distant language pairs with large word order differences (such as English-Japanese).

[0110] As shown in Table 1, in the English-German translation task, this scheme improves the BLEU value by 2-3 points compared to the existing scheme.

[0111] Furthermore, it should be noted that although some embodiments of this disclosure are discussed in relation to non-autoregressive models, some example embodiments proposed in this disclosure can also be applied to autoregressive models in real-world scenarios, and have been verified to bring significant performance improvements. For example, combining Tables 1 and 2 above, the average BLEU score of this solution is approximately 29.5 across all languages, achieving an inference speedup of approximately 5.67 times, reaching a level comparable to or even better than autoregressive models. Therefore, this solution has a certain degree of universality.

[0112] It should be understood that in the embodiments of this disclosure, "first," "second," "third," etc., are only used to indicate that multiple objects may be different, but at the same time, it does not exclude that two objects are the same, and should not be interpreted as any limitation on the embodiments of this disclosure.

[0113] It should also be understood that the manner, situation, category and division of embodiments in the present disclosure are for the convenience of description only and should not constitute a special limitation. Various manners, categories, situations and features in the embodiments can be combined with each other where logically consistent.

[0114] It should also be understood that the foregoing is merely to help those skilled in the art better understand the embodiments of this disclosure, and is not intended to limit the scope of the embodiments of this disclosure. Those skilled in the art can make various modifications, variations, or combinations based on the foregoing. Such modifications, variations, or combinations are also within the scope of the embodiments of this disclosure.

[0115] It should also be understood that the above description focuses on highlighting the differences between the various embodiments. Similarities or commonalities can be referenced or learned from each other, and for the sake of brevity, they will not be repeated here.

[0116] Figure 7 A schematic block diagram of an example device 700 according to some embodiments of the present disclosure is shown. Device 700 can be implemented by software, hardware, or a combination of both. Figure 7 As shown, the device 700 includes a model acquisition unit 710 and an output determination unit 720.

[0117] Model acquisition unit 710 is configured to acquire a trained speech translation model, wherein the trained speech translation model includes an acoustic encoder and a text encoder, wherein the acoustic encoder and / or text encoder includes an intermediate CTC module between adjacent first and second layers, the intermediate CTC module being used to determine the input of the second layer based on the output of the first layer and the word embedding matrix. Output determination unit 720 is configured to input the source language speech to be processed into the trained speech translation model to obtain the corresponding target language text.

[0118] In some embodiments, during the process of the output determining unit 720 obtaining the target language text, the output determining unit 720 may be configured to: determine the CTC distribution of the output of the first layer; generate a bias associated with the output of the first layer based on the CTC distribution and the word embedding matrix; and integrate the output of the first layer with the bias to determine the input of the second layer.

[0119] For example, at least one layer of the text encoder includes a cross-attention module, where the keys and values ​​across the attention module are based on the output of a reference layer of the text encoder. Optionally, at least one layer of the text encoder includes each layer located after the reference layer of the text encoder.

[0120] For example, each layer of the acoustic encoder and text encoder includes a self-attention module. Optionally, each layer of the acoustic encoder and text encoder also includes an FFN.

[0121] In some embodiments, such as Figure 7 As shown, the device 700 may further include a training unit 705 configured to: construct a training dataset, the training dataset including multiple data items, each of the multiple data items including source language speech samples and corresponding target language text samples; and generate a trained speech translation model based on the training dataset.

[0122] In some examples, a specific layer of the text encoder includes a self-attention module and a cross-layer attention module, and the training unit 705 can be configured to discard the self-attention module in a specific layer of the text encoder based on randomly generated probabilities.

[0123] In some examples, the text encoder includes an intermediate CTC module located between adjacent first and second intermediate layers, and wherein training unit 705 can be configured to: determine the CTC prediction distribution of the output of the first intermediate layer; generate a bias associated with the output of the first intermediate layer based on the CTC prediction distribution and the word embedding matrix; replace at least one of a plurality of incorrectly predicted positions in the bias with the correct output according to the replacement rate to obtain a replaced bias; and integrate the output of the first intermediate layer with the replaced bias to determine the input of the second intermediate layer.

[0124] Optionally, at least one position is determined through random sampling. Optionally, the word embedding matrix is ​​obtained through random initialization in the initial stage and is iteratively updated in the training stage.

[0125] In some examples, training unit 705 can be configured to: construct a loss function, which includes a first loss function associated with an intermediate CTC module in the acoustic encoder and a second loss function associated with an intermediate CTC module in the text encoder; and generate a trained speech translation model based on the loss function.

[0126] Figure 7 The device 700 can be used to achieve the above-mentioned combination. Figures 1A to 6 For the sake of brevity, the process described will not be repeated here.

[0127] The division of modules or units in the embodiments of this disclosure is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the disclosed embodiments may be integrated into one unit, exist as separate physical entities, or two or more units may be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional unit.

[0128] Figure 8 A block diagram of an example device 800 that can be used to implement embodiments of the present disclosure is shown. It should be understood that... Figure 8 The device 800 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementation described herein. For example, device 800 can be used to perform the functions described above. Figures 1A to 6 The process described.

[0129] like Figure 8 As shown, device 800 is in the form of a general-purpose computing device. Components of computing device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage devices 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. Processing unit 810 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 820. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 800.

[0130] Computing device 800 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 830 can be removable or non-removable media and may include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data (e.g., training data for training) and accessible within computing device 800.

[0131] The computing device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 8 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 820 may include computer program product 825 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0132] The communication unit 840 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 800 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0133] Input device 850 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 860 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 800 can also communicate with one or more external devices (not shown) via communication unit 840 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 800, or with any device that enables computing device 800 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).

[0134] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described above.

[0135] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0137] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0139] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A speech processing method, comprising: A trained speech translation model is obtained, wherein the trained speech translation model includes an acoustic encoder and a text encoder, and the trained speech translation model does not include a decoder, wherein the acoustic encoder and / or the text encoder includes an intermediate connection temporal classification (CTC) module between adjacent first and second layers, the intermediate CTC module being used to determine the input of the second layer based on the output of the first layer and a word embedding matrix, wherein at least one layer of the text encoder includes a cross-layer attention module, the keys and values ​​of the cross-layer attention module being based on the output of a reference layer of the text encoder, wherein the reference layer is a pre-defined layer in the multiple layers of the text encoder that does not include the cross-layer attention module; as well as The source language speech to be processed is input into the trained speech translation model to obtain the corresponding target language text.

2. The method according to claim 1, wherein in obtaining the target language text, the method includes: Determine the CTC distribution of the output of the first layer; Based on the CTC distribution and word embedding matrix, a bias associated with the output of the first layer is generated; as well as The output of the first layer is integrated with the bias to determine the input of the second layer.

3. The method of claim 1, wherein at least one layer of the text encoder comprises each layer located after the reference layer of the text encoder.

4. The method of claim 1, wherein each layer of the acoustic encoder and the text encoder includes a self-attention module.

5. The method of claim 1, wherein the trained speech translation model is generated by means of: Construct a training dataset, wherein the training dataset includes multiple data items, each of which includes source language speech samples and corresponding target language text samples; and The trained speech translation model is generated based on the training dataset.

6. The method of claim 5, wherein a particular layer of the text encoder further comprises a self-attention module, the particular layer being at least one of the layers located after the reference layer and containing the cross-layer attention module, and wherein generating the trained speech translation model comprises: The self-attention module in the specific layer of the text encoder is discarded based on the randomly generated probability.

7. The method of claim 5, wherein the text encoder includes an intermediate CTC module located between adjacent first and second intermediate layers, and wherein generating the trained speech translation model comprises: Determine the CTC prediction distribution of the output of the first intermediate layer; Based on the CTC prediction distribution and word embedding matrix, a bias associated with the output of the first intermediate layer is generated; According to the replacement rate, at least one of the multiple positions in the bias that are predicted incorrectly is replaced with the correct output to obtain the replaced bias; as well as The output of the first intermediate layer is integrated with the replaced bias to determine the input of the second intermediate layer.

8. The method of claim 7, wherein the at least one location is determined by random sampling.

9. The method of claim 7, wherein the word embedding matrix is ​​obtained by random initialization in the initial stage and is iteratively updated in the training stage.

10. The method of claim 5, wherein generating the trained speech translation model comprises: Construct a loss function, the loss function including a first loss function associated with an intermediate CTC module in the acoustic encoder and a second loss function associated with an intermediate CTC module in the text encoder; and The trained speech translation model is generated based on the loss function.

11. An electronic device, comprising: At least one processing unit; At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform an action when executed by the at least one processing unit, the action including: A trained speech translation model is obtained, wherein the trained speech translation model includes an acoustic encoder and a text encoder, and the trained speech translation model does not include a decoder, wherein the acoustic encoder and / or the text encoder includes an intermediate connection temporal classification CTC module between adjacent first and second layers, the intermediate CTC module being used to determine the input of the second layer based on the output of the first layer and a word embedding matrix, wherein at least one layer of the text encoder includes a layer-to-layer attention module, the keys and values ​​of the layer-to-layer attention module being based on the output of a reference layer of the text encoder, wherein the reference layer is a pre-defined layer in the multiple layers of the text encoder that does not include the layer-to-layer attention module; as well as The source language speech to be processed is input into the trained speech translation model to obtain the corresponding target language text.

12. A voice processing device, comprising: A model acquisition unit is configured to acquire a trained speech translation model, wherein the trained speech translation model includes an acoustic encoder and a text encoder, and the trained speech translation model does not include a decoder, wherein the acoustic encoder and / or the text encoder includes an intermediate connection temporal classification (CTC) module between adjacent first and second layers, the intermediate CTC module being used to determine the input of the second layer based on the output of the first layer and a word embedding matrix, wherein at least one layer of the text encoder includes a cross-layer attention module, the keys and values ​​of the cross-layer attention module being based on the output of a reference layer of the text encoder, wherein the reference layer is a pre-defined layer in the multiple layers of the text encoder that does not include the cross-layer attention module; as well as The output determination unit is configured to input the source language speech to be processed into the trained speech translation model to obtain the corresponding target language text.

13. A computer-readable storage medium having a computer program stored thereon, the program, when executed by a processor, implementing the method according to any one of claims 1 to 10.

Images