Data processing method and device, equipment and computer readable storage medium

Abstract

The application discloses a data processing method, device and equipment and a computer readable storage medium. The method comprises the following steps: obtaining sample voice information and sample text information; processing the sample voice information through an acoustic model in an initial recognition model to obtain acoustic feature information; processing the acoustic feature information and the sample text information through a translation model in the initial recognition model to obtain a first predicted translation result and a second predicted translation result respectively; training the initial recognition model based on the first predicted translation result and the second predicted translation result to obtain a target recognition model; and the K vector and the V vector in the acoustic model are spliced with a prefix vector and / or the K vector and the V vector in the translation model are spliced with a prefix vector. The training efficiency of the target recognition model is improved.

Description

Data processing methods, apparatus, equipment and computer-readable storage media Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a data processing method, apparatus, device and computer-readable storage medium. Background Technology

[0002] In related technologies, speech-to-text translation methods are generally based on a cascaded system composed of an automatic speech recognition model and a machine translation model, i.e., a cascaded model implementation. The model training process involves a large amount of parameter adjustment, the model training speed is slow, and the training efficiency is low. Summary of the Invention

[0003] This application provides an implementation scheme that differs from related technologies to address the technical problems in related technologies, such as the large amount of parameter adjustment involved in the model training process, the slow training speed, and the low training efficiency when training a cascaded model for determining text translation results based on speech.

[0004] Firstly, this application provides a data processing method, including:

[0005] Obtain the sample speech information and the corresponding sample text information;

[0006] The sample speech information is processed by the acoustic model in the initial recognition model to obtain acoustic feature information;

[0007] The acoustic feature information is processed by the translation model in the initial recognition model to obtain the first predicted translation result corresponding to the sample speech information;

[0008] The sample text information is processed by the translation model to obtain a second predicted translation result corresponding to the sample text information;

[0009] The initial recognition model is trained based on the first predicted translation result and the second predicted translation result to obtain a target recognition model. The target recognition model is used to recognize the text translation result of the input speech information to be recognized.

[0010] Wherein, the acoustic model and / or the translation model are pre-trained models, and the acoustic model and the translation model satisfy at least one of the following conditions:

[0011] In the acoustic model, the first K vector and the first V vector in the first self-attention mechanism unit of at least one of the multiple first data processing layers are respectively concatenated with a first prefix vector and a second prefix vector.

[0012] In the translation model, the second K vector and the second V vector in the second self-attention mechanism unit of at least one of the multiple third data processing layers are respectively concatenated with a third prefix vector and a fourth prefix vector.

[0013] Secondly, this application provides a data processing apparatus, comprising:

[0014] The acquisition unit is used to acquire sample speech information and sample text information corresponding to the sample speech information;

[0015] The processing unit is used to process the sample speech information through the acoustic model in the initial recognition model to obtain acoustic feature information;

[0016] The processing unit is further configured to process the acoustic feature information through the translation model in the initial recognition model to obtain a first predicted translation result corresponding to the sample speech information;

[0017] The processing unit is further configured to process the sample text information through the translation model to obtain a second predicted translation result corresponding to the sample text information;

[0018] The training unit is used to train the initial recognition model based on the first predicted translation result and the second predicted translation result to obtain a target recognition model. The target recognition model is used to recognize the text translation result of the input speech information to be recognized.

[0019] Wherein, the acoustic model and / or the translation model are pre-trained models, and the acoustic model and the translation model satisfy at least one of the following conditions:

[0020] In the acoustic model, the first K vector and the first V vector in the first self-attention mechanism unit of at least one of the multiple first data processing layers are respectively concatenated with a first prefix vector and a second prefix vector.

[0021] In the translation model, the second K vector and the second V vector in the second self-attention mechanism unit of at least one of the multiple third data processing layers are respectively concatenated with a third prefix vector and a fourth prefix vector.

[0022] Thirdly, this application provides an electronic device, comprising:

[0023] Processor; and

[0024] Memory for storing the executable instructions of the processor;

[0025] The processor is configured to execute the first aspect or any of the possible implementations of the first aspect by executing the executable instructions.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods in the first aspect or any of the possible implementations of the first aspect.

[0027] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the methods in the first aspect or any possible implementations of the first aspect.

[0028] The solution provided in this application adds prefix vectors to the self-attention mechanism units of the pre-trained models, i.e., the acoustic model and the translation model, with their original parameters fixed, and trains based on prefix-tuning. The solution adds additional information to the key structures in the pre-trained model, enabling the training of a target recognition model with good translation quality with fewer parameter adjustments. It achieves the training of a high-performance target recognition model with a small number of parameter adjustments, thus improving the training efficiency of the target recognition model. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0030] Figure 1 is a schematic diagram of the system provided in an embodiment of this application;

[0031] Figure 2a is a schematic flowchart of a data processing method provided in an embodiment of this application;

[0032] Figure 2b is a schematic diagram of the data processing process through the second data processing layer according to an embodiment of this application;

[0033] Figure 2c is a schematic diagram of the data processing process through the second data processing layer provided in another embodiment of this application;

[0034] Figure 2d is a schematic diagram of the structure of an initial recognition model provided in an embodiment of this application;

[0035] Figure 2e is a graph comparing the translation quality of the target recognition model trained by the scheme of this application with that trained by other methods in related technologies;

[0036] Figure 3 is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0037] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application are described in detail below, with examples of these embodiments illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0039] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] First, some terms used in the embodiments of this application will be explained below to facilitate understanding by those skilled in the art.

[0041] The attention mechanism actually originates from the human body's ability to process external information. Because the amount of information a person receives at any given moment is incredibly vast and complex, far exceeding the processing capacity of the human brain, when processing information, a person will focus their attention on the information that needs attention and filter out other irrelevant external information. This processing method is called the attention mechanism.

[0042] A feedforward neural network is the simplest type of neural network, with neurons arranged in layers. Each neuron is connected only to neurons in the layer preceding it. It receives the output of the previous layer and outputs it to the next layer, with no feedback between layers.

[0043] Acoustic features refer to the high-dimensional vector representation output by an acoustic model (neural network).

[0044] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0045] Figure 1 is a schematic diagram of a system structure provided in an exemplary embodiment of this application. The structure includes a translation device 10 and a model training device 20, wherein the model training device 20 is used to: acquire sample speech information and sample text information corresponding to the sample speech information; process the sample speech information through an acoustic model in an initial recognition model to obtain acoustic feature information; process the acoustic feature information through a translation model in the initial recognition model to obtain a first predicted translation result corresponding to the sample speech information; process the sample text information through the translation model to obtain a second predicted translation result corresponding to the sample text information; train the initial recognition model based on the first predicted translation result and the second predicted translation result to obtain a target recognition model, wherein the target recognition model is used to recognize the text translation result of the input speech information to be recognized; wherein the acoustic model and / or the translation model are pre-trained models, and the acoustic model and the translation model satisfy at least one of the following conditions:

[0046] In the acoustic model, the first K vector and the first V vector in the first self-attention mechanism unit of at least one of the multiple first data processing layers are respectively concatenated with a first prefix vector and a second prefix vector.

[0047] In the translation model, the second K vector and the second V vector in the second self-attention mechanism unit of at least one of the multiple third data processing layers are respectively concatenated with a third prefix vector and a fourth prefix vector.

[0048] The translation device 10 is used to acquire the speech information to be recognized after the target recognition model has been trained; and to identify the text translation result of the speech information to be recognized through the target recognition model.

[0049] In some embodiments, the translation device 10 and the model training device 20 may be the same device.

[0050] The execution principle and interaction process of each functional module in this system embodiment can be found in the following descriptions of the method embodiments.

[0051] Figure 2a is a flowchart illustrating a data processing method provided in an exemplary embodiment of this application. The execution entity of this method can be the aforementioned model training device 20, and the method includes at least the following steps S201-S205:

[0052] S201. Obtain the sample speech information and the sample text information corresponding to the sample speech information;

[0053] Optionally, the sample speech information is untranslated speech information in a certain language, and the sample text information is the untranslated text information of the sample speech information.

[0054] S202. The sample speech information is processed using the acoustic model in the initial recognition model to obtain acoustic feature information;

[0055] Optionally, the acoustic model can be wav2vec2.0.

[0056] Optionally, the acoustic model is trained based on unlabeled speech data.

[0057] S203. The acoustic feature information is processed by the translation model in the initial recognition model to obtain the first predicted translation result corresponding to the sample speech information;

[0058] Optionally, the translation model is used to translate the input information to be translated to obtain the translation result.

[0059] S204. The sample text information is processed by the translation model to obtain the second predicted translation result corresponding to the sample text information;

[0060] S205. The initial recognition model is trained based on the first predicted translation result and the second predicted translation result to obtain a target recognition model. The target recognition model is used to recognize the text translation result of the speech information to be recognized based on the input speech information to be recognized.

[0061] Wherein, the acoustic model and / or the translation model are pre-trained models, and the acoustic model and the translation model satisfy at least one of the following conditions:

[0062] In the acoustic model, the first K vector and the first V vector in the first self-attention mechanism unit of at least one of the multiple first data processing layers are respectively concatenated with a first prefix vector and a second prefix vector.

[0063] In the translation model, the second K vector and the second V vector in the second self-attention mechanism unit of at least one of the multiple third data processing layers are respectively concatenated with a third prefix vector and a fourth prefix vector.

[0064] In some embodiments, the acoustic model and the translation model also satisfy at least one of the following conditions:

[0065] A first processing unit is connected between the input and output of the first feedforward neural network in at least one second data processing layer. The first processing unit includes two fully connected layers and a nonlinear activation function.

[0066] A second processing unit is connected between the input and output of the second feedforward neural network in at least one fourth data processing layer; the second processing unit includes two fully connected layers and a nonlinear activation function.

[0067] In some embodiments, in the aforementioned S202, the sample speech information is processed using the acoustic model in the initial recognition model to obtain acoustic feature information, including:

[0068] For each of the multiple first data processing layers in the acoustic model, the first input information of the first data processing layer is processed by the first data processing layer to obtain the first output information of the first data processing layer.

[0069] In this process, the output of any first data processing layer is the input of the next adjacent first data processing layer, the input of the first first data processing layer is the sample speech information, and the first output information of the last first data processing layer is acoustic feature information.

[0070] In some embodiments, the acoustic model includes a first convolutional neural network.

[0071] In some embodiments, the sample speech information processed by the first data processing layer is the sample speech information that has been processed by the first convolutional neural network.

[0072] In some embodiments, the first data processing layer includes a first self-attention mechanism unit, a first normalization unit, a first feedforward neural network, and a second normalization unit; regarding the second data processing layer, as shown in FIG2b, the second data processing layer processes the second input information of the second data processing layer to obtain the second output information of the second data processing layer, including the following S1-S4:

[0073] S1. The second input information is processed by the first self-attention mechanism unit in the second data processing layer to obtain the first processing result;

[0074] In the first self-attention mechanism unit of the second data processing layer, the first K vector and the first V vector are respectively concatenated with a first prefix vector and a second prefix vector;

[0075] Specifically, in Figure 2b, Pk is the first prefix vector and Pk is the second prefix vector.

[0076] S2. The first processing result and the second input information are processed by the first normalization unit in the second data processing layer to obtain the first normalization result;

[0077] In some embodiments, in S2, the first processing result and the second input information are processed by the first normalization unit in the second data processing layer to obtain a first normalization result, including:

[0078] The first normalization unit sums the first processing result and the second input information, and normalizes the summation result to obtain the first normalization result.

[0079] S3. The first normalization result is processed by the first feedforward neural network in the second data processing layer to obtain the second processing result;

[0080] S4. The first normalization result and the second processing result are processed by the second normalization unit to obtain the second output information of the second data processing layer.

[0081] In some embodiments, in S4, the first normalization result and the second processing result are processed by the second normalization unit to obtain the second output information of the second data processing layer, including:

[0082] The first normalization result and the second processing result are summed by the second normalization unit to obtain the first summation result; and the first summation result is normalized to obtain the second output information of the second data processing layer.

[0083] In other embodiments, referring to FIG2c, the second data processing layer processes the second input information of the second data processing layer to obtain the second output information of the second data processing layer, including the following S01-S05:

[0084] S01. The second input information is processed by the first self-attention mechanism unit in the second data processing layer to obtain the first processing result;

[0085] In the first self-attention mechanism unit of the second data processing layer, the first K vector and the first V vector are respectively concatenated with a first prefix vector and a second prefix vector;

[0086] Specifically, in Figure 2c, Pk is the first prefix vector and Pk is the second prefix vector.

[0087] S02. The first processing result and the second input information are processed by the first normalization unit in the second data processing layer to obtain the first normalization result;

[0088] In some embodiments, in S02, the first processing result and the second input information are processed by the first normalization unit in the second data processing layer to obtain a first normalization result, including:

[0089] The first normalization unit sums the first processing result and the second input information, and normalizes the summation result to obtain the first normalization result.

[0090] S03. The first normalization result is processed by the first feedforward neural network in the second data processing layer to obtain the second processing result;

[0091] S04. The first normalization result is processed by the first processing unit to obtain the third processing result;

[0092] The first processing unit may include a normalization unit for normalizing the first normalization result, a dimensionality reduction unit for reducing the dimensionality of the output of the normalization unit (the dimensionality reduction unit being a fully connected layer), a nonlinear activation function for processing the dimensionality reduction result, and a dimensionality increase unit for increasing the dimensionality of the output of the nonlinear activation function. The output of the dimensionality increase unit is the third processing result. The dimensionality increase unit is also a fully connected layer.

[0093] S05. The first normalization result, the second processing result, and the third processing result are processed by the second normalization unit to obtain the second output information of the second data processing layer.

[0094] In some embodiments, in S05, the first normalization result, the second processing result, and the third processing result are processed by the second normalization unit to obtain the second output information of the second data processing layer, including:

[0095] The first normalization result, the second processing result, and the third processing result are summed by the second normalization unit to obtain the first summation result; and the first summation result is normalized to obtain the second output information of the second data processing layer.

[0096] In some embodiments, in the aforementioned S203, processing the acoustic feature information using the translation model in the initial recognition model to obtain a first predicted translation result corresponding to the sample speech information includes:

[0097] For each of the plurality of third data processing layers, the third input information of the third data processing layer is processed by the third data processing layer to obtain the third output information of the third data processing layer.

[0098] In this process, the output of any third data processing layer is the input of the next adjacent third data processing layer, the input of the first third data processing layer is the acoustic feature information, and the third output information of the last third data processing layer is the first predicted translation result.

[0099] In some embodiments, the third data processing layer includes a second self-attention mechanism unit, a third normalization unit, a second feedforward neural network, and a fourth normalization unit; for the fourth data processing layer, the fourth input information of the fourth data processing layer is processed by the fourth data processing layer to obtain the fourth output information of the fourth data processing layer, including the following S11-S15:

[0100] S11. The fourth input information is processed by the second self-attention mechanism unit in the fourth data processing layer to obtain the fourth processing result;

[0101] S12. The fourth processing result and the fourth input information are processed by the third normalization unit in the fourth data processing layer to obtain the second normalization result;

[0102] In some embodiments, in S12, the fourth processing result and the fourth input information are processed by the third normalization unit in the fourth data processing layer to obtain a second normalization result, including:

[0103] The third normalization unit sums the fourth processing result and the fourth input information, and normalizes the summation result to obtain the second normalization result.

[0104] S13. The second normalization result is processed by the second feedforward neural network in the fourth data processing layer to obtain the fifth processing result;

[0105] S14. The second normalization result and the fifth processing result are processed by the fourth normalization unit to obtain the fourth output information of the fourth data processing layer.

[0106] In some embodiments, in S14, the second normalization result and the fifth processing result are processed by the fourth normalization unit to obtain the fourth output information of the fourth data processing layer, including:

[0107] The second normalization result and the fifth processing result are summed by the fourth normalization unit to obtain the corresponding summation result, and the summation result is normalized to obtain the fourth output information of the fourth data processing layer.

[0108] In other embodiments, for the fourth data processing layer, the fourth input information of the fourth data processing layer is processed by the fourth data processing layer to obtain the fourth output information of the fourth data processing layer, including the following S001-S005:

[0109] S001. The fourth input information is processed by the second self-attention mechanism unit in the fourth data processing layer to obtain the fourth processing result;

[0110] S002. The fourth processing result and the fourth input information are processed by the third normalization unit in the fourth data processing layer to obtain the second normalization result.

[0111] In some embodiments, in S002, the fourth processing result and the fourth input information are processed by the third normalization unit in the fourth data processing layer to obtain a second normalization result, including:

[0112] The third normalization unit sums the fourth processing result and the fourth input information, and normalizes the summation result to obtain the second normalization result.

[0113] S003. The second normalization result is processed by the second feedforward neural network in the fourth data processing layer to obtain the fifth processing result;

[0114] S004. The second normalization result is processed by the second processing unit to obtain the sixth processing result;

[0115] The second processing unit may include a normalization unit for normalizing the second normalization result, a dimensionality reduction unit for reducing the dimensionality of the output of the normalization unit (the dimensionality reduction unit being a fully connected layer), a nonlinear activation function for processing the dimensionality reduction result, and a dimensionality increase unit for increasing the dimensionality of the output of the nonlinear activation function. The output of the dimensionality increase unit is the third processing result. The dimensionality increase unit is also a fully connected layer.

[0116] S005. The second normalization result, the fifth processing result, and the sixth processing result are processed by the fourth normalization unit to obtain the fourth output information of the fourth data processing layer.

[0117] In some embodiments, in S005, the second normalization result, the fifth processing result, and the sixth processing result are processed by the fourth normalization unit to obtain the fourth output information of the fourth data processing layer, including:

[0118] The fourth normalization unit sums the second normalization result, the fifth processing result, and the sixth processing result to obtain the corresponding summation result. Then, the summation result is normalized to obtain the fourth output information of the fourth data processing layer.

[0119] In some embodiments, in S204 above, the sample text information is processed by the translation model in the initial recognition model to obtain a second predicted translation result corresponding to the sample text information, including:

[0120] For each of the plurality of third data processing layers, the third input information of the third data processing layer is processed by the third data processing layer to obtain the third output information of the third data processing layer.

[0121] In this process, the output of any third data processing layer is the input of the next adjacent third data processing layer, the input of the first third data processing layer is the sample text information, and the third output information of the last third data processing layer is the second predicted translation result.

[0122] In some embodiments, when the fourth data processing layer is used to process sample text information, the third prefix vector concatenated by the second K vector in the second self-attention mechanism unit of the fourth data processing layer is different from the third prefix vector concatenated by the second K vector in the second self-attention mechanism unit of the fourth data processing layer when the fourth data processing layer is used to process acoustic feature information.

[0123] In some embodiments, when the fourth data processing layer is used to process sample text information, the fourth prefix vector concatenated by the second V vector in the second self-attention mechanism unit of the fourth data processing layer is different from the fourth prefix vector concatenated by the second V vector in the second self-attention mechanism unit of the fourth data processing layer when the fourth data processing layer is used to process acoustic feature information.

[0124] In some embodiments, when the fourth data processing layer is used to process sample text information, the second processing unit connected between the input and output of the second feedforward neural network in the fourth data processing layer is different from the second processing unit connected between the input and output of the second feedforward neural network in the fourth data processing layer when the fourth data processing layer is used to process acoustic feature information.

[0125] In some embodiments, before processing the acoustic feature information through a translation model in an initial recognition model to obtain a first predicted translation result corresponding to the sample speech information, the method further includes: performing downsampling processing on the acoustic feature information to obtain processed acoustic feature information.

[0126] This method can further reduce the amount of data processing and further improve the training efficiency of the target recognition model.

[0127] Specifically, as shown in Figure 2d, the acoustic feature information can be downsampled using a convolutional neural network to obtain the processed acoustic feature information. The translation model in this application can have a Transformer-Encoder (the second encoding unit in Figure 2d) and a Transformer Decoder (the decoding unit in Figure 2d). The aforementioned fourth data processing layer can be either the third data processing layer in the second encoding unit or the third data processing layer in the decoding unit.

[0128] In some embodiments, the acoustic model can be structured as a CNN and a Transformer Encoder (the first encoding unit in Figure 2d).

[0129] It should be noted that the second data processing layer in this application is one of a plurality of first data processing layers, and the fourth data processing layer is one of a third data processing layer.

[0130] It should be noted that the acoustic model and translation model in this application also include other units in related technologies, which will not be described in detail here.

[0131] In some embodiments, in the aforementioned S205, the initial recognition model is trained based on the first predicted translation result and the second predicted translation result to obtain a target recognition model, including S2051-S2055:

[0132] S2051. Obtain the first target label corresponding to the first predicted translation result and the second target label corresponding to the second predicted translation result;

[0133] In some embodiments, the first target label and the second target label may be set by relevant personnel.

[0134] S2052. Determine the first loss information corresponding to the first predicted translation result using a preset loss function and the first predicted translation result;

[0135] S2053. Determine the second loss information corresponding to the second predicted translation result using the loss function and the second predicted translation result;

[0136] In some embodiments, the loss function in this application may be the cross-entropy function.

[0137] S2054. Determine the target loss information based on the first loss information and the second loss information;

[0138] In some embodiments, determining the target loss information based on the first loss information and the second loss information may include: using the sum of the first loss information and the second loss information as the target loss information.

[0139] S2055. If the target loss information is less than a preset threshold, then the initial recognition model is used as the target recognition model; otherwise, the parameters of any one or more of the following are adjusted according to the target loss information: the first prefix vector, the second prefix vector, the first processing unit, the third prefix vector, the fourth prefix vector, and the second processing unit, and the process is returned to execute the processing of the sample speech information through the acoustic model in the initial recognition model to obtain acoustic feature information. In some embodiments, the parameters of any one or more of the following can also be adjusted according to the target loss information: the first normalization unit, the second normalization unit, the third normalization unit, and the fourth normalization unit.

[0140] In some embodiments, the method further includes: acquiring speech information to be recognized; and identifying the text translation result of the speech information to be recognized using the target recognition model.

[0141] Table 1 compares the translation quality of the target recognition model trained using the proposed method with that trained using other methods in related technologies, based on the MUSTC dataset commonly used in speech translation. As shown in Table 1, the proposed method can train a target recognition model with significantly higher translation quality than models trained using other methods in related technologies, even with fewer parameter adjustments.

[0142] Table 1: Comparison of translation quality between the target recognition model trained using the scheme of this application and models trained using other methods in related technologies.

[0143]

[0144] Further, as shown in Figure 2e, which is a graph comparing the translation quality of the target recognition model trained using the scheme of this application with models trained using other methods in related technologies, neither the first nor the second related model incorporates the prefix vectors of this scheme and neither contains a first or second processing unit. The first related model has fewer data processing layers than the second related model. As can be seen from Figure 2e, the scheme of this application achieves higher translation quality compared to related models.

[0145] The proposed solution achieves speech translation transfer learning based on the pre-trained model by adding parameter-efficient components (such as a first processing unit and / or a second processing monocular unit) with fixed original parameters in the acoustic and translation models. For Attention in the Transformer Block, this solution concatenates a prefix vector with the K and V vectors and trains based on prefix-tuning. The Transformer Encoder accepts inputs from different modalities, namely sound feature information and sample text information. This solution adds additional processing units to the critical structures in the pre-trained model, enabling faster training of a target recognition model with good translation quality with fewer parameter adjustments. Furthermore, this solution can adjust the parameters of the normalization unit in the pre-trained acoustic model and / or the normalization unit in the pre-trained translation model, achieving high-performance target recognition model training with a relatively small number of parameters. The proposed solution exhibits better data efficiency under low-resource conditions and maintains good performance even when data is extremely scarce.

[0146] Figure 3 is a schematic diagram of a data processing apparatus provided in an exemplary embodiment of this application. The data processing apparatus includes:

[0147] Acquisition unit 31 is used to acquire sample speech information and sample text information corresponding to the sample speech information;

[0148] Processing unit 32 is used to process the sample speech information through the acoustic model in the initial recognition model to obtain acoustic feature information;

[0149] The processing unit 32 is further configured to process the acoustic feature information through the translation model in the initial recognition model to obtain the first predicted translation result corresponding to the sample speech information;

[0150] The processing unit 32 is further configured to process the sample text information through the translation model to obtain a second predicted translation result corresponding to the sample text information;

[0151] Training unit 33 is used to train the initial recognition model based on the first predicted translation result and the second predicted translation result to obtain a target recognition model. The target recognition model is used to recognize the text translation result of the input speech information to be recognized.

[0152] Wherein, the acoustic model and / or the translation model are pre-trained models, and the acoustic model and the translation model satisfy at least one of the following conditions:

[0153] In the acoustic model, the first K vector and the first V vector in the first self-attention mechanism unit of at least one of the multiple first data processing layers are respectively concatenated with a first prefix vector and a second prefix vector.

[0154] In the translation model, the second K vector and the second V vector in the second self-attention mechanism unit of at least one of the multiple third data processing layers are respectively concatenated with a third prefix vector and a fourth prefix vector.

[0155] In some embodiments, the acoustic model and the translation model also satisfy at least one of the following conditions:

[0156] A first processing unit is connected between the input and output of the first feedforward neural network in at least one second data processing layer. The first processing unit includes two fully connected layers and a nonlinear activation function.

[0157] A second processing unit is connected between the input and output of the second feedforward neural network in at least one fourth data processing layer; the second processing unit includes two fully connected layers and a nonlinear activation function.

[0158] In some embodiments, when the aforementioned apparatus processes the sample speech information using an acoustic model in an initial recognition model to obtain acoustic feature information, it is specifically used for:

[0159] For each of the multiple first data processing layers in the acoustic model, the first input information of the first data processing layer is processed by the first data processing layer to obtain the first output information of the first data processing layer.

[0160] In this process, the output of any first data processing layer is the input of the next adjacent first data processing layer, the input of the first first data processing layer is the sample speech information, and the first output information of the last first data processing layer is acoustic feature information.

[0161] In some embodiments, the first data processing layer includes a first self-attention mechanism unit, a first normalization unit, a first feedforward neural network, and a second normalization unit; regarding the second data processing layer, when the aforementioned apparatus processes the second input information of the second data processing layer through the second data processing layer to obtain the second output information of the second data processing layer, it is specifically used for:

[0162] The second input information is processed by the first self-attention mechanism unit in the second data processing layer to obtain the first processing result;

[0163] The first processing result and the second input information are processed by the first normalization unit in the second data processing layer to obtain the first normalization result;

[0164] The first normalization result is processed by the first feedforward neural network in the second data processing layer to obtain the second processing result;

[0165] The first processing unit processes the first normalization result to obtain the third processing result;

[0166] The second normalization unit processes the first normalization result, the second processing result, and the third processing result to obtain the second output information of the second data processing layer.

[0167] In some embodiments, the aforementioned transposition, when used to process the acoustic feature information through a translation model in an initial recognition model to obtain a first predicted translation result corresponding to the sample speech information, is specifically used for:

[0168] For each of the plurality of third data processing layers, the third input information of the third data processing layer is processed by the third data processing layer to obtain the third output information of the third data processing layer.

[0169] In this process, the output of any third data processing layer is the input of the next adjacent third data processing layer, the input of the first third data processing layer is the acoustic feature information, and the third output information of the last third data processing layer is the first predicted translation result.

[0170] In some embodiments, the third data processing layer includes a second self-attention mechanism unit, a third normalization unit, a second feedforward neural network, and a fourth normalization unit; the aforementioned apparatus, when used to process the fourth input information of the fourth data processing layer to obtain the fourth output information of the fourth data processing layer, is specifically used for:

[0171] The fourth input information is processed by the second self-attention mechanism unit in the fourth data processing layer to obtain the fourth processing result;

[0172] The fourth processing result and the fourth input information are processed by the third normalization unit in the fourth data processing layer to obtain the second normalization result;

[0173] The second normalization result is processed by the second feedforward neural network in the fourth data processing layer to obtain the fifth processing result;

[0174] The second processing unit processes the second normalization result to obtain the sixth processing result;

[0175] The fourth normalization unit processes the second normalization result, the fifth processing result, and the sixth processing result to obtain the fourth output information of the fourth data processing layer.

[0176] In some embodiments, before processing the acoustic feature information using a translation model in the initial recognition model to obtain a first predicted translation result corresponding to the sample speech information, the aforementioned apparatus is further configured to:

[0177] The acoustic feature information is downsampled to obtain the processed acoustic feature information.

[0178] In some embodiments, when the aforementioned apparatus is used to train the initial recognition model based on the first predicted translation result and the second predicted translation result to obtain a target recognition model, it is specifically used for:

[0179] Obtain the first target label corresponding to the first predicted translation result, and the second target label corresponding to the second predicted translation result;

[0180] The first loss information corresponding to the first predicted translation result is determined by using a preset loss function and the first predicted translation result;

[0181] The second loss information corresponding to the second predicted translation result is determined by using the loss function and the second predicted translation result;

[0182] Determine the target loss information based on the first loss information and the second loss information;

[0183] If the target loss information is less than a preset threshold, the initial recognition model is used as the target recognition model; otherwise, the parameters of any one or more of the following are adjusted according to the target loss information: the first prefix vector, the second prefix vector, the first processing unit, the third prefix vector, the fourth prefix vector, and the second processing unit, and the process is returned to execute the acoustic model in the initial recognition model to process the sample speech information to obtain acoustic feature information.

[0184] In some embodiments, the aforementioned apparatus is further configured to:

[0185] Obtain the speech information to be recognized;

[0186] The target recognition model identifies the text translation result of the speech information to be recognized.

[0187] For specific implementation details of this device embodiment, please refer to the foregoing content, which will not be repeated here.

[0188] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, they will not be repeated here. Specifically, the device can execute the above method embodiments, and the foregoing and other operations and / or functions of each module in the device correspond to the corresponding processes in the various methods in the above method embodiments, which will not be repeated here for the sake of brevity.

[0189] The apparatus of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0190] Figure 4 is a schematic block diagram of an electronic device provided in an embodiment of this application. The electronic device may include:

[0191] The system includes a memory 401 and a processor 402. The memory 401 stores computer programs and transfers the program code to the processor 402. In other words, the processor 402 can retrieve and run the computer programs from the memory 401 to implement the methods described in the embodiments of this application.

[0192] For example, the processor 402 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0193] In some embodiments of this application, the processor 402 may include, but is not limited to:

[0194] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0195] In some embodiments of this application, the memory 401 includes, but is not limited to:

[0196] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0197] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 401 and executed by the processor 402 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0198] As shown in Figure 4, the electronic device may also include:

[0199] Transceiver 403, which can be connected to processor 402 or memory 401.

[0200] The processor 402 can control the transceiver 403 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 403 may include a transmitter and a receiver. The transceiver 403 may further include antennas, and the number of antennas may be one or more.

[0201] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0202] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0203] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0204] According to one or more embodiments of this application, a data processing method is provided, comprising:

[0205] Obtain the sample speech information and the corresponding sample text information;

[0206] The sample speech information is processed by the acoustic model in the initial recognition model to obtain acoustic feature information;

[0207] The acoustic feature information is processed by the translation model in the initial recognition model to obtain the first predicted translation result corresponding to the sample speech information;

[0208] The sample text information is processed by the translation model to obtain a second predicted translation result corresponding to the sample text information;

[0209] The initial recognition model is trained based on the first predicted translation result and the second predicted translation result to obtain a target recognition model. The target recognition model is used to recognize the text translation result of the input speech information to be recognized.

[0210] Wherein, the acoustic model and / or the translation model are pre-trained models, and the acoustic model and the translation model satisfy at least one of the following conditions:

[0211] In the acoustic model, the first K vector and the first V vector in the first self-attention mechanism unit of at least one of the multiple first data processing layers are respectively concatenated with a first prefix vector and a second prefix vector.

[0212] In the translation model, the second K vector and the second V vector in the second self-attention mechanism unit of at least one of the multiple third data processing layers are respectively concatenated with a third prefix vector and a fourth prefix vector.

[0213] According to one or more embodiments of this application, the acoustic model and the translation model further satisfy at least one of the following conditions:

[0214] A first processing unit is connected between the input and output of the first feedforward neural network in at least one second data processing layer. The first processing unit includes two fully connected layers and a nonlinear activation function.

[0215] A second processing unit is connected between the input and output of the second feedforward neural network in at least one fourth data processing layer; the second processing unit includes two fully connected layers and a nonlinear activation function.

[0216] According to one or more embodiments of this application, the sample speech information is processed by an acoustic model in an initial recognition model to obtain acoustic feature information, including:

[0217] For each of the multiple first data processing layers in the acoustic model, the first input information of the first data processing layer is processed by the first data processing layer to obtain the first output information of the first data processing layer.

[0218] In this process, the output of any first data processing layer is the input of the next adjacent first data processing layer, the input of the first first data processing layer is the sample speech information, and the first output information of the last first data processing layer is acoustic feature information.

[0219] According to one or more embodiments of this application, the first data processing layer includes a first self-attention mechanism unit, a first normalization unit, a first feedforward neural network, and a second normalization unit; for the second data processing layer, the second input information of the second data processing layer is processed by the second data processing layer to obtain the second output information of the second data processing layer, including:

[0220] The second input information is processed by the first self-attention mechanism unit in the second data processing layer to obtain the first processing result;

[0221] The first processing result and the second input information are processed by the first normalization unit in the second data processing layer to obtain the first normalization result;

[0222] The first normalization result is processed by the first feedforward neural network in the second data processing layer to obtain the second processing result;

[0223] The first processing unit processes the first normalization result to obtain the third processing result;

[0224] The second normalization unit processes the first normalization result, the second processing result, and the third processing result to obtain the second output information of the second data processing layer.

[0225] According to one or more embodiments of this application, the acoustic feature information is processed by a translation model in an initial recognition model to obtain a first predicted translation result corresponding to the sample speech information, including:

[0226] For each of the plurality of third data processing layers, the third input information of the third data processing layer is processed by the third data processing layer to obtain the third output information of the third data processing layer.

[0227] In this process, the output of any third data processing layer is the input of the next adjacent third data processing layer, the input of the first third data processing layer is the acoustic feature information, and the third output information of the last third data processing layer is the first predicted translation result.

[0228] According to one or more embodiments of this application, the third data processing layer includes a second self-attention mechanism unit, a third normalization unit, a second feedforward neural network, and a fourth normalization unit; for the fourth data processing layer, the fourth input information of the fourth data processing layer is processed by the fourth data processing layer to obtain the fourth output information of the fourth data processing layer, including:

[0229] The fourth input information is processed by the second self-attention mechanism unit in the fourth data processing layer to obtain the fourth processing result;

[0230] The fourth processing result and the fourth input information are processed by the third normalization unit in the fourth data processing layer to obtain the second normalization result;

[0231] The second normalization result is processed by the second feedforward neural network in the fourth data processing layer to obtain the fifth processing result;

[0232] The second processing unit processes the second normalization result to obtain the sixth processing result;

[0233] The fourth normalization unit processes the second normalization result, the fifth processing result, and the sixth processing result to obtain the fourth output information of the fourth data processing layer.

[0234] According to one or more embodiments of this application, before processing the acoustic feature information through a translation model in an initial recognition model to obtain a first predicted translation result corresponding to the sample speech information, the method further includes:

[0235] The acoustic feature information is downsampled to obtain the processed acoustic feature information.

[0236] According to one or more embodiments of this application, the initial recognition model is trained based on the first predicted translation result and the second predicted translation result to obtain a target recognition model, including:

[0237] Obtain the first target label corresponding to the first predicted translation result, and the second target label corresponding to the second predicted translation result;

[0238] The first loss information corresponding to the first predicted translation result is determined by using a preset loss function and the first predicted translation result;

[0239] The second loss information corresponding to the second predicted translation result is determined by using the loss function and the second predicted translation result;

[0240] Determine the target loss information based on the first loss information and the second loss information;

[0241] If the target loss information is less than a preset threshold, the initial recognition model is used as the target recognition model; otherwise, the parameters of any one or more of the following are adjusted according to the target loss information: the first prefix vector, the second prefix vector, the first processing unit, the third prefix vector, the fourth prefix vector, and the second processing unit, and the process is returned to execute the acoustic model in the initial recognition model to process the sample speech information to obtain acoustic feature information.

[0242] According to one or more embodiments of this application, the method further includes:

[0243] Obtain the speech information to be recognized;

[0244] The target recognition model identifies the text translation result of the speech information to be recognized.

[0245] According to one or more embodiments of this application, a data processing apparatus is provided, comprising:

[0246] The acquisition unit is used to acquire sample speech information and sample text information corresponding to the sample speech information;

[0247] The processing unit is used to process the sample speech information through the acoustic model in the initial recognition model to obtain acoustic feature information;

[0248] The processing unit is further configured to process the acoustic feature information through the translation model in the initial recognition model to obtain a first predicted translation result corresponding to the sample speech information;

[0249] The processing unit is further configured to process the sample text information through the translation model to obtain a second predicted translation result corresponding to the sample text information;

[0250] The training unit is used to train the initial recognition model based on the first predicted translation result and the second predicted translation result to obtain a target recognition model. The target recognition model is used to recognize the text translation result of the input speech information to be recognized.

[0251] Wherein, the acoustic model and / or the translation model are pre-trained models, and the acoustic model and the translation model satisfy at least one of the following conditions:

[0252] In the acoustic model, the first K vector and the first V vector in the first self-attention mechanism unit of at least one of the multiple first data processing layers are respectively concatenated with a first prefix vector and a second prefix vector.

[0253] In the translation model, the second K vector and the second V vector in the second self-attention mechanism unit of at least one of the multiple third data processing layers are respectively concatenated with a third prefix vector and a fourth prefix vector.

[0254] According to one or more embodiments of this application, when the aforementioned apparatus is used to process the sample speech information through an acoustic model in an initial recognition model to obtain acoustic feature information, it is specifically used for:

[0255] For each of the multiple first data processing layers in the acoustic model, the first input information of the first data processing layer is processed by the first data processing layer to obtain the first output information of the first data processing layer.

[0256] In this process, the output of any first data processing layer is the input of the next adjacent first data processing layer, the input of the first first data processing layer is the sample speech information, and the first output information of the last first data processing layer is acoustic feature information.

[0257] According to one or more embodiments of this application, the first data processing layer includes a first self-attention mechanism unit, a first normalization unit, a first feedforward neural network, and a second normalization unit; regarding the second data processing layer, when the aforementioned device processes the second input information of the second data processing layer through the second data processing layer to obtain the second output information of the second data processing layer, it is specifically used for:

[0258] The second input information is processed by the first self-attention mechanism unit in the second data processing layer to obtain the first processing result;

[0259] The first processing result and the second input information are processed by the first normalization unit in the second data processing layer to obtain the first normalization result;

[0260] The first normalization result is processed by the first feedforward neural network in the second data processing layer to obtain the second processing result;

[0261] The first processing unit processes the first normalization result to obtain the third processing result;

[0262] The second normalization unit processes the first normalization result, the second processing result, and the third processing result to obtain the second output information of the second data processing layer.

[0263] According to one or more embodiments of this application, the aforementioned transposition, when used to process the acoustic feature information through a translation model in an initial recognition model to obtain a first predicted translation result corresponding to the sample speech information, is specifically used for:

[0264] For each of the plurality of third data processing layers, the third input information of the third data processing layer is processed by the third data processing layer to obtain the third output information of the third data processing layer.

[0265] In this process, the output of any third data processing layer is the input of the next adjacent third data processing layer, the input of the first third data processing layer is the acoustic feature information, and the third output information of the last third data processing layer is the first predicted translation result.

[0266] According to one or more embodiments of this application, the third data processing layer includes a second self-attention mechanism unit, a third normalization unit, a second feedforward neural network, and a fourth normalization unit; the aforementioned apparatus, when used to process the fourth input information of the fourth data processing layer to obtain the fourth output information of the fourth data processing layer, is specifically used for:

[0267] The fourth input information is processed by the second self-attention mechanism unit in the fourth data processing layer to obtain the fourth processing result;

[0268] The fourth processing result and the fourth input information are processed by the third normalization unit in the fourth data processing layer to obtain the second normalization result;

[0269] The second normalization result is processed by the second feedforward neural network in the fourth data processing layer to obtain the fifth processing result;

[0270] The second processing unit processes the second normalization result to obtain the sixth processing result;

[0271] The fourth normalization unit processes the second normalization result, the fifth processing result, and the sixth processing result to obtain the fourth output information of the fourth data processing layer.

[0272] According to one or more embodiments of this application, before the aforementioned apparatus processes the acoustic feature information using a translation model in an initial recognition model to obtain a first predicted translation result corresponding to the sample speech information, it is further configured to:

[0273] The acoustic feature information is downsampled to obtain the processed acoustic feature information.

[0274] According to one or more embodiments of this application, when the aforementioned apparatus is used to train the initial recognition model based on the first predicted translation result and the second predicted translation result to obtain a target recognition model, it is specifically used for:

[0275] Obtain the first target label corresponding to the first predicted translation result, and the second target label corresponding to the second predicted translation result;

[0276] The first loss information corresponding to the first predicted translation result is determined by using a preset loss function and the first predicted translation result;

[0277] The second loss information corresponding to the second predicted translation result is determined by using the loss function and the second predicted translation result;

[0278] Determine the target loss information based on the first loss information and the second loss information;

[0279] If the target loss information is less than a preset threshold, the initial recognition model is used as the target recognition model; otherwise, the parameters of any one or more of the following are adjusted according to the target loss information: the first prefix vector, the second prefix vector, the first processing unit, the third prefix vector, the fourth prefix vector, and the second processing unit, and the process is returned to execute the acoustic model in the initial recognition model to process the sample speech information to obtain acoustic feature information.

[0280] According to one or more embodiments of this application, the aforementioned apparatus is further used for:

[0281] Obtain the speech information to be recognized;

[0282] The target recognition model identifies the text translation result of the speech information to be recognized.

[0283] According to one or more embodiments of this application, an electronic device is provided, comprising:

[0284] Processor; and

[0285] Memory for storing the executable instructions of the processor;

[0286] The processor is configured to execute the aforementioned methods by executing the executable instructions.

[0287] According to one or more embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the aforementioned methods.

[0288] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0289] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0290] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0291] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, include: Obtain the sample speech information and the corresponding sample text information; The sample speech information is processed by the acoustic model in the initial recognition model to obtain acoustic feature information; The acoustic feature information is processed by the translation model in the initial recognition model to obtain a first predicted translation result corresponding to the sample speech information; the sample text information is processed by the translation model to obtain a second predicted translation result corresponding to the sample text information. The initial recognition model is trained based on the first predicted translation result and the second predicted translation result to obtain a target recognition model. The target recognition model is used to identify the text translation result of the input speech information to be recognized. The acoustic model and / or the translation model are pre-trained models, and the acoustic model and the translation model satisfy at least one of the following conditions: the first K vector and the first V vector in the first self-attention mechanism unit of at least one of the multiple first data processing layers in the acoustic model are respectively concatenated with a first prefix vector and a second prefix vector; the second K vector and the second V vector in the second self-attention mechanism unit of at least one of the multiple third data processing layers in the translation model are respectively concatenated with a third prefix vector and a fourth prefix vector.

2. The method according to claim 1, characterized in that, The acoustic model and the translation model also satisfy at least one of the following conditions: a first processing unit is connected between the input and output of the first feedforward neural network in the at least one second data processing layer, the first processing unit comprising two fully connected layers and a nonlinear activation function; a second processing unit is connected between the input and output of the second feedforward neural network in the at least one fourth data processing layer; the second processing unit comprising two fully connected layers and a nonlinear activation function.

3. The method according to claim 1, characterized in that, The sample speech information is processed by the acoustic model in the initial recognition model to obtain acoustic feature information, including: for each of the multiple first data processing layers in the acoustic model, the first input information of the first data processing layer is processed by the first data processing layer to obtain the first output information of the first data processing layer; wherein, the output of any first data processing layer is the input of the next adjacent first data processing layer, the input of the first first data processing layer is the sample speech information, and the first output information of the last first data processing layer is the acoustic feature information.

4. The method according to claim 3, characterized in that, The first data processing layer includes a first self-attention mechanism unit, a first normalization unit, a first feedforward neural network, and a second normalization unit. For the second data processing layer, the second input information of the second data processing layer is processed to obtain the second output information of the second data processing layer, including: processing the second input information through the first self-attention mechanism unit in the second data processing layer to obtain a first processing result; processing the first processing result and the second input information through the first normalization unit in the second data processing layer to obtain a first normalized result; processing the first normalized result through the first feedforward neural network in the second data processing layer to obtain a second processing result; processing the first normalized result through the first processing unit to obtain a third processing result; and processing the first normalized result, the second processing result, and the third processing result through the second normalization unit to obtain the second output information of the second data processing layer.

5. The method according to claim 1, characterized in that, The acoustic feature information is processed by the translation model in the initial recognition model to obtain the first predicted translation result corresponding to the sample speech information. This includes: processing the third input information of each of the plurality of third data processing layers through the third data processing layer to obtain the third output information of the third data processing layer; wherein the output of any third data processing layer is the input of the next adjacent third data processing layer, the input of the first third data processing layer is the acoustic feature information, and the third output information of the last third data processing layer is the first predicted translation result.

6. The method according to claim 5, characterized in that, The third data processing layer includes a second self-attention mechanism unit, a third normalization unit, a second feedforward neural network, and a fourth normalization unit. For the fourth data processing layer, the fourth input information is processed by the fourth data processing layer to obtain the fourth output information, including: processing the fourth input information through the second self-attention mechanism unit to obtain a fourth processing result; processing the fourth processing result and the fourth input information through the third normalization unit to obtain a second normalization result; processing the second normalization result through the second feedforward neural network to obtain a fifth processing result; processing the second normalization result through the second processing unit to obtain a sixth processing result; and processing the second normalization result, the fifth processing result, and the sixth processing result through the fourth normalization unit to obtain the fourth output information of the fourth data processing layer.

7. The method according to claim 1, characterized in that, Before processing the acoustic feature information through the translation model in the initial recognition model to obtain the first predicted translation result corresponding to the sample speech information, the method further includes: performing downsampling processing on the acoustic feature information to obtain processed acoustic feature information.

8. The method according to claim 2, characterized in that, Training the initial recognition model based on the first predicted translation result and the second predicted translation result to obtain a target recognition model includes: obtaining a first target label corresponding to the first predicted translation result and a second target label corresponding to the second predicted translation result; determining a first loss information corresponding to the first predicted translation result using a preset loss function and the first predicted translation result; determining a second loss information corresponding to the second predicted translation result using the loss function and the second predicted translation result; determining target loss information based on the first loss information and the second loss information; if the target loss information is less than a preset threshold, then the initial recognition model is used as the target recognition model; otherwise, adjusting one or more of the following parameters according to the target loss information: the first prefix vector, the second prefix vector, the first processing unit, the third prefix vector, the fourth prefix vector, and the second processing unit, and returning to execute processing of the sample speech information through the acoustic model in the initial recognition model to obtain acoustic feature information.

9. The method according to claim 1, characterized in that, The method further includes: acquiring speech information to be recognized; and identifying the text translation result of the speech information to be recognized through the target recognition model.

10. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire sample speech information and sample text information corresponding to the sample speech information; the processing unit is used to process the sample speech information through the acoustic model in the initial recognition model to obtain acoustic feature information. The processing unit is further configured to process the acoustic feature information through the translation model in the initial recognition model to obtain a first predicted translation result corresponding to the sample speech information; the processing unit is further configured to process the sample text information through the translation model to obtain a second predicted translation result corresponding to the sample text information. A training unit is used to train the initial recognition model based on the first predicted translation result and the second predicted translation result to obtain a target recognition model. The target recognition model is used to recognize the text translation result of the input speech information to be recognized. The acoustic model and / or the translation model are pre-trained models, and the acoustic model and the translation model satisfy at least one of the following conditions: the first K vector and the first V vector in the first self-attention mechanism unit of at least one of the multiple first data processing layers in the acoustic model are respectively concatenated with a first prefix vector and a second prefix vector; the second K vector and the second V vector in the second self-attention mechanism unit of at least one of the multiple third data processing layers in the translation model are respectively concatenated with a third prefix vector and a fourth prefix vector.

11. An electronic device, characterized in that, include: processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of any one of claims 1-9 by executing the executable instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-9.

Images