Training method for speech translation device based on feature transfer loss

Abstract

The present disclosure provides a method for training a speech translation device based on feature transfer loss, comprising: performing alignment training between a speech sequence and a text sequence on a speech recognition module of the speech translation device; performing text translation training on a semantic encoder and decoder of the speech translation device; performing speech translation training on the speech recognition module after alignment training, the semantic encoder after text translation training, and the decoder after text translation training; constructing a feature transfer loss function based on a representation containing semantic information of a source language text sequence generated by the semantic encoder during text translation training and a representation containing semantic information of a source language speech feature sequence generated by the semantic encoder during speech translation training; and minimizing the feature transfer loss based on the feature transfer loss function to obtain a speech translation device. The present disclosure also provides a speech translation device, an electronic device, and a readable storage medium.

Description

Technical Field

[0001] The present disclosure relates to the technical field of speech translation, and more particularly to a training method for a speech translation device based on feature transfer loss, a speech translation device based on feature transfer loss, an electronic device, and a storage medium. Background Art

[0002] In the field of machine translation, end-to-end speech translation is a cross-language and cross-modal translation task that not only requires processing long speech input sequences and extracting acoustic information, but also requires learning the relationship between the source language speech and the target language text. This makes it difficult to model end-to-end speech translation devices, and the performance of the modeled models is poor.

[0003] The modality difference between speech modality and text modality exacerbates the difficulty of modeling.

[0004] The problem of inconsistent representation lengths between speech and text further exacerbates the difficulty of modeling. The applicant's patent application series, "Speech Translation Method and Apparatus Based on Invalid Information Filtering," proposes filtering invalid information to address this issue.

[0005] However, during the modeling process, it is also very difficult to learn and acquire the alignment relationship between the source language speech sequence and the target language text sequence. Summary of the Invention

[0006] In order to solve at least one of the above technical problems, the present disclosure provides a training method for a speech translation device based on feature transfer loss, a speech translation device based on feature transfer loss, an electronic device, and a storage medium.

[0007] According to one aspect of the present disclosure, a method for training a speech translation device based on feature transfer loss is provided, comprising:

[0008] S102, performing alignment training between a speech sequence and a text sequence on a speech recognition module of a speech translation device based on a data set including a source language speech feature sequence, a source language text sequence, and a target language text sequence, to obtain a speech recognition module after alignment training;

[0009] S104, performing text translation training on the semantic encoder and decoder of the speech translation device based on the data set including the source language speech feature sequence-the source language text sequence-the target language text sequence, to obtain a semantic encoder and decoder after text translation training;

[0010] S106, performing speech translation training on the speech recognition module after the alignment training, the semantic encoder after the text translation training, and the decoder after the text translation training based on the data set including the source language speech feature sequence-the source language text sequence-the target language text sequence, to obtain the speech recognition module, the semantic decoder, and the decoder after the speech translation training;

[0011] S108, a representation containing semantic information of the source language text sequence generated by the semantic encoder in the text translation training (h x ) and the representation of the source language speech feature sequence containing semantic information generated by the semantic encoder in the speech translation training (h s ), construct feature transfer loss function (L AD );

[0012] S110 : Minimize feature transfer loss based on the feature transfer loss function to obtain the speech translation device.

[0013] According to the method for training a speech translation device based on feature transfer loss according to at least one embodiment of the present disclosure, the source language text sequence and the source language speech feature sequence have the same length.

[0014] According to at least one embodiment of the present disclosure, a training method for a speech translation device based on feature transfer loss is provided, wherein the feature transfer loss function is obtained based on the L2 norm of a word-level representation containing semantic information of a source language text sequence and a word-level representation containing semantic information of a source language speech feature sequence.

[0015] According to at least one embodiment of the present disclosure, a training method for a speech translation device based on feature transfer loss is provided, wherein the feature transfer loss function is obtained based on the L2 norm of a sequence-level representation containing semantic information of a source language text sequence and a sequence-level representation containing semantic information of a source language speech feature sequence.

[0016] According to at least one embodiment of the present disclosure, a method for training a speech translation device based on feature transfer loss includes performing alignment training between a speech feature sequence and a text sequence on a speech recognition module of the speech translation device based on a data set including a source language speech feature sequence, a source language text sequence, and a target language text sequence. The method includes:

[0017] The speech recognition module generates a plurality of source language text prediction sequences based on the source language speech feature sequences in the data set, and performs the alignment training based on the plurality of source language text prediction sequences and the corresponding source language text sequences in the data set.

[0018] According to at least one embodiment of the present disclosure, a training method for a speech translation device based on feature transfer loss is disclosed, wherein the speech recognition module includes a multi-head attention network, a feedforward neural network, and a softmax function module, wherein the source language speech feature sequence serves as the input of the multi-head attention network, the output of the multi-head attention network serves as the input of the feedforward neural network, the output of the feedforward neural network serves as the input of the softmax function module, and the soft function module outputs the multiple source language text prediction sequences.

[0019] According to the training method of the speech translation device based on feature transfer loss in at least one embodiment of the present disclosure, the speech recognition module further includes a CTC loss function module, which constructs a CTC loss function (L) based on the sum of the prediction probabilities of each source language text prediction sequence. CTC ), minimize the CTC loss to end the alignment training.

[0020] According to at least one embodiment of the present disclosure, a method for training a speech translation device based on feature transfer loss includes performing text translation training on a semantic encoder and a decoder of the speech translation device based on a data set including a source language speech feature sequence, a source language text sequence, and a target language text sequence, to obtain a semantic encoder and a decoder after text translation training, including:

[0021] Construct the objective function of the text translation task (L MT ), minimize the objective function of the text translation task (L MT ) to end the text translation training of the semantic encoder and decoder; among them, the objective function of the text translation task (L MT ) is expressed as: Among them, x represents the source language text sequence, y represents the target language text sequence, and D ST Represents a dataset.

[0022] According to at least one embodiment of the present disclosure, the training method for a speech translation device based on feature transfer loss, in S106, performs speech translation training on the speech recognition module after alignment training, the semantic encoder after text translation training, and the decoder after text translation training based on the data set including the source language speech feature sequence, the source language text sequence, and the target language text sequence, including:

[0023] Construct the objective function of the speech translation task (L ST ), minimize the objective function of the speech translation task (L ST) to end the speech translation training of the speech recognition module, semantic encoder and decoder; Among them, the objective function of the speech translation task (L ST ) is expressed as: Where s represents the source language speech feature sequence, y represents the target language text sequence, and D ST Represents a dataset.

[0024] According to at least one embodiment of the present disclosure, a training method for a speech translation device based on feature transfer loss is provided. The semantic encoder includes a multi-head attention network, a feedforward neural network, and a transfer loss calculation module. The source language text sequence is used as the input of the multi-head attention network, and the output of the multi-head attention network is used as the input of the feedforward neural network. The feedforward neural network outputs a representation (h) of the source language text sequence containing semantic information. x ), the representation of the source language text sequence containing semantic information (h x ) is input into the migration loss calculation module to calculate the characteristic migration loss.

[0025] According to at least one embodiment of the present disclosure, a training method for a speech translation device based on feature transfer loss is provided. The decoder includes a multi-head self-attention network, a multi-head cross-attention network, a feedforward neural network, a softmax function module, and a cross-entropy loss calculation module. The target language text sequence in the data set is positionally encoded and used as the input of the multi-head self-attention network. The output of the multi-head self-attention network is used as the input of the multi-head cross-attention network. The semantic encoder generates a representation (h x ) is used as the input of the multi-head cross-attention network of the decoder, and the output of the multi-head cross-attention network is used as the input of the feedforward neural network of the decoder. The decoder performs decoding training on the output of the feedforward neural network based on the softmax function module and the cross-entropy loss calculation module.

[0026] According to another aspect of the present disclosure, a speech translation device based on feature transfer loss is provided, comprising: a speech recognition module, the speech recognition module being configured to recognize source language speech to output source language text; a semantic encoder, the semantic encoder encoding the source language text into a representation containing semantic information; and a decoder, the decoder decoding the representation containing semantic information to obtain target language text;

[0027] The speech recognition module, the semantic encoder and the decoder are speech recognition modules, semantic encoders and decoders trained by any one of the training methods described above.

[0028] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0029] A memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor performs any one of the training methods described above.

[0030] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and the execution instructions are used to implement any of the above-mentioned training methods when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0032] Figure 1 It is a flowchart of a training method for a speech translation device based on feature transfer loss according to an embodiment of the present disclosure.

[0033] Figure 2 It is a flowchart of a training method for a speech translation device based on feature transfer loss according to another embodiment of the present disclosure.

[0034] Figure 3 It is a schematic block diagram of the structure of a speech translation device based on feature transfer loss in a hardware implementation manner of a processing system according to one embodiment of the present disclosure.

[0035] Description of Reference Numerals

[0036] 1000 Voice Translation Device

[0037] 1002 Speech Recognition Module

[0038] 1004 Semantic Encoder

[0039] 1006 Decoder

[0040] 1100 bus

[0041] 1200 processor

[0042] 1300 Memory

[0043] 1400 Other circuits. DETAILED DESCRIPTION

[0044] The present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0045] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] Unless otherwise specified, the exemplary embodiments / examples shown are to be understood as providing exemplary features of various details of some ways in which the technical concepts of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / examples may be further combined, separated, interchanged, and / or rearranged without departing from the technical concepts of the present disclosure.

[0047] The use of cross hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise indicated, the presence or absence of cross hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the order described. In addition, the same figure numbers represent the same components.

[0048] When a component is referred to as being “on,” “over,” “connected to,” or “coupled to” another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being “directly on,” “directly connected to,” or “directly coupled to” another component, there are no intervening components present. For this purpose, the term “connected” may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0049] The terms used herein are for the purpose of describing specific embodiments and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, the features, integral bodies, steps, operations, parts, assemblies and / or their groups stated are indicated, but the presence or addition of one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups is not excluded. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, and as such, they are used to explain the inherent deviations of the measured values, calculated values ​​and / or values ​​provided that will be recognized by those of ordinary skill in the art.

[0050] Combined with the following Figures 1 to 3 The present invention provides a detailed description of a training method for a speech translation device based on feature transfer loss, a speech translation device based on feature transfer loss, an electronic device, and a storage medium.

[0051] refer to Figure 1 According to an embodiment of the present disclosure, a training method S100 for a speech translation device based on feature transfer loss includes:

[0052] S102, performing alignment training between a speech sequence and a text sequence on a speech recognition module of the speech translation device to obtain a speech recognition module after alignment training;

[0053] S104, performing text translation training on the semantic encoder and decoder of the speech translation device to obtain semantic encoder and decoder after text translation training;

[0054] S106, performing speech translation training on the speech recognition module after alignment training, the semantic encoder after text translation training, and the decoder after text translation training to obtain a speech recognition module, a semantic decoder, and a decoder after speech translation training;

[0055] S108, a representation containing semantic information of the source language text sequence generated by the semantic encoder in the text translation training (h x ) and the semantic information of the source language speech feature sequence generated by the semantic encoder in speech translation training (h s ), construct the feature transfer loss function (L AD );

[0056] S110 : Based on the feature transfer loss function, minimize the feature transfer loss to obtain a speech translation device.

[0057] According to a preferred embodiment of the present disclosure, step S102 is: based on a data set including a source language speech feature sequence-a source language text sequence-a target language text sequence, a speech recognition module of the speech translation device is subjected to alignment training between the speech sequence and the text sequence to obtain a speech recognition module after alignment training.

[0058] According to a preferred embodiment of the present disclosure, step S104 is: performing text translation training on the semantic encoder and decoder of the speech translation device based on a data set including a source language speech feature sequence-a source language text sequence-a target language text sequence to obtain a semantic encoder and decoder after text translation training.

[0059] According to a preferred embodiment of the present disclosure, step S106 is: based on a data set including a source language speech feature sequence-source language text sequence-target language text sequence, the speech recognition module after alignment training, the semantic encoder after text translation training, and the decoder after text translation training are subjected to speech translation training to obtain a speech recognition module, a semantic decoder, and a decoder after speech translation training.

[0060] The triplet dataset of source language speech feature sequence-source language text sequence-target language text sequence described above in this disclosure can be obtained, for example, by crawling the audio and bilingual subtitles of foreign language movies and foreign language videos with bilingual subtitles.

[0061] Regarding the training method S100 for a speech translation device based on feature transfer loss in each of the above embodiments, preferably, the lengths of the source language text sequence and the source language speech feature sequence are the same.

[0062] That is, when constructing the above triplet dataset, the present disclosure makes the lengths of the source language text sequence and the source language speech feature sequence the same. If the lengths of the source language text sequence and the source language speech feature sequence are different, padding processing is performed.

[0063] For the training method S100 of the speech translation device based on feature transfer loss in each of the above-mentioned embodiments, preferably, the feature transfer loss function is obtained based on the L2 norm of the word-level representation containing semantic information of the source language text sequence and the word-level representation containing semantic information of the source language speech feature sequence.

[0064] For the training method S100 of the speech translation device based on feature transfer loss in each of the above-mentioned embodiments, preferably, the feature transfer loss function is obtained based on the L2 norm of the sequence-level representation containing semantic information of the source language text sequence and the sequence-level representation containing semantic information of the source language speech feature sequence.

[0065] For the training method S100 of the speech translation device based on feature transfer loss in each of the above embodiments, refer to Figure 2 Preferably, in S102, the speech recognition module of the speech translation device is trained to align the speech feature sequence with the text sequence based on a data set including a source language speech feature sequence, a source language text sequence, and a target language text sequence, including:

[0066] The speech recognition module generates multiple source language text prediction sequences based on the source language speech feature sequence in the data set, and performs alignment training based on the multiple source language text prediction sequences and the corresponding source language text sequences in the data set.

[0067] Figure 2 FIG. 4 is a schematic diagram showing a complete flow chart of a training method S100 for a speech translation device based on feature transfer loss according to this embodiment.

[0068] For the training method S100 of the speech translation device based on feature transfer loss in each of the above-mentioned embodiments, preferably, the speech recognition module includes a multi-head attention network, a feedforward neural network and a softmax function module, the source language speech feature sequence serves as the input of the multi-head attention network, the output of the multi-head attention network serves as the input of the feedforward neural network, the output of the feedforward neural network serves as the input of the softmax function module, and the soft function module outputs multiple source language text prediction sequences.

[0069] Among them, the speech recognition module can be an encoder structure based on the transformer architecture.

[0070] According to the training method S100 of the speech translation device based on feature transfer loss in the preferred embodiment of the present disclosure, the speech recognition module further includes a CTC loss function module, which constructs a CTC loss function (L) based on the sum of the prediction probabilities of each source language text prediction sequence. CTC ), minimize the CTC loss to end the alignment training.

[0071] In step S104 of the feature transfer loss-based speech translation device training method S100 according to a preferred embodiment of the present disclosure, text translation training is performed on the semantic encoder and decoder of the speech translation device based on a data set including a source language speech feature sequence, a source language text sequence, and a target language text sequence to obtain the semantic encoder and decoder after text translation training, including:

[0072] Construct the objective function of the text translation task (L MT ), minimize the objective function of the text translation task (L MT) to end the text translation training of the semantic encoder and decoder; among them, the objective function of the text translation task (L MT ) is expressed as: Among them, x represents the source language text sequence, y represents the target language text sequence, and D ST Represents a dataset.

[0073] In step S106 of the training method S100 for a speech translation device based on feature transfer loss according to a preferred embodiment of the present disclosure, speech translation training is performed on a speech recognition module after alignment training, a semantic encoder after text translation training, and a decoder after text translation training based on a data set including a source language speech feature sequence, a source language text sequence, and a target language text sequence, including:

[0074] Construct the objective function of the speech translation task (L ST ), minimize the objective function of the speech translation task (L ST ) to end the speech translation training of the speech recognition module, semantic encoder and decoder; Among them, the objective function of the speech translation task (L ST ) is expressed as: Where s represents the source language speech feature sequence, y represents the target language text sequence, and D ST Represents a dataset.

[0075] For the training method S100 of the speech translation device based on feature transfer loss in each of the above embodiments, preferably, the semantic encoder includes a multi-head attention network, a feedforward neural network and a transfer loss calculation module, the source language text sequence is used as the input of the multi-head attention network, the output of the multi-head attention network is used as the input of the feedforward neural network, and the feedforward neural network outputs a representation (h x ), the representation of the source language text sequence containing semantic information (h x ) is input to the migration loss calculation module to calculate the feature migration loss.

[0076] For the training method S100 of the speech translation device based on feature transfer loss in each of the above embodiments, preferably, the decoder includes a multi-head self-attention network, a multi-head cross-attention network, a feedforward neural network, a softmax function module, and a cross-entropy loss calculation module. The target language text sequence in the data set is positionally encoded and used as the input of the multi-head self-attention network. The output of the multi-head self-attention network is used as the input of the multi-head cross-attention network. The semantic encoder generates a representation (h x) is used as the input of the multi-head cross-attention network of the decoder, and the output of the multi-head cross-attention network is used as the input of the feedforward neural network of the decoder. The decoder decodes the output of the feedforward neural network based on the softmax function module and the cross entropy loss calculation module.

[0077] According to an embodiment of the present disclosure, a speech translation device 1000 based on feature transfer loss includes: a speech recognition module 1002, which is used to recognize source language speech to output source language text; a semantic encoder 1004, which encodes the source language text into a representation containing semantic information; and a decoder 1006, which decodes the representation containing semantic information to obtain target language text.

[0078] The speech recognition module, semantic encoder, and decoder are the speech recognition module, semantic encoder, and decoder trained by the training method S100 of any one of the above-mentioned embodiments.

[0079] The speech translation device 1000 based on feature transfer loss of the present disclosure can be implemented in the form of a computer software architecture.

[0080] Figure 3 FIG2 shows a schematic structural block diagram of a speech translation apparatus 1000 based on feature transfer loss in a hardware implementation manner of a processing system.

[0081] The speech translation device 1000 based on feature transfer loss may include corresponding modules for executing each or several steps in the above flow chart. Therefore, each step or several steps in the above flow chart may be executed by the corresponding module, and the device may include one or more modules in these modules. The module may be one or more hardware modules specifically configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0082] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0083] Bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, this figure shows only one connecting line, but this does not imply that there is only one bus or only one type of bus.

[0084] Any process or method description in the flowchart or otherwise described herein can be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which the functions may be performed in a manner not shown or discussed, including performing the functions substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which the embodiments of the present disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of the present disclosure can be implemented as a software program that is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0085] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0086] For the purposes of this specification, a "readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples (a non-exhaustive list) of readable storage media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium can even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a memory.

[0087] It should be understood that various parts of the present disclosure can be implemented using hardware, software, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the hardware: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0088] Those skilled in the art will understand that all or part of the steps of the above-mentioned implementation method can be accomplished by instructing related hardware through a program, and the program can be stored in a readable storage medium. When the program is executed, it includes one or a combination of the steps of the method implementation method.

[0089] Furthermore, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as independent products, they may also be stored in a readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0090] The present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module performs the above-mentioned training method.

[0091] The present disclosure also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above-mentioned training method.

[0092] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are mutually inconsistent.

[0093] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0094] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. A training method for a speech translation device based on feature transfer loss, characterized in that: include: S102, performing alignment training between a speech sequence and a text sequence on a speech recognition module of a speech translation device based on a data set including a source language speech feature sequence, a source language text sequence, and a target language text sequence, to obtain a speech recognition module after alignment training; S104, performing text translation training on the semantic encoder and decoder of the speech translation device based on the data set including the source language speech feature sequence-the source language text sequence-the target language text sequence, to obtain a semantic encoder and decoder after text translation training; S106, performing speech translation training on the speech recognition module after the alignment training, the semantic encoder after the text translation training, and the decoder after the text translation training based on the data set including the source language speech feature sequence-the source language text sequence-the target language text sequence, to obtain the speech recognition module, the semantic decoder, and the decoder after the speech translation training; S108, constructing a feature transfer loss function based on the representation of the source language text sequence containing semantic information generated by the semantic encoder in the text translation training and the representation of the source language speech feature sequence containing semantic information generated by the semantic encoder in the speech translation training; and S110 : Minimize feature transfer loss based on the feature transfer loss function to obtain the speech translation device. In S102, alignment training between the speech feature sequence and the text sequence is performed on the speech recognition module of the speech translation device based on a data set including a source language speech feature sequence, a source language text sequence, and a target language text sequence, including: The speech recognition module generates a plurality of source language text prediction sequences based on the source language speech feature sequences in the data set, and performs the alignment training based on the plurality of source language text prediction sequences and the corresponding source language text sequences in the data set; The speech recognition module includes a multi-head attention network, a feedforward neural network and a softmax function module, wherein the source language speech feature sequence serves as the input of the multi-head attention network, the output of the multi-head attention network serves as the input of the feedforward neural network, the output of the feedforward neural network serves as the input of the softmax function module, and the softmax function module outputs the multiple source language text prediction sequences; The speech recognition module further includes a CTC loss function module, which constructs a CTC loss function based on the sum of the prediction probabilities of each source language text prediction sequence, and minimizes the CTC loss to end the alignment training; S104, performing text translation training on the semantic encoder and decoder of the speech translation device based on a data set including a source language speech feature sequence, a source language text sequence, and a target language text sequence to obtain a semantic encoder and decoder after text translation training, including: Construct an objective function for the text translation task and minimize the maximum likelihood loss of the objective function for the text translation task to complete the text translation training for the semantic encoder and decoder; wherein the objective function of the text translation task is expressed as for: Among them, x represents the source language text sequence, y represents the target language text sequence, and D ST Represents a dataset. In S106, speech translation training is performed on the speech recognition module after alignment training, the semantic encoder after text translation training, and the decoder after text translation training based on the data set including the source language speech feature sequence, the source language text sequence, and the target language text sequence, including: Construct an objective function for the speech translation task and minimize the maximum likelihood loss of the objective function of the speech translation task to complete the speech translation training of the speech recognition module, the semantic encoder, and the decoder; wherein the objective function of the speech translation task is expressed as: Among them, s represents the source language speech feature sequence, y represents the target language text sequence, and DST represents the dataset.

2. The training method for a speech translation device based on feature transfer loss according to claim 1, wherein: The semantic encoder includes a multi-head attention network, a feedforward neural network and a transfer loss calculation module. The source language text sequence serves as the input of the multi-head attention network, and the output of the multi-head attention network serves as the input of the feedforward neural network. The feedforward neural network outputs a representation of the source language text sequence containing semantic information, and the representation of the source language text sequence containing semantic information is input into the transfer loss calculation module to calculate the feature transfer loss.

3. A speech translation device based on feature transfer loss, comprising: A speech recognition module, configured to recognize the source language speech and output the source language text; a semantic encoder that encodes the source language text into a representation containing semantic information; and a decoder that decodes the representation containing semantic information to obtain a target language text; It is characterized in that the speech recognition module, the semantic encoder and the decoder are the speech recognition module, the semantic encoder and the decoder trained by the training method according to any one of claims 1 to 2.

4. An electronic device, characterized in that: include: a memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor performs the training method according to any one of claims 1 to 2.

5. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the training method according to any one of claims 1 to 2 when executed by a processor.

Images