Speech translation method, speech translation device, equipment and storage medium

Abstract

This application discloses a speech translation method, which obtains acoustic information and a first hidden state of a source language speech; wherein the first hidden state is used to represent the fusion weight of the hidden representation of the source language speech; the acoustic information and the first hidden state are input into a trained first sequence converter to obtain a source language text and a second hidden state; wherein the second hidden state is used to represent the contextual relationship of the hidden representation of the acoustic information of the source language speech; the source language text and the second hidden state are input into a trained second sequence converter to obtain a target text corresponding to the source language speech; and the target text is speech-converted to obtain a target speech of the source language speech. This application also discloses a speech translation device, an electronic device, and a computer-readable storage medium.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the field of computers, and in particular to a speech translation method, a speech translation device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Speech translation is an important voice interaction method, aiming to directly convert source language speech into target language speech. In related art, the speech translation process uses a speech recognition model, a machine translation model, and a text-to-speech model to sequentially convert source language speech into source language text, source language text into target language text, and finally target language text into target language speech. In other words, during the actual translation process, the input of the lower-level model depends on the output of the upper-level model. Clearly, the different models in related art lack contextual connections, resulting in poor recognition performance across the entire system. Summary of the Invention

[0003] In view of this, the present application provides a speech translation method, a speech translation device, an electronic device and a computer-readable storage medium, which solve the problem of lack of contextual association between different models in the related art, resulting in poor recognition performance of the entire system.

[0004] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0005] A speech translation method, comprising:

[0006] Acquiring acoustic information and a first hidden state of a source language speech; wherein the first hidden state is used to represent a fusion weight of a hidden representation of the source language speech;

[0007] Inputting the acoustic information and the first hidden state into a trained first sequence converter to obtain a source language text and a second hidden state; wherein the second hidden state is used to represent the contextual relationship of the hidden representation of the acoustic information of the source language speech;

[0008] Inputting the source language text and the second hidden state into a trained second sequence converter to obtain a target text corresponding to the source language speech;

[0009] The target text is converted into speech to obtain a target speech of the source language speech.

[0010] A speech translation device, comprising:

[0011] an acquisition unit, configured to acquire acoustic information and a first hidden state of a source language speech; wherein the first hidden state is used to represent a fusion weight of a hidden representation of the source language speech;

[0012] The processing unit is further configured to input the acoustic information and the first hidden state into a trained first sequence converter to obtain a source language text and a second hidden state; wherein the second hidden state is used to represent the contextual relationship of the hidden representation of the acoustic information of the source language speech;

[0013] The processing unit is further configured to input the source language text and the second hidden state into a trained second sequence converter to obtain a target text corresponding to the source language speech;

[0014] The processing unit is further configured to perform speech conversion on the target text to obtain a target speech of the source language speech.

[0015] An electronic device, comprising:

[0016] a memory for storing executable instructions;

[0017] The processor is configured to implement the steps of the above-mentioned speech translation method when executing the executable instructions stored in the memory.

[0018] A computer-readable storage medium stores one or more applications, which can be executed by one or more processors to implement the steps of the speech translation method described above.

[0019] The speech translation method, speech translation device, electronic device and computer-readable storage medium provided in the present application obtain acoustic information and a first hidden state of a source language speech; wherein the first hidden state is used to represent the fusion weight of the hidden representation of the source language speech; the acoustic information and the first hidden state are input into a trained first sequence converter to obtain a source language text and a second hidden state; wherein the second hidden state is used to represent the contextual connection of the hidden representation of the acoustic information of the source language speech; the source language text and the second hidden state are input into a trained second sequence converter to obtain a target text corresponding to the source language speech; and speech conversion is performed on the target text to obtain a target speech of the source language speech. That is to say, the trained first sequence converter and the trained second sequence converter in the present application establish the connection between the speech recognition model, translation model and text-to-speech model in the related art; solve the problem of the lack of contextual association between different models in the related art, which leads to poor recognition performance of the entire system. When generating the target language text, the present application refers to the contextual connection of the output of the trained first sequence converter carrying the acoustic information hidden representation of the source language speech, that is, the first hidden state, so that the generated target language text has higher accuracy; and when generating the target language speech, the present application refers to the contextual connection of the output of the trained second sequence converter carrying the acoustic information hidden representation of the source language speech and the contextual connection of the hidden representation of the target text, that is, the third hidden state, so that the generated target language speech has higher accuracy. At the same time, the structure of the speech translation model obtained by the present application does not need to be redesigned. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of an implementation flow of the speech translation method provided in an embodiment of the present application;

[0021] Figure 2 This is a schematic diagram of the topological structure of the speech translation model provided in this application;

[0022] Figure 3 This is a schematic diagram of an implementation flow of the speech translation method provided in an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of the structure of the first sequence converter trained in this application;

[0024] Figure 5 This is a schematic diagram of the structure of the speech translation model provided by this application using DARTS neural architecture search to determine the optimal components;

[0025] Figure 6 A schematic diagram of the structure of a speech translation device provided for the implementation of this application;

[0026] Figure 7A schematic structural diagram of an electronic device provided for the implementation of this application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it will be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the art to which the embodiments of this application pertain. The terms used in the embodiments of this application are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0029] See also Figure 1 , Figure 1 1 is a schematic diagram of an implementation flow of a speech translation method provided in an embodiment of the present application. The speech translation method can be applied to electronic devices. The speech translation method includes the following steps:

[0030] Step 101: Acquire acoustic information and a first hidden state of the source language speech.

[0031] Among them, the first hidden state is used to represent the fusion weight of the hidden representation of the source language speech.

[0032] In an embodiment of the present application, an electronic device obtains a speech recognition model to be trained, a translation model to be trained, and a text-to-speech model to be trained; wherein the speech recognition model to be trained, the translation model to be trained, and the text-to-speech model to be trained are all codec structures. The speech recognition model is used to convert source language speech into source language text; the translation model is used to convert source language text into target language text; and the text-to-speech model is used to convert target language text into target language speech.

[0033] In some embodiments, the dimensions of the sequences processed by the speech recognition model to be trained, the translation model to be trained, and the text-to-speech model to be trained are the same.

[0034] In the embodiment of the present application, the electronic device extracts the encoder of the speech recognition model to be trained as the encoder component of the speech translation model of the present application. It should be noted that the electronic device trains the encoder component based on the source language sample speech to obtain a trained encoder component.

[0035] In the embodiment of the present application, the first hidden state is a fusion weight obtained after the trained encoder component encodes the source language speech and is used to characterize the hidden representation of the source language speech.

[0036] In an embodiment of the present application, an electronic device obtains source language speech, inputs the source language speech into a trained encoder component, obtains a source language vector carrying a first hidden state; and inputs the source language vector carrying the first hidden state into a first attention component to obtain acoustic information of the source language speech. Here, the first hidden state is the state value of the last step in the hidden layer of the trained encoder component, and the value of the first hidden state can be represented by a vector; the first hidden state in the present application is used to characterize the fusion weight of the hidden representation of the source language speech. It should be noted that, except for the input layer and the output layer of the encoder component, the other layers are called hidden layers, and the hidden layers do not directly receive signals from the outside world, nor do they directly send signals to the outside world.

[0037] Step 102: Input the acoustic information and the first hidden state into the trained first sequence converter to obtain the source language text and the second hidden state.

[0038] Among them, the second hidden state is used to represent the contextual connection of the hidden representation of the acoustic information of the source language speech.

[0039] In an embodiment of the present application, the electronic device inputs the acoustic information and the first hidden state of the source language speech into a trained first sequence converter, directly obtaining the source language text corresponding to the source language speech, and the second hidden state for characterizing the contextual connection of the hidden representation of the acoustic information of the source language speech.

[0040] In some embodiments, the electronic device outputs the source language text U=[u1,...,u L ], L is the sequence length corresponding to the source language text.

[0041] In some embodiments, the trained first sequence converter first processes the trained encoder component and the first attention component, and then decodes the acoustic information and the first hidden state of the source language speech to obtain the source language text corresponding to the source language speech. It should be noted that the trained first sequence converter has the same topological structure as the encoder of the translation model to be trained. That is, after obtaining the source language text, the trained first sequence converter encodes the source speech text according to the source speech text and the first hidden state to obtain a second hidden state for characterizing the contextual connection of the acoustic information hidden representation of the source language speech. That is, the trained first sequence converter in this application couples the functions of the decoder component of the speech recognition model and the encoder component of the translation model in the related art, that is, the vector output by the trained first sequence converter carries the contextual connection of the acoustic information hidden representation of the source language speech. When the target text generator component generates the target language text according to the content output by the trained first sequence converter, it will refer to the contextual connection of the acoustic information hidden representation of the source language speech. In this way, the target language text generated by the target text generator component is more accurate, and the trained first sequence converter establishes the connection between the speech recognition model and the translation model.

[0042] In the embodiment of the present application, the acoustic information of the source language speech includes but is not limited to the linear prediction coefficient (LPC), reflection coefficient, Mel-scale frequency cepstral coefficient (MFCC) and perceptual linear prediction (PLP) of the source language speech.

[0043] Step 103: Input the source language text and the second hidden state into the trained second sequence converter to obtain the target text corresponding to the source language speech.

[0044] In the embodiment of the present application, the electronic device inputs the source language text into the second attention component, obtains the reference weight of the source language text, and weights the source language text according to the reference weight to obtain the weighted source language text A MT Furthermore, the weighted source language text and the second hidden state are input into the trained second sequence converter to obtain the target text T = [t1, ..., t F ]. Here, the target text is the target speech text, that is, when the trained second sequence converter generates the target language text, it will refer to the contextual connection of the acoustic information hidden representation of the source language speech, so that the generated target language text is more accurate.

[0045] In the embodiment of the present application, the target language can be any language different from the source language.

[0046] In the embodiment of the present application, the trained second sequence converter first processes the target text processed by the trained second sequence converter according to the second hidden state to obtain the target language text T = [t1, ..., t F ]. Wherein, F represents the length corresponding to the target language text. It should be noted that the trained second sequence converter has the same topological structure as the encoder of the text-to-speech model to be trained. That is to say, after obtaining the target language text, the trained second sequence converter encodes the target language text according to the target language text and the second hidden state, and obtains the contextual connection of the acoustic information hidden representation for representing the source language speech, and the third hidden state for representing the contextual connection of the target text hidden representation, that is, the vector output by the trained second sequence converter carries the contextual connection of the acoustic information hidden representation of the source language speech and the contextual connection of the target text hidden representation. When the target speech generator component generates the target language speech according to the content output by the trained second sequence converter, it will refer to the contextual connection of the acoustic information hidden representation of the source language speech and the contextual connection of the target text hidden representation. In this way, the target language speech generated by the target speech generator component is more accurate, and the trained second sequence converter establishes a connection between the translation model and the text-to-speech model.

[0047] Step 104: Perform speech conversion on the target text to obtain a target speech of the source language speech.

[0048] In the embodiment of the present application, the electronic device inputs the target text into the target speech generator component to obtain the target language speech of the source language speech.

[0049] In some embodiments, the speech translation model is based on the trained first sequence of the transformer output hidden state The decoder of the translation model is used to convert the source language speech sequence into the target language text output T = [t1,...,t F ];

[0050] t f ,a f =Decoder(t f-1 ) (1)

[0051] Wherein, f represents the position of the text to be converted in the target language text.

[0052] It should be noted that the speech translation model is based on the second attention component to obtain the weighted source language text A MT =[a1,...,aL ]. This application is based on the trained second sequence converter to convert the weighted source language text A MT The linguistic latent representation of is mapped to the acoustic latent representation H of the text-to-speech model encoder. e Finally, use the target speech generator component to generate the speech sequence Y in the target language:

[0053] H e =Switching(A MT ) (2)

[0054] Y=Decoder(H e ) (3)

[0055] Figure 2 This is a schematic diagram of the topological structure of the speech translation model provided in this application. The topological structure of the speech translation model provided in this application includes a trained encoder component, a first attention component, a source language text decoder, a trained first sequence converter, a second attention component, a trained second sequence converter, a target speech generator component, and a target language text decoder. A source language speech sequence is input into the speech translation model provided in this application, and a target language speech sequence is output.

[0056] The present application discloses a speech translation method, comprising: obtaining acoustic information and a first hidden state of a source language speech; wherein the first hidden state is used to represent a fusion weight of a hidden representation of the source language speech; inputting the acoustic information and the first hidden state into a trained first sequence converter to obtain a source language text and a second hidden state; wherein the second hidden state is used to represent a contextual connection of the hidden representation of the acoustic information of the source language speech; inputting the source language text and the second hidden state into a trained second sequence converter to obtain a target text corresponding to the source language speech; and performing speech conversion on the target text to obtain a target speech of the source language speech. That is to say, the trained first sequence converter and the trained second sequence converter in the present application establish the connection between the speech recognition model, translation model and text-to-speech model in the related art; solve the problem of lack of contextual association between different models in the related art, resulting in poor recognition performance of the entire system. When generating the target language text, the present application refers to the contextual connection of the hidden representation of the acoustic information of the source language speech carried by the output of the trained first sequence converter, that is, the first hidden state, so that the generated target language text has higher accuracy; and when generating the target language speech, the present application refers to the contextual connection of the hidden representation of the acoustic information of the source language speech carried by the output of the trained second sequence converter and the contextual connection of the hidden representation of the target text, that is, the third hidden state, so that the generated target language speech has higher accuracy.

[0057] See also Figure 3 , Figure 3 1 is a schematic diagram of an implementation flow of a speech translation method provided in an embodiment of the present application. The speech translation method can be applied to electronic devices. The speech translation method includes the following steps:

[0058] Step 301: Acquire acoustic information and a first hidden state of the source language speech.

[0059] Among them, the first hidden state is used to represent the fusion weight of the hidden representation of the source language speech.

[0060] Step 302: Input the acoustic information and the first hidden state into the trained first sequence converter to obtain the source language text and the second hidden state.

[0061] Among them, the second hidden state is used to represent the contextual connection of the hidden representation of the acoustic information of the source language speech.

[0062] In other embodiments of the present application, the trained first sequence converter in step 302 can be implemented by the following steps:

[0063] Step A1: Create a first sequence converter and encoder component.

[0064] In an embodiment of the present application, an electronic device obtains a speech recognition model to be trained, a translation model to be trained, and a text-to-speech model to be trained; wherein the speech recognition model to be trained, the translation model to be trained, and the text-to-speech model to be trained are all codec structures. The speech recognition model is used to convert source language speech into source language text; the translation model is used to convert source language text into target language text; and the text-to-speech model is used to convert target language text into target language speech.

[0065] In an embodiment of the present application, the decoder of the speech recognition model to be trained is taken out as the source language text generator component in the speech translation model of the present application, the decoder of the translation model to be trained is taken out as the source language text generator component in the speech translation model of the present application, and the decoder of the text-to-speech model to be trained is taken out as the generator component of the target language speech of the present application.

[0066] In the embodiment of the present application, the electronic device extracts the encoder of the speech recognition model to be trained as the encoder component of the speech translation model of the present application and creates a first sequence converter having the same topological structure as the encoder of the translation model.

[0067] Step A2: Input the source language sample speech into the encoder component to obtain a speech vector carrying the fourth hidden state.

[0068] Among them, the fourth hidden state is used to represent the fusion weight of the hidden representation of the source language sample speech.

[0069] In the embodiment of the present application, the electronic device inputs the source language sample speech into the encoder component to obtain a speech vector carrying the fourth hidden state. It should be noted that the first hidden state and the fourth hidden state are output by the encoder component with different parameters.

[0070] Step A3: Input the speech vector carrying the fourth hidden state into the first attention component to obtain the acoustic information of the source language sample speech.

[0071] In the embodiment of the present application, the electronic device inputs the speech vector carrying the fourth hidden state into the first attention component to obtain the acoustic information A of the source language sample speech. ASR ;in, Q is used to represent the length of the acoustic information of the source language sample speech. The acoustic information of the source language sample speech will be used as input for the next step of speech-to-text conversion.

[0072] Step A4: Input the acoustic information of the source language sample speech and the fourth hidden state into the first sequence converter to obtain the source language training text and the first reference information.

[0073] The first reference information is used to characterize the contextual connection of the acoustic information hidden representation of the source language sample speech.

[0074] In the embodiment of the present application, according to formula (4), the electronic device converts the acoustic information A of the source language sample speech into ASR And the fourth hidden state is input into the first sequence converter to obtain the source language training text and the first reference information H TC Here, the first reference information has the same dimension as the second hidden state and is output by the first sequence converter with different parameters.

[0075] H TC =Switching(A ASR ) (4)

[0076] Step A5: The encoder of the translation model encodes the source language sample text to obtain first target information.

[0077] The trained first sequence transformer has the same topology as the encoder of the translation model.

[0078] In the embodiment of the present application, the first target information and the second reference information have the same dimension.

[0079] In the embodiment of the present application, the source language sample text corresponds to the source language sample speech, and the source language sample text is obtained by processing the source language sample speech by a speech recognition model in the related art.

[0080] In the embodiment of the present application, according to formula (5), the electronic device uses the first target information H output by the translation model encoder S As the training target, the source language training text S is input into the encoder of the translation model to obtain the first target information H S .

[0081] H S =MTEncoder(S) (5)

[0082] Step A6: Training the first sequence converter and encoder component based on the first reference information and the first target information.

[0083] The first target information is used to represent the contextual connection of the hidden representation of the source language training text.

[0084] In the embodiment of the present application, the electronic device may train the first sequence converter and the encoder component according to the similarity between the first reference information and the first target information.

[0085] In the embodiment of the present application, the electronic device may train the first sequence converter and the encoder component according to the loss value between the first reference information and the first target information.

[0086] In other embodiments of the present application, step A6 may train the first sequence converter and encoder component based on the first reference information and the first target information by the following steps:

[0087] Step A61: Determine a first loss value between first target information and first reference information.

[0088] In the embodiment of the present application, the first target information is H S , the first reference information is H TC For example, the electronic device calculates H according to formula (6) TC and H S The loss value between .

[0089]

[0090] Here, l represents the index of the source language text.

[0091] Step A62: If the first loss value is greater than the loss threshold, adjust the parameters of the encoder component and / or the first sequence converter to obtain the encoder component and / or the first sequence converter after adjusting the parameters.

[0092] In the embodiment of the present application, if the first loss value is less than the loss threshold, it is determined that the encoder component is a trained encoder component and the first sequence converter is a trained first sequence converter.

[0093] In the embodiment of the present application, the loss threshold may be pre-set by the electronic device or may be set by the user according to actual circumstances, and the present application does not impose any limitation on this. The present application takes the loss threshold of 0.05 as an example.

[0094] In the embodiment of the present application, if the first loss value is less than the loss threshold, the electronic device fixes the parameters of other components of the speech translation model and only adjusts the weight parameters of the encoder component and / or the first sequence converter so that the first reference information H output by the first sequence converter is TC Approaching the first target information H S , even if H TC and H S The loss value between them is less than the loss threshold.

[0095] Step A63: Continue to process the source language sample speech according to the encoder component after adjusting the parameters and the first sequence converter after adjusting the parameters until the loss value between the reference information obtained after processing and the first target information is less than the loss threshold, and determine that the training of the encoder component and the first sequence converter is completed.

[0096] Figure 4 This is a schematic diagram of the structure of the first sequence converter trained in this application, such as Figure 4 As shown, the source language speech sequence passes through the encoder component, the first attention component, and the first sequence converter in sequence to obtain the hidden state output by the first sequence converter, namely the first reference information; the source language text sequence is input into the translation model encoder to obtain the hidden state output by the translation model, namely the first target data, and the first sequence converter is trained based on the hidden state output by the translation model.

[0097] Step 303: Input the source language text and the second hidden state into the trained second sequence converter to obtain the corresponding target text of the source language speech carrying the third hidden state.

[0098] The third hidden state includes the contextual connections for representing the acoustic information hidden representation of the source language speech and the contextual connections for representing the target text hidden representation.

[0099] In other embodiments of the present application, the trained second sequence converter in step 303 can be implemented by the following steps:

[0100] Step B1: Create a second sequence converter.

[0101] In the embodiment of the present application, the electronic device creates a second sequence converter having the same topology as the encoder of the text-to-speech model.

[0102] Step B2: Input the source language training text into the second attention component to obtain the reference weight of the source language training text.

[0103] In an embodiment of the present application, when the training of the first sequence converter is completed, the source language training text output by the first sequence converter is obtained and input into the second attention component to obtain a reference weight of the source language training text.

[0104] Step B3: Input the source language training text with the reference weight into the second sequence converter to obtain the target language training text and the second reference information.

[0105] The second reference information includes a contextual connection for representing the hidden representation of acoustic information of the source language sample speech and a contextual connection for representing the hidden representation of the target language training text.

[0106] Step B4: The encoder of the text-to-speech model encodes the target language sample text to obtain second target information.

[0107] The trained second sequence converter has the same topology as the encoder of the text-to-speech model.

[0108] In an embodiment of the present application, the target language sample text corresponds to the source language sample speech. The target language sample text is first processed by a speech recognition model in the related art to obtain the source language sample text, and then the source language sample text is processed by a translation model in the related art to obtain the target language sample text; the encoder of the text-to-speech model encodes the target language sample text to obtain the second target information.

[0109] In some embodiments, the target language sample text corresponds to the source language sample speech. The target language sample text is first processed by a trained first sequence converter on the source language sample speech to obtain a source language training text, and then the source language training text is processed by a translation model in the related technology to obtain the target language sample text; the encoder of the text-to-speech model encodes the target language sample text to obtain the second target information.

[0110] Step B5: Training the second sequence converter based on the second reference information and the second target information; wherein the second target information is used to characterize the contextual connection of the hidden representation of the target language training text.

[0111] In the embodiment of the present application, the electronic device may train the second sequence converter and the encoder component according to the similarity between the second reference information and the second target information.

[0112] In the embodiment of the present application, the electronic device may train the second sequence converter and the encoder component according to the loss value between the second reference information and the second target information.

[0113] In other embodiments of the present application, step B5 may be performed by training the second sequence converter based on the second reference information and the second target information through the following steps:

[0114] Step B51: Determine a second loss value between the second target information and the second reference information.

[0115] In the embodiment of the present application, the second reference information is H TC , the second target information is H S For example, the electronic device calculates H according to formula (6) TC and H S The loss value between .

[0116] Step B52: If the second loss value is greater than the loss threshold, adjust the parameters of the second sequence converter to obtain a second sequence converter with adjusted parameters.

[0117] In the embodiment of the present application, if the first loss value is less than the loss threshold, it is determined that the second sequence converter is a trained second sequence converter.

[0118] In the embodiment of the present application, if the second loss value is less than the loss threshold, the electronic device fixes the parameters of other components of the speech translation model and only adjusts the weight parameters of the second sequence converter so that the second reference information H output by the second sequence converter is TC Approaching the second target information H S , even if H TC and H S The loss value between them is less than the loss threshold.

[0119] Step B53: Continue to process the source language training text according to the second sequence converter with adjusted parameters until the loss value between the reference information obtained after processing and the second target information is less than the loss threshold, and determine that the training of the second sequence converter is completed.

[0120] In the embodiment of the present application, after the second sequence converter, the second sequence converter and the encoder component training are completed, the trained second sequence converter, the trained first sequence converter, the trained encoder component and the target speech generator component are formed into the speech translation model of this application. It should be noted that the speech translation model of the present application is coupled, and the internal structure of each component is not optimal. The application uses differentiable architecture search (Differentiable Architecture Search, DARTS) neural architecture search to carry out a joint search on the network topology structure inside each component, so that the performance of the coupled model is optimal.

[0121] It should be noted that the present application defines the search space as a directed acyclic graph (DAG). (i) is the potential variable to be selected, and each directed edge (i, j) corresponds to a specific operation o (i,j) , which corresponds to the specific operators inside each component in the network, such as activation functions, convolution kernels of different sizes, and fully connected layers. (j) The output of comes from the input of all nodes before this node:

[0122]

[0123] Among them, i and j represent the node numbers, and the intermediate node is the sum of all the intermediate nodes after operation.

[0124] make is the operator search space of length M. According to formula (8), the final output of each directed edge is:

[0125]

[0126] Among them, α0 is the initial configuration set, η and β are soft loading factors and hard loading factors respectively, and η and β are hyperparameters that DARTS dynamically learns during the search process. This application defines the training set loss and validation set loss as l train 、l val According to formula (9), the optimal configuration set to be searched is α, and α is updated using the gradient descent strategy:

[0127]

[0128] Among them, ξ is the set learning rate, w and w' are the network weights before and after differentiation respectively.

[0129] Then, determine the base search space Ω for each component p . Figure 5 This is a schematic diagram of the structure of the speech translation model provided by this application using DARTS neural architecture search to determine the optimal components, such as Figure 5As shown, the basic search space of each component contains several operators, such as operator 1, operator 2, operator 3 and operator n. Each operator is connected to each other according to pre-set rules. Each line in the figure (except the output end) represents a candidate operation and its weight. n represents the number of inputs to be selected. For nodes with multiple inputs, all inputs are summed. This application normalizes the loss value between each operator and sums the loss values ​​of all components, such as component 1, component 2 and component m, to obtain the overall loss value of the speech translation model provided by this application. Wherein, m represents the number of all components in the speech translation model provided by this application.

[0130] Finally, the gradient descent is repeatedly performed to find the configuration α that minimizes the loss function value as the final structure of the translation network. Using the above search algorithm, the speech translation model provided by this application has strong robustness, so that the coupled model has a certain error correction capability. At the same time, the search space is searched through the DARTS neural architecture to integrate the intrinsic information of the speech translation model, and the loss function contains learnable hyperparameters and has self-optimization capabilities. Multiple attention mechanisms are used to reduce the spatial differences between components.

[0131] Step 304: Perform speech conversion on the target text carrying the third latent state to obtain the target speech.

[0132] It should be noted that, for the description of the same steps and contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0133] Based on the above embodiments, the embodiments of the present application provide a speech translation device, which can be applied to Figure 1 、 Figure 3 In the speech translation method provided in the corresponding embodiment, refer to Figure 6 As shown, the speech translation device 6 may include:

[0134] An acquisition unit 602 is configured to acquire acoustic information and a first hidden state of the source language speech; wherein the first hidden state is used to represent a fusion weight of a hidden representation of the source language speech;

[0135] Processing unit 601 is configured to input the acoustic information and the first hidden state into a trained first sequence converter to obtain a source language text and a second hidden state; wherein the second hidden state is used to represent the contextual relationship of the hidden representation of the acoustic information of the source language speech;

[0136] The processing unit 601 is further configured to input the source language text and the second hidden state into a trained second sequence converter to obtain a target text corresponding to the source language speech;

[0137] The processing unit 601 is further configured to perform speech conversion on the target text to obtain a target speech of the source language speech.

[0138] In other embodiments of the present application, the speech translation device 6 further includes:

[0139] The processing unit 601 is used to perform speech conversion on the target text carrying the third latent state to obtain the target speech, wherein the target text carries the third latent state, and the third latent state includes the contextual connection for representing the hidden representation of the acoustic information of the source language speech, and the contextual connection for representing the hidden representation of the target text.

[0140] In other embodiments of the present application, the speech translation device 6 further includes:

[0141] Processing unit 601 is used to create a first sequence converter and encoder component;

[0142] The processing unit 601 is further configured to input the source language sample speech into the encoder component to obtain a speech vector carrying a fourth hidden state; the fourth hidden state is a fusion weight for characterizing the hidden representation of the source language sample speech;

[0143] The processing unit 601 is further configured to input the speech vector carrying the fourth hidden state into the first attention component to obtain acoustic information of the source language sample speech;

[0144] The processing unit 601 is further configured to input the acoustic information of the source language sample speech and the fourth hidden state into the first sequence converter to obtain the source language training text and the first reference information;

[0145] The processing unit 601 is further configured to train the first sequence converter and encoder component based on the first reference information and the first target information, wherein the first target information is used to represent the contextual relationship of the hidden representation of the source language training text.

[0146] In other embodiments of the present application, the speech translation device 6 further includes:

[0147] The processing unit 601 is configured to determine a first loss value between first target information and first reference information;

[0148] The processing unit 601 is further configured to adjust parameters of the encoder component and / or the first sequence converter if the first loss value is greater than the loss threshold, to obtain an encoder component after parameter adjustment and / or a first sequence converter after parameter adjustment;

[0149] The processing unit 601 is further used to continue processing the source language sample speech according to the encoder component after adjusting the parameters and the first sequence converter after adjusting the parameters until the loss value between the reference information obtained after processing and the first target information is less than the loss threshold, and it is determined that the training of the encoder component and the first sequence converter is completed.

[0150] In other embodiments of the present application, the speech translation device 6 further includes:

[0151] Processing unit 601, configured to create a second sequence converter;

[0152] The processing unit 601 is further configured to input the source language training text into the second attention component to obtain a reference weight of the source language training text;

[0153] The processing unit 601 is further configured to input the source language training text with the reference weight into a second sequence converter to obtain the target language training text and second reference information;

[0154] The processing unit 601 is further configured to train the second sequence converter based on the second reference information and the second target information; wherein the second target information is used to represent the contextual relationship of the hidden representation of the target language training text.

[0155] In other embodiments of the present application, the speech translation device 6 further includes:

[0156] The processing unit 601 is configured to determine a second loss value between the second target information and the second reference information;

[0157] The processing unit 601 is further configured to adjust parameters of the second sequence converter if the second loss value is greater than the loss threshold, to obtain a second sequence converter with adjusted parameters;

[0158] The processing unit 601 is further configured to continue processing the source language training text according to the second sequence converter after adjusting the parameters, until the loss value between the reference information obtained after processing and the second target information is less than the loss threshold, thereby determining that the training of the second sequence converter is completed.

[0159] In other embodiments of the present application, the speech translation device 6 further includes:

[0160] Processing unit 601 is used for an encoder of a translation model to encode a source language sample text to obtain first target information; wherein the trained first sequence converter has the same topological structure as the encoder of the translation model.

[0161] In other embodiments of the present application, the speech translation device 6 further includes:

[0162] Processing unit 601, an encoder for a text-to-speech model encodes a target language sample text to obtain second target information; wherein the trained second sequence converter has the same topological structure as the encoder of the text-to-speech model.

[0163] It should be noted that the specific implementation process of the processing unit 601 and the obtaining unit 602 and the steps performed in this embodiment can be referred to Figure 1 、 Figure 3 The implementation process of the speech translation method provided in the corresponding embodiment will not be repeated here.

[0164] The present application provides an electronic device which can be used in Figure 1 、 Figure 3 In a speech translation method provided by a corresponding embodiment, referring to Figure 7 As shown, the electronic device 7 includes: a processor 701, a memory 702 and a communication bus 703, wherein:

[0165] The communication bus 703 is used to implement communication between the processor 701 and the memory 702 .

[0166] The processor 701 is configured to execute the speech translation program stored in the memory 702 to implement the following steps:

[0167] Acquiring acoustic information and a first hidden state of the source language speech; wherein the first hidden state is used to represent the fusion weight of the hidden representation of the source language speech;

[0168] Inputting the acoustic information and the first hidden state into the trained first sequence converter to obtain the source language text and the second hidden state; wherein the second hidden state is used to represent the contextual relationship of the hidden representation of the acoustic information of the source language speech;

[0169] Inputting the source language text and the second hidden state into the trained second sequence converter to obtain the target text corresponding to the source language speech;

[0170] Perform speech conversion on the target text to obtain the target speech of the source language speech.

[0171] The processor 701 is configured to execute the speech translation program stored in the memory 702 to implement the following steps:

[0172] A target text carrying a third latent state is converted into speech to obtain a target speech; the target text carries a third latent state, and the third latent state includes a contextual connection for representing a hidden representation of acoustic information of the source language speech and a contextual connection for representing a hidden representation of the target text.

[0173] The processor 701 is configured to execute the speech translation program stored in the memory 702 to implement the following steps:

[0174] Create the first sequence converter and encoder components;

[0175] Input the source language sample speech into the encoder component to obtain a speech vector carrying the fourth hidden state; the fourth hidden state is the fusion weight used to characterize the hidden representation of the source language sample speech;

[0176] Input the speech vector carrying the fourth hidden state into the first attention component to obtain the acoustic information of the source language sample speech;

[0177] Inputting the acoustic information of the source language sample speech and the fourth hidden state into the first sequence converter to obtain the source language training text and the first reference information;

[0178] A first sequence converter and encoder component are trained based on first reference information and first target information, wherein the first target information is used to characterize the contextual relationship of the hidden representation of the source language training text.

[0179] The processor 701 is configured to execute the speech translation program stored in the memory 702 to implement the following steps:

[0180] determining a first loss value between the first target information and the first reference information;

[0181] If the first loss value is greater than the loss threshold, adjusting parameters of the encoder component and / or the first sequence converter to obtain an encoder component after parameter adjustment and / or a first sequence converter after parameter adjustment;

[0182] Continue to process the source language sample speech according to the encoder component with adjusted parameters and the first sequence converter with adjusted parameters until the loss value between the reference information obtained after processing and the first target information is less than the loss threshold, and determine that the training of the encoder component and the first sequence converter is completed.

[0183] The processor 701 is configured to execute the speech translation program stored in the memory 702 to implement the following steps:

[0184] Create a second sequence converter;

[0185] Input the source language training text into the second attention component to obtain the reference weight of the source language training text;

[0186] Inputting the source language training text with the reference weight into the second sequence converter to obtain the target language training text and second reference information;

[0187] The second sequence converter is trained based on the second reference information and the second target information, wherein the second target information is used to represent the contextual connection of the hidden representation of the target language training text.

[0188] The processor 701 is configured to execute the speech translation program stored in the memory 702 to implement the following steps:

[0189] determining a second loss value between the second target information and the second reference information;

[0190] If the second loss value is greater than the loss threshold, adjusting the parameters of the second sequence converter to obtain a second sequence converter with adjusted parameters;

[0191] The source language training text is continuously processed according to the second sequence converter with adjusted parameters until the loss value between the reference information obtained after processing and the second target information is less than the loss threshold, and the training of the second sequence converter is determined to be completed.

[0192] The processor 701 is configured to execute the speech translation program stored in the memory 702 to implement the following steps:

[0193] The encoder of the translation model encodes the source language sample text to obtain first target information; wherein the trained first sequence converter has the same topological structure as the encoder of the translation model.

[0194] The processor 701 is configured to execute the speech translation program stored in the memory 702 to implement the following steps:

[0195] The encoder of the text-to-speech model encodes the target language sample text to obtain second target information; wherein the trained second sequence converter has the same topological structure as the encoder of the text-to-speech model.

[0196] It should be noted that the specific implementation process of the steps executed by the processor 701 in this embodiment can be referred to Figure 1 、 Figure 3 The implementation process of the speech translation method provided in the corresponding embodiment will not be repeated here.

[0197] Based on the foregoing embodiments, embodiments of the present application provide a computer-readable storage medium storing one or more applications, which can be executed by one or more processors to implement the following. Figure 1 、 Figure 3 The corresponding embodiment provides a speech translation method.

[0198] It should be noted that the above-mentioned computer-readable storage medium can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface storage, an optical disc, or a compact disc read-only memory (CD-ROM); it can also be various electronic devices that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0199] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0200] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0201] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0202] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer application products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer application instructions. These computer application instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0203] These computer application instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0204] These computer application instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0205] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A speech translation method, characterized in that: The method comprises: Acquiring acoustic information and a first hidden state of a source language speech; wherein the first hidden state is used to represent a fusion weight of a hidden representation of the source language speech; Inputting the acoustic information and the first hidden state into a trained first sequence converter to obtain a source language text and a second hidden state; wherein the second hidden state is used to represent the contextual relationship of the hidden representation of the acoustic information of the source language speech; Inputting the source language text and the second hidden state into a trained second sequence converter to obtain a target text corresponding to the source language speech; Performing speech conversion on the target text to obtain a target speech of the source language speech; The method further comprises: Create the first sequence converter and encoder components; Inputting a source language sample speech into the encoder component to obtain a speech vector carrying a fourth hidden state; the fourth hidden state is used to represent the fusion weight of the hidden representation of the source language sample speech; Inputting the speech vector carrying the fourth hidden state into the first attention component to obtain acoustic information of the source language sample speech; Inputting the acoustic information of the source language sample speech and the fourth hidden state into the first sequence converter to obtain a source language training text and first reference information; The first sequence converter and the encoder component are trained based on the first reference information and first target information; wherein the first target information is used to represent the contextual relationship of the hidden representation of the source language training text.

2. The method according to claim 1, characterized in that The target text carries a third hidden state, the third hidden state including a contextual relationship for representing a hidden representation of acoustic information of the source language speech and a contextual relationship for representing a hidden representation of the target text; The performing speech conversion on the target text to obtain the target speech of the source language speech includes: Perform speech conversion on the target text carrying the third latent state to obtain the target speech.

3. The method according to claim 1, characterized in that The training of the first sequence converter and the encoder component based on the first reference information and the first target information includes: determining a first loss value between the first target information and the first reference information; If the first loss value is greater than a loss threshold, adjusting parameters of the encoder component and / or the first sequence converter to obtain an encoder component after parameter adjustment and / or a first sequence converter after parameter adjustment; Continue to process the source language sample speech according to the encoder component after adjusting the parameters and the first sequence converter after adjusting the parameters until the loss value between the reference information obtained after processing and the first target information is less than the loss threshold, and determine that the training of the encoder component and the first sequence converter is completed.

4. The method according to claim 1, wherein The method further comprises: Create a second sequence converter; Inputting the source language training text into the second attention component to obtain a reference weight of the source language training text; Inputting the source language training text with the reference weight into the second sequence converter to obtain the target language training text and second reference information; The second sequence converter is trained based on the second reference information and the second target information; wherein the second target information is used to represent the contextual relationship of the hidden representation of the target language training text.

5. The method according to claim 4, characterized in that The training of the second sequence converter based on the second reference information and the second target information includes: determining a second loss value between the second target information and the second reference information; If the second loss value is greater than the loss threshold, adjusting the parameters of the second sequence converter to obtain a second sequence converter with adjusted parameters; The source language training text is continuously processed according to the second sequence converter after adjusting the parameters until the loss value between the reference information obtained after processing and the second target information is less than the loss threshold, and it is determined that the training of the second sequence converter is completed.

6. The method according to claim 1, characterized in that Before training the first sequence converter and the encoder component based on the first reference information and the first target information, the method further includes: The encoder of the translation model encodes the source language training text to obtain the first target information; wherein the trained first sequence converter has the same topological structure as the encoder of the translation model.

7. The method according to claim 4, characterized in that Before training the second sequence converter based on the second reference information and the second target information, the method further includes: The encoder of the text-to-speech model encodes the target language training text to obtain the second target information; wherein the trained second sequence converter has the same topological structure as the encoder of the text-to-speech model.

8. A speech translation device, characterized in that: The speech translation device comprises: an acquisition unit, configured to acquire acoustic information and a first hidden state of a source language speech; wherein the first hidden state is used to represent a fusion weight of a hidden representation of the source language speech; a processing unit configured to input the acoustic information and the first hidden state into a trained first sequence converter to obtain a source language text and a second hidden state; wherein the second hidden state is used to represent a contextual relationship of a hidden representation of the acoustic information of the source language speech; The processing unit is further configured to input the source language text and the second hidden state into a trained second sequence converter to obtain a target text corresponding to the source language speech; The processing unit is further configured to perform speech conversion on the target text to obtain a target speech of the source language speech; The processing unit is further used to create a first sequence converter and an encoder component; input the source language sample speech into the encoder component to obtain a speech vector carrying a fourth hidden state; the fourth hidden state is used to represent the fusion weight of the hidden representation of the source language sample speech; input the speech vector carrying the fourth hidden state into the first attention component to obtain acoustic information of the source language sample speech; input the acoustic information of the source language sample speech and the fourth hidden state into the first sequence converter to obtain source language training text and first reference information; based on the first reference information and first target information, the first sequence converter and the encoder component are trained; wherein the first target information is used to represent the contextual connection of the hidden representation of the source language training text.

9. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable instructions; The processor is configured to implement the speech translation method according to any one of claims 1 to 7 when executing the executable instructions stored in the memory.

Images