Audio translation method based on end-to-end speech translation model and related equipment

Abstract

The invention belongs to the technical field of voice processing, and relates to an end-to-end voice translation model-based audio translation method and related equipment, and the method comprises the steps: carrying out the encoding of a to-be-processed audio sent by a user terminal through a voice feature encoder, and extracting important feature data; further encoding through a semantic feature encoder, and converting into semantic feature data; the source language decoder decodes the semantic feature data into a source language text, so that the system can understand and process original voice information; the text sequence decoder decodes the semantic feature data by using the source language text to generate detailed text sequence data; and finally, splitting the text sequence data, and generating and outputting an accurate target translation text. According to the invention, through close cooperation of the encoders and the decoders, high-accuracy translation from audios to texts is effectively realized, and the accuracy of translation contents and proper expression of contexts are ensured; in addition, the system can process multiple languages, and the applicability and flexibility of the system are enhanced.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to an audio translation method and related equipment based on an end-to-end speech translation model. Background Art

[0002] In the audio translation system, we face a complex technical challenge. When users upload audio to be processed, the system needs to handle two tasks at the same time: speech recognition and translation. However, the traditional process is to first perform speech recognition to obtain the source language text, and then perform text translation. This method has the problems of information loss and error accumulation.

[0003] There is an audio translation method that designs an end-to-end audio translation model to simultaneously process two tasks, speech recognition and translation.

[0004] However, the applicant has found that it is difficult for traditional audio translation methods to simultaneously optimize the performance of both speech recognition and translation tasks, resulting in the inability of traditional audio translation methods to achieve the technical effect of high-accuracy translation. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose an audio translation method and related equipment based on an end-to-end speech translation model to solve the problem that traditional audio translation methods cannot achieve the technical effect of high-accuracy translation.

[0006] In order to solve the above technical problems, the embodiment of the present application provides an audio translation method based on an end-to-end speech translation model, which adopts the following technical solution:

[0007] Receiving an audio translation request carrying audio to be processed sent by a user terminal;

[0008] Performing a speech feature encoding operation on the audio to be processed according to a speech feature encoder to obtain speech feature data;

[0009] Performing a semantic feature encoding operation on the speech feature data according to a semantic feature encoder to obtain semantic feature data;

[0010] Performing source language decoding operation on the semantic feature data according to the source language decoder to obtain source language text;

[0011] Decoding the semantic feature data according to the text sequence decoder and the source language text to obtain text sequence data;

[0012] Performing a splitting operation on the text sequence data to obtain a target translation text;

[0013] The target translation text is output to the user terminal.

[0014] Furthermore, after the step of performing a speech feature encoding operation on the audio to be processed according to the speech feature encoder to obtain speech feature data, the following steps are also included:

[0015] The speech feature data is subjected to contraction processing.

[0016] Furthermore, the step of performing a semantic feature encoding operation on the speech feature data according to the semantic feature encoder to obtain the semantic feature data specifically includes the following steps:

[0017] Extracting time series information of the speech feature data according to a bidirectional long short-term memory network;

[0018] Analyze the relationship between each element in the speech feature data and all other elements according to the self-attention mechanism and the time series information to obtain self-attention feature data;

[0019] Performing weighted fusion on the self-attention feature data to obtain weighted fused self-attention feature data;

[0020] The weighted fused self-attention feature data is nonlinearly transformed according to a multi-layer neural network to obtain the semantic feature data.

[0021] Furthermore, after the step of performing a semantic feature encoding operation on the speech feature data according to the semantic feature encoder to obtain the semantic feature data, the following step is also included:

[0022] The semantic feature data is clustered according to a clustering algorithm.

[0023] Furthermore, the step of decoding the semantic feature data according to the text sequence decoder and the source language text to obtain the text sequence data specifically includes the following steps:

[0024] Correcting the text sequence data according to a text error correction algorithm to obtain an initial corrected text;

[0025] Calculating the perplexity of the initial revised text according to the statistical language model, and re-recognizing the initial revised text whose perplexity is higher than a preset perplexity threshold to obtain an intermediate revised text;

[0026] Optimizing the boundary of the intermediate revised text according to the context analysis method to obtain revised text sequence data;

[0027] The step of performing a splitting operation on the text sequence data to obtain a target translation text specifically comprises the following steps:

[0028] The modified text sequence data is split to obtain the target translation text.

[0029] In order to solve the above technical problems, the embodiment of the present application also provides an audio translation device based on an end-to-end speech translation model, which adopts the following technical solution:

[0030] A request receiving module, used for receiving an audio translation request carrying audio to be processed sent by a user terminal;

[0031] A speech encoding module, used for performing a speech feature encoding operation on the audio to be processed according to a speech feature encoder to obtain speech feature data;

[0032] A semantic encoding module, used for performing a semantic feature encoding operation on the speech feature data according to a semantic feature encoder to obtain semantic feature data;

[0033] A source language decoding module, used for performing a source language decoding operation on the semantic feature data according to a source language decoder to obtain a source language text;

[0034] A text sequence decoding module, used for decoding the semantic feature data according to a text sequence decoder and the source language text to obtain text sequence data;

[0035] A splitting and obtaining module, used for performing a splitting operation on the text sequence data to obtain a target translation text;

[0036] An output module is used to output the target translation text to the user terminal.

[0037] Furthermore, the device also includes:

[0038] The contraction module is used to perform contraction processing on the speech feature data.

[0039] Furthermore, the semantic encoding module includes:

[0040] A time series extraction submodule, used for extracting time series information of the speech feature data according to a bidirectional long short-term memory network;

[0041] A self-attention feature acquisition submodule, used for analyzing the relationship between each element in the speech feature data and all other elements according to the self-attention mechanism and the time series information to obtain self-attention feature data;

[0042] A weighted fusion submodule is used to perform weighted fusion on the self-attention feature data.

[0043] Obtain weighted fused self-attention feature data;

[0044] The nonlinear transformation submodule is used to perform nonlinear transformation on the weighted fused self-attention feature data according to a multi-layer neural network to obtain the semantic feature data.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0046] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the audio translation method based on the end-to-end speech translation model when executing the computer-readable instructions.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0048] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the audio translation method based on the end-to-end speech translation model as described above are implemented.

[0049] The present application provides an audio translation method based on an end-to-end speech translation model, comprising: receiving an audio translation request carrying an audio to be processed sent by a user terminal; performing speech feature encoding operation on the audio to be processed according to a speech feature encoder to obtain speech feature data; performing semantic feature encoding operation on the speech feature data according to a semantic feature encoder to obtain semantic feature data; performing source language decoding operation on the semantic feature data according to a source language decoder to obtain a source language text; performing decoding operation on the semantic feature data according to a text sequence decoder and the source language text to obtain text sequence data; performing a split operation on the text sequence data to obtain a target translation text; and outputting the target translation text to the user terminal. Compared with the prior art, the present application effectively realizes high-accuracy translation from audio to text through the close cooperation of each encoder and decoder, ensuring the accuracy of the translation content and the appropriate expression of the context. In addition, the system can also handle multiple languages, enhancing its applicability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0052] Figure 2 It is a flow chart for implementing the audio translation method based on the end-to-end speech translation model provided in the first embodiment of the present application;

[0053] Figure 3 It is a structural diagram of an audio translation device based on an end-to-end speech translation model provided in Embodiment 2 of the present application;

[0054] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0056] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0058] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0059] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0060] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0061] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0062] It should be noted that the audio translation method based on the end-to-end speech translation model provided in the embodiment of the present application is generally executed by a server / terminal device. Accordingly, the audio translation device based on the end-to-end speech translation model is generally arranged in the server / terminal device.

[0063] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0064] Embodiment 1

[0065] Continue to refer Figure 2 , shows a flow chart of an embodiment of an audio translation method based on an end-to-end speech translation model according to the present application. The audio translation method based on the end-to-end speech translation model comprises: step S201, step S202, step S203, step S204, step S205, step S206 and step S207.

[0066] In step S201, an audio translation request carrying audio to be processed sent by a user terminal is received.

[0067] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for the convenience of understanding and are not used to limit the present application.

[0068] In step S202, a speech feature encoding operation is performed on the audio to be processed according to the speech feature encoder to obtain speech feature data.

[0069] In the embodiment of the present application, the speech feature encoder is mainly used to convert the sound signal into digital features for subsequent analysis and processing. The processing steps of the speech feature encoder are as follows:

[0070] (1) Analysis stage: extract speech features from the original sound waveform, such as linear spectrum, low frequency, MFCC, etc.

[0071] (2) Manipulation stage: The extracted speech features are compressed and processed to improve their representation ability.

[0072] (3) Synthesis stage: restore the processed speech features to the original sound waveform.

[0073] In an embodiment of the present application, the speech feature encoder can be a short-time Fourier transform (STFT), a multivariate linear model (MLM) or a deep neural network (DNN), etc. It should be understood that the examples of speech feature encoders here are only for convenience of understanding and are not used to limit the present application.

[0074] In step S203, a semantic feature encoding operation is performed on the speech feature data according to the semantic feature encoder to obtain semantic feature data.

[0075] In the embodiment of the present application, the semantic feature encoder is mainly used to convert natural language text into low-dimensional vector representations. The semantic feature encoder captures the semantic and grammatical structures in the language and uses these vector representations for subsequent machine translation work. The processing steps of the semantic feature encoder are as follows:

[0076] (1) Sequence encoding: Encode the input word vector to capture the semantic and grammatical information of the input text, such as using LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit) technologies;

[0077] (2) Pooling: Aggregate the sequence-encoded vectors into a fixed-length vector for subsequent processing, such as using average pooling or maximum pooling.

[0078] (3) Mapping: Map the pooled vector into a low-dimensional space to obtain a compact vector representation, namely the semantic feature data mentioned above.

[0079] In an embodiment of the present application, the semantic feature encoder can be a hidden Markov model (HMM), a language model (LM) or a neural network language model (NNLM), etc. It should be understood that the examples of semantic feature encoders here are only for convenience of understanding and are not used to limit the present application.

[0080] In step S204, source language decoding operation is performed on the semantic feature data according to the source language decoder to obtain source language text.

[0081] In the embodiment of the present application, the source language decoder is mainly used to decode the corresponding source language text from the semantic features.

[0082] In step S205, the semantic feature data is decoded according to the text sequence decoder and the source language text to obtain text sequence data.

[0083] In the embodiment of the present application, the text sequence decoder is mainly used to decode semantic features according to the source language text to obtain a corresponding text sequence.

[0084] In step S206, a splitting operation is performed on the text sequence data to obtain a target translation text.

[0085] In the embodiment of the present application, the text sequence is split to obtain the target language text corresponding to the speech to be processed. As an example, for example, after the above steps, the text sequence "see you <st>Goodbye", then you only need to split the text sequence according to the position of the link identifier, so as to translate the to-be-processed speech "see you" into the target language text "goodbye".

[0086] In step S207, the target translation text is output to the user terminal.

[0087] In an embodiment of the present application, an audio translation method based on an end-to-end speech translation model is provided, comprising: receiving an audio translation request carrying audio to be processed sent by a user terminal; performing speech feature encoding operation on the audio to be processed according to a speech feature encoder to obtain speech feature data; performing semantic feature encoding operation on the speech feature data according to a semantic feature encoder to obtain semantic feature data; performing source language decoding operation on the semantic feature data according to a source language decoder to obtain a source language text; performing decoding operation on the semantic feature data according to a text sequence decoder and the source language text to obtain text sequence data; performing a split operation on the text sequence data to obtain a target translation text; and outputting the target translation text to the user terminal. Compared with the prior art, the present application effectively realizes high-accuracy translation from audio to text through the close cooperation of each encoder and decoder, ensuring the accuracy of the translation content and the appropriate expression of the context. In addition, the system can also handle multiple languages, enhancing its applicability and flexibility.

[0088] In some optional implementations of the embodiments of the present application, after step S202, the following steps are further included:

[0089] Perform contraction processing on the speech feature data.

[0090] In the embodiment of the present application, the shrinkage processing is mainly used to eliminate blank frames in the speech features and merge repeated frames in the speech features, reduce the interference of blank frames and repeated frames, and facilitate the semantic feature encoder to extract high-level semantic features, thereby improving the encoding performance of the encoder, and further improving the prediction performance of the entire end-to-end speech translation model.

[0091] In some optional implementations of the embodiments of the present application, the above step S203 specifically includes the following steps:

[0092] Extracting time series information of speech feature data based on a bidirectional long short-term memory network;

[0093] Analyze the relationship between each element in the speech feature data and all other elements according to the self-attention mechanism and time series information to obtain self-attention feature data;

[0094] Perform weighted fusion on the self-attention feature data to obtain weighted fused self-attention feature data;

[0095] The weighted fused self-attention feature data is nonlinearly transformed according to the multi-layer neural network to obtain semantic feature data.

[0096] In an embodiment of the present application, a bidirectional long short-term memory network (Bi-LSTM) is used to effectively extract time series information from these acoustic features. Bi-LSTM can capture the dependencies of speech data before and after time through its forward and backward memory units, respectively, and enhance the model's ability to understand context. Specifically, for each time step, Bi-LSTM may calculate updated values ​​of the hidden state, which depend on the hidden state of the previous time step and the feature vector of the current input.

[0097] In an embodiment of the present application, a self-attention mechanism is combined, which enables the model to focus on the most important part of the speech sequence by calculating the weight distribution within the feature vector, wherein the self-attention mechanism generally includes a scaled dot product attention layer, which captures long-distance speech data dependencies by weighted summing the query, key, and value vectors at each sequence position. As an example, by calculating the similarity scores between different time steps, the model can focus more on other words that are semantically related to the currently analyzed words.

[0098] In an embodiment of the present application, by performing weighted fusion of semantic features between different languages ​​on the feature vector processed by the self-attention mechanism, the ability of the model to process multilingual data is further improved.

[0099] In an embodiment of the present application, the fused semantic feature vector is nonlinearly transformed through a multi-layer fully connected neural network and mapped to a higher-dimensional semantic space. The deep structure of the neural network can not only further purify the semantic representation, but also increase the nonlinear processing capability of the model through activation functions such as ReLU or tanh, thereby capturing more complex semantic feature data.

[0100] In some optional implementations of the embodiments of the present application, the following steps are further included after step S203:

[0101] The semantic feature data is clustered according to the clustering algorithm.

[0102] In the embodiments of the present application, the clustering algorithm may be K-means, hierarchical clustering, DBSCAN, spectral clustering, etc. It should be understood that the examples of clustering algorithms given here are only for the convenience of understanding and are not used to limit the present application.

[0103] In the embodiment of the present application, by performing clustering processing on the semantic feature data, the semantic information of the semantic feature data can be better represented.

[0104] In some optional implementations of the embodiments of the present application, the following steps are included after step S205:

[0105] Correcting the text sequence data according to the text error correction algorithm to obtain an initial corrected text;

[0106] Calculate the perplexity of the initial revised text according to the statistical language model, and re-recognize the initial revised text whose perplexity is higher than the preset perplexity threshold to obtain the intermediate revised text;

[0107] Optimize the boundary of the intermediate revised text according to the context analysis method to obtain revised text sequence data;

[0108] The above step S206 specifically includes the following steps:

[0109] The corrected text sequence data is split to obtain the target translation text.

[0110] In the embodiment of the present application, after obtaining the text sequence data, it is necessary to check and correct the text sequence data, specifically:

[0111] First, the text is spelled corrected using a rule-based text correction algorithm;

[0112] Next, the corrected text will be input into a statistical language model, which uses n-gram or long short-term memory network (LSTM) to calculate the perplexity of the text. For text fragments with high perplexity, the system will use advanced language recognition technology, such as Transformer-based models, to perform deep semantic analysis to ensure the correctness and fluency of its language expression.

[0113] After that, the system performs contextual analysis on all texts and uses techniques such as dependency syntactic analysis to optimize the division of sentence boundaries. By analyzing the syntactic tree structure, the system can effectively identify sentence components, adjust the start and end positions of sentences, and improve the logic and coherence of the text.

[0114] In some optional implementations of the embodiments of the present application, the entire text sequence may also be quality checked to check its sentence coherence, grammatical correctness, and spelling accuracy.

[0115] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0116] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0118] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0119] Embodiment 2

[0120] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of an audio translation device based on an end-to-end speech translation model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0121] like Figure 3 As shown, the audio translation device 200 based on the end-to-end speech translation model of the embodiment of the present application includes:

[0122] The request receiving module 210 is used to receive an audio translation request carrying the audio to be processed sent by a user terminal;

[0123] The speech coding module 220 is used to perform speech feature coding operation on the audio to be processed according to the speech feature encoder to obtain speech feature data;

[0124] A semantic encoding module 230, configured to perform a semantic feature encoding operation on the speech feature data according to a semantic feature encoder to obtain semantic feature data;

[0125] A source language decoding module 240 is used to perform source language decoding operation on the semantic feature data according to a source language decoder to obtain a source language text;

[0126] A text sequence decoding module 250 is used to decode the semantic feature data according to the text sequence decoder and the source language text to obtain text sequence data;

[0127] The splitting and obtaining module 260 is used to perform a splitting operation on the text sequence data to obtain a target translation text;

[0128] The output module 270 is used to output the target translation text to the user terminal.

[0129] In an embodiment of the present application, an audio translation device 200 based on an end-to-end speech translation model is provided, comprising: a request receiving module 210, for receiving an audio translation request carrying an audio to be processed sent by a user terminal; a speech encoding module 220, for performing a speech feature encoding operation on the audio to be processed according to a speech feature encoder to obtain speech feature data; a semantic encoding module 230, for performing a semantic feature encoding operation on the speech feature data according to a semantic feature encoder to obtain semantic feature data; a source language decoding module 240, for performing a source language decoding operation on the semantic feature data according to a source language decoder to obtain a source language text; a text sequence decoding module 250, for performing a decoding operation on the semantic feature data according to a text sequence decoder and a source language text to obtain text sequence data; a split acquisition module 260, for performing a split operation on the text sequence data to obtain a target translation text; an output module 270, for outputting the target translation text to the user terminal. Compared with the prior art, the present application effectively realizes a high-accuracy translation from audio to text through the close cooperation of each encoder and decoder, ensuring the accuracy of the translation content and the appropriate expression of the context. In addition, the system is able to handle multiple languages, enhancing its applicability and flexibility.

[0130] In some optional implementations of the embodiments of the present application, the audio translation device 200 based on the end-to-end speech translation model further includes:

[0131] The contraction module is used to perform contraction processing on the speech feature data.

[0132] In some optional implementations of the embodiments of the present application, the semantic encoding module 230 includes:

[0133] A time series extraction submodule, used to extract time series information of speech feature data based on a bidirectional long short-term memory network;

[0134] A self-attention feature acquisition submodule is used to analyze the relationship between each element in the speech feature data and all other elements according to the self-attention mechanism and time series information to obtain self-attention feature data;

[0135] The weighted fusion submodule is used to perform weighted fusion on the self-attention feature data to obtain weighted fused self-attention feature data;

[0136] The nonlinear transformation submodule is used to perform nonlinear transformation on the weighted fused self-attention feature data according to the multi-layer neural network to obtain semantic feature data.

[0137] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0138] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.

[0139] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0140] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store an operating system and various application software installed on the computer device 300, such as computer-readable instructions of an audio translation method based on an end-to-end speech translation model, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0141] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for running the audio translation method based on the end-to-end speech translation model.

[0142] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0143] The computer device provided by the present application effectively realizes high-accuracy translation from audio to text through the close cooperation of various encoders and decoders, ensuring the accuracy of the translated content and the appropriate expression of the context. In addition, the system can also handle multiple languages, enhancing its applicability and flexibility.

[0144] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the audio translation method based on the end-to-end speech translation model as described above.

[0145] The computer-readable storage medium provided by the present application effectively realizes high-accuracy translation from audio to text through the close cooperation of various encoders and decoders, ensuring the accuracy of the translated content and the appropriate expression of the context. In addition, the system can also handle multiple languages, enhancing its applicability and flexibility.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0147] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.< / st>

Claims

1. An audio translation method based on an end-to-end speech translation model, characterized in that: The method comprises the following steps: Receiving an audio translation request carrying audio to be processed sent by a user terminal; Performing a speech feature encoding operation on the audio to be processed according to a speech feature encoder to obtain speech feature data; Performing a semantic feature encoding operation on the speech feature data according to a semantic feature encoder to obtain semantic feature data; Performing source language decoding operation on the semantic feature data according to the source language decoder to obtain source language text; Decoding the semantic feature data according to the text sequence decoder and the source language text to obtain text sequence data; Performing a splitting operation on the text sequence data to obtain a target translation text; The target translation text is output to the user terminal.

2. The audio translation method based on the end-to-end speech translation model according to claim 1, characterized in that: After the step of performing a speech feature encoding operation on the audio to be processed according to the speech feature encoder to obtain speech feature data, the following steps are also included: The speech feature data is subjected to contraction processing.

3. The audio translation method based on the end-to-end speech translation model according to claim 1 is characterized in that: The step of performing a semantic feature encoding operation on the speech feature data according to the semantic feature encoder to obtain the semantic feature data specifically comprises the following steps: Extracting time series information of the speech feature data according to a bidirectional long short-term memory network; Analyze the relationship between each element in the speech feature data and all other elements according to the self-attention mechanism and the time series information to obtain self-attention feature data; Performing weighted fusion on the self-attention feature data to obtain weighted fused self-attention feature data; The weighted fused self-attention feature data is nonlinearly transformed according to a multi-layer neural network to obtain the semantic feature data.

4. The audio translation method based on the end-to-end speech translation model according to claim 1 is characterized in that: After the step of performing a semantic feature encoding operation on the speech feature data according to the semantic feature encoder to obtain the semantic feature data, the following step is also included: The semantic feature data is clustered according to a clustering algorithm.

5. The audio translation method based on the end-to-end speech translation model according to claim 1 is characterized in that: The step of decoding the semantic feature data according to the text sequence decoder and the source language text to obtain the text sequence data specifically comprises the following steps: Correcting the text sequence data according to a text error correction algorithm to obtain an initial corrected text; Calculating the perplexity of the initial revised text according to the statistical language model, and re-recognizing the initial revised text whose perplexity is higher than a preset perplexity threshold to obtain an intermediate revised text; Optimizing the boundary of the intermediate revised text according to the context analysis method to obtain revised text sequence data; The step of performing a splitting operation on the text sequence data to obtain a target translation text specifically comprises the following steps: The modified text sequence data is split to obtain the target translation text.

6. An audio translation device based on an end-to-end speech translation model, characterized in that: include: A request receiving module, used for receiving an audio translation request carrying audio to be processed sent by a user terminal; A speech encoding module, used for performing a speech feature encoding operation on the audio to be processed according to a speech feature encoder to obtain speech feature data; A semantic encoding module, used for performing a semantic feature encoding operation on the speech feature data according to a semantic feature encoder to obtain semantic feature data; A source language decoding module, used for performing a source language decoding operation on the semantic feature data according to a source language decoder to obtain a source language text; A text sequence decoding module, used for decoding the semantic feature data according to a text sequence decoder and the source language text to obtain text sequence data; A splitting and obtaining module, used for performing a splitting operation on the text sequence data to obtain a target translation text; An output module is used to output the target translation text to the user terminal.

7. The audio translation device based on the end-to-end speech translation model according to claim 6 is characterized in that: The device also includes: The contraction module is used to perform contraction processing on the speech feature data.

8. The audio translation device based on the end-to-end speech translation model according to claim 6, characterized in that: The semantic encoding module comprises: A time series extraction submodule, used for extracting time series information of the speech feature data according to a bidirectional long short-term memory network; A self-attention feature acquisition submodule, used for analyzing the relationship between each element in the speech feature data and all other elements according to the self-attention mechanism and the time series information to obtain self-attention feature data; A weighted fusion submodule, used for performing weighted fusion on the self-attention feature data to obtain weighted fused self-attention feature data; The nonlinear transformation submodule is used to perform nonlinear transformation on the weighted fused self-attention feature data according to a multi-layer neural network to obtain the semantic feature data.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the audio translation method based on the end-to-end speech translation model as described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the audio translation method based on the end-to-end speech translation model as described in any one of claims 1 to 5 are implemented.

Images