Multi-language automatic speech recognition method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of artificial intelligence, and relates to a multilingual automatic speech recognition method, which comprises the following steps: processing input speech to obtain a speech embedding vector sequence; an encoder based on a pre-trained ByT5 model extracts features of the voice embedded vector sequence to obtain bit codes of the voice embedded vector sequence; inputting the bit code of the voice embedded vector sequence into a decoder of a ByT5 model according to a preset input data structure, and outputting the bit code of the target language character according to a preset output data structure after decoding by the decoder of the ByT5 model; the bit codes of the target language characters output by the decoder of the ByT5 model form a sequence to serve as the next input to be decoded, and a target text sequence is generated. The invention also provides a multilingual automatic speech recognition device, computer equipment and a storage medium. According to the invention, the cross-modal conversion of the information from the audio to the text is successfully realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multilingual automatic speech recognition method, apparatus, computer equipment and storage medium. Background Art

[0002] ASR (Automatic Speech Recognition) is a technology that converts speech signals into text. It is widely used in many fields, including but not limited to voice assistants, voice search, voice translation, voice commands, etc. ASR technology greatly enhances the natural interaction between people and computers, so it can greatly optimize the user experience of business voice assistants. However, the current ASR technology often shows inconsistent processing capabilities when facing multiple languages ​​or accents in actual application scenarios, so it is usually necessary to train for a single language to obtain a practical ASR system. In addition, the ASR text generation process generally uses a token-based language model decoder. The token-based method needs to be encoded and decoded according to the dictionary, so it is limited by the dictionary and cannot process words that do not appear in the dictionary. At the same time, in the multilingual scenario, the required dictionary size is very large, which will lead to a significant increase in the model volume and computational complexity, and will also introduce a certain amount of label noise. For this reason, it is indeed necessary to develop a multilingual automatic speech recognition method. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a multilingual automatic speech recognition method, apparatus, computer device and storage medium to solve the problem that the token-level model is limited by the dictionary.

[0004] In order to solve the above technical problems, the present application embodiment provides a multilingual automatic speech recognition method, which adopts the following technical solutions:

[0005] A multilingual automatic speech recognition method comprises the following steps:

[0006] Process the input speech to obtain a speech embedding vector sequence;

[0007] The encoder based on the pre-trained ByT5 model extracts the features of the speech embedding vector sequence and obtains the bit encoding of the speech embedding vector sequence;

[0008] Inputting the bit encoding of the speech embedding vector sequence into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure after decoding;

[0009] The bit encoding sequence of the target language text output by the decoder of the ByT5 model is used as the next input for decoding to generate a target text sequence.

[0010] Furthermore, the step of processing the input speech to obtain a speech embedding vector sequence includes:

[0011] Convert the input speech into the corresponding Mel spectrum;

[0012] One-dimensional convolution and GELU activation function are used to extract features from the Mel spectrum, and the extracted features are added to the sinusoidal position code to obtain the speech embedding vector sequence.

[0013] Furthermore, the preset input data structure is specifically:

[0014] SOB ORI-LAN TAR-LAN ST Bytes Seq ET

[0015] Among them, SOB represents the sequence start point, ORI-LAN represents the language corresponding to the input speech, TAR-LAN represents the language of the target generated text, ST represents the starting time point of the speech segment, Bytes Seq represents the bit code corresponding to the single word, and ET represents the end time point of the speech segment;

[0016] The step of outputting the bit encoding of the target language text according to the preset output data structure after decoding by the decoder of the ByT5 model specifically includes:

[0017] Enter the SOB flag, ORI-LAN flag, and TAR-LAN flag;

[0018] The decoder of the ByT5 model predicts the starting time ST of the first speech segment;

[0019] The sequence of the SOB flag, ORI-LAN flag, TAR-LAN flag and ST flag is input into the decoder of the ByT5 model to obtain the bit encoding of the first segment of speech text corresponding to the target language text.

[0020] Furthermore, the data structure of the bit encoding of the first segment of speech text corresponding to the target language text is specifically:

[0021] ORI-LAN TAR-LAN ST Bytes-Seq ET

[0022] Among them, ORI-LAN represents the language corresponding to the input speech, TAR-LAN represents the language of the target generated text, ST represents the starting time point of the speech segment, Bytes-Seq represents the bit encoding of the target language text, and ET represents the ending time point of the speech segment;

[0023] The step of using the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input for decoding to generate the target text sequence specifically includes:

[0024] The bit encoding of the first speech text corresponding to the target language text is concatenated with the speech embedding vector sequence and input into the decoder of the ByT5 model;

[0025] The decoder of the ByT5 model obtains the bit encoding of the second speech text corresponding to the target language text after decoding;

[0026] After each decoding, the bit codes of the speech text corresponding to the target language text are concatenated and then decoded by the decoder of the ByT5 model until the end time point of the triggered speech segment is reached, thereby obtaining the generated target text sequence of the input speech.

[0027] Furthermore, after the step of using the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input for decoding to generate the target text sequence, the step further includes:

[0028] Performing grammar checking on the target text sequence using a preset grammar library;

[0029] If a grammatical error is detected, it will be prompted that the target text sequence has a grammatical error;

[0030] The grammar library is used to correct the target text sequence with errors.

[0031] Furthermore, the step of concatenating the bit encoding of the first segment of speech text corresponding to the target language text with the speech embedding vector sequence and inputting the concatenated bit encoding into the decoder of the ByT5 model comprises:

[0032] The ET flag in the bit encoding of the target language text corresponding to the first segment of speech text is used as the ST flag of the second segment of speech text, and then the sequence of the ST flag, SOB flag, ORI-LAN flag and TAR-LAN flag is input into the decoder of the ByT5 model.

[0033] Furthermore, the training steps of the pre-trained ByT5 model include:

[0034] Acquire a multilingual speech-text dataset, and divide the multilingual speech-text dataset into a training set and a test set of a certain proportion;

[0035] Select the ByT5 model, which includes a ByT5 encoder and a ByT5 decoder;

[0036] A certain proportion of the training set is used to train the ByT5 model until the test set tests that the ByT5 model meets the set standards.

[0037] In order to solve the above technical problems, the present application also provides a multilingual automatic speech recognition device, which adopts the following technical solution:

[0038] A multilingual automatic speech recognition device, comprising:

[0039] A processing module, used to process the input speech to obtain a speech embedding vector sequence;

[0040] An extraction module, used to extract features of a speech embedding vector sequence based on an encoder of a pre-trained ByT5 model, and obtain a bit encoding of the speech embedding vector sequence;

[0041] A coding and decoding module, used for inputting the bit coding of the speech embedding vector sequence into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit coding of the target language text according to a preset output data structure after decoding;

[0042] A generation module is used to decode the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input to generate the target text.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0044] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned multilingual automatic speech recognition method when executing the computer-readable instructions.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0046] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the multilingual automatic speech recognition method as described above.

[0047] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: the present application processes the input speech to obtain a speech embedding vector sequence, and then extracts the features of the speech embedding vector sequence based on the encoder of the pre-trained ByT5 model to obtain the bit encoding of the speech embedding vector sequence; then the bit encoding of the speech embedding vector sequence is input into the decoder of the ByT5 model according to the preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language and text according to the preset output data structure after decoding; finally, the bit encoding sequence of the target language and text output by the decoder of the ByT5 model is used as the next input for decoding to generate a target text sequence, which not only has a stronger multi-language processing capability and effectively improves the robustness of the model, but also the bit encoding data representation method greatly reduces the model size, reduces the computational complexity of the model, and successfully realizes the cross-modal conversion of information from audio to text. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0050] Figure 2 A flowchart of an embodiment of a multilingual automatic speech recognition method according to the present application;

[0051] Figure 3 It is a schematic diagram of the encoder and decoder of the ByT5 model in the embodiment of the present application.

[0052] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S203;

[0053] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S204;

[0054] Figure 6 is a schematic structural diagram of an embodiment of a multilingual automatic speech recognition device according to the present application;

[0055] Figure 7 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0057] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0059] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0060] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0061] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Eperts Group Audio Layer III), MP4 (Moving Picture Eperts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0062] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0063] It should be noted that the multilingual automatic speech recognition method provided in the embodiment of the present application is generally executed by a server, and accordingly, the multilingual automatic speech recognition device is generally set in a terminal device.

[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0065] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for multilingual automatic speech recognition according to the present application. The multilingual automatic speech recognition method comprises the following steps:

[0066] Step S201, processing the input speech to obtain a speech embedding vector sequence.

[0067] In this embodiment, the step of processing the input speech to obtain a speech embedding vector sequence includes:

[0068] First, the input speech is converted into the corresponding Mel spectrum; then one-dimensional convolution and GELU activation function are used to extract features from the Mel spectrum, and the extracted features are added to the sinusoidal position code to obtain a speech embedding vector sequence.

[0069] The step of converting the input speech into the corresponding Mel spectrum includes: dividing the speech signal into short time windows, generally using a window with a length of 20 to 30 milliseconds, and performing window function Q windowing in each window to reduce spectrum leakage; for each window, calculating FFTQ to obtain the spectrum of the signal, wherein the size of the spectrum is half the number of FFT points; converting the spectrum value into a frequency on the Mel frequency scale, which can be achieved by applying a Mel filter group; taking the logarithm of the output of the Mel filter to obtain the Mel cepstral coefficient (MFC); for a specified time interval, extracting the Mel cepstral coefficient in the corresponding window to obtain the corresponding Mel spectrum.

[0070] Step S202, extracting features of the speech embedding vector sequence based on the encoder of the pre-trained ByT5 model to obtain the bit encoding of the speech embedding vector sequence.

[0071] In this embodiment, the ByT5 model specifically refers to the document content disclosed in https: / / arxiv.org / abs / 2105.13626. Since text is stored in the form of bytes in a computer, any text and vocabulary phrase can be represented by bytes or byte sequences. Therefore, the ByT5 architecture that uses byte encoding to represent characters has shown outstanding advantages in solving multilingual and new word problems. And because the ByT5 model is a byte-level model, only 256 embeddings are needed to represent any word in all languages. Compared with the token-level model, the model size and calculation amount are greatly reduced, and the complex modification operations required when the model needs to learn a new language are also omitted.

[0072] In this embodiment, the pre-trained ByT5 model specifically uses a multilingual speech-text dataset to train the ByT5 model, and the multilingual speech-text dataset can be selected from existing datasets and the like.

[0073] Specifically, the training steps of the pre-trained ByT5 model include:

[0074] Acquire a multilingual speech-text dataset, and divide the multilingual speech-text dataset into a training set and a test set of a certain proportion;

[0075] Select the ByT5 model, which includes a ByT5 encoder and a ByT5 decoder;

[0076] A certain proportion of the training set is used to train the ByT5 model until the test set tests that the ByT5 model meets the set standards.

[0077] In this embodiment, refer to Figure 3 As shown in the figure, the encoder of the ByT5 model has a functional block composed of a self-attention mechanism, a multi-layer perceptron, and a residual connection. Multiple such functional blocks are connected in sequence to form the encoder structure of ByT5. The self-attention mechanism is used to extract the relationship between positions to improve the model's ability to understand the context and enable the model to focus on important features in the input.

[0078] Step S203, inputting the bit encoding of the speech embedding vector sequence into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure after decoding.

[0079] In this embodiment, the preset input data structure is specifically:

[0080] SOB ORI-LAN TAR-LAN ST Bytes Seq ET

[0081] Among them, SOB represents the starting point of the sequence, ORI-LAN represents the language corresponding to the input speech, TAR-LAN represents the language of the target generated text, ST represents the starting time point of the speech segment, Bytes Seq represents the bit encoding corresponding to the single word, and ET represents the end time point of the speech segment.

[0082] The ORI-LAN and TAR-LAN flags enable controllability of the language of the generated text, while the ST and ET flags can effectively locate the start time of speech in the audio, allowing the decoder to align more accurately with the speech during text generation.

[0083] See also Figure 3 As shown in the figure, the decoder of the ByT5 model is also composed of multiple identical functional blocks connected together. The decoder functional blocks are composed of self-attention mechanism, cross-attention mechanism, multi-layer perceptron and residual connection. The cross-attention mechanism needs to receive the encoded output of the encoder and the encoded input of the decoder at the same time. Unlike the encoder, which processes the input sequence in parallel, the decoder inputs and outputs sequentially from left to right.

[0084] Among them, Figure 4 As shown, the decoder of the ByT5 model outputs the bit encoding of the target language text according to the preset output data structure after decoding, specifically including:

[0085] Step S2031, input the SOB flag, ORI-LAN flag and TAR-LAN flag to indicate the start of conversion, and specify the original language and target language of the conversion.

[0086] Step S2032: The decoder of the ByT5 model predicts the start time ST of the first speech segment.

[0087] Step S2033, input the concatenated sequence of the SOB flag, the ORI-LAN flag, the TAR-LAN flag and the ST flag into the decoder of the ByT5 model to obtain the bit encoding of the first segment of speech text corresponding to the target language text.

[0088] The data structure of the bit encoding of the first segment of speech text corresponding to the target language text is specifically:

[0089] ORI-LAN TAR-LAN ST Bytes-Seq ET

[0090] Among them, ORI-LAN represents the language corresponding to the input speech, TAR-LAN represents the language of the target generated text, ST represents the starting time point of the speech segment, Bytes-Seq represents the bit encoding of the target language text, and ET represents the ending time point of the speech segment.

[0091] Step S204, using the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input for decoding to generate a target text sequence.

[0092] In this embodiment, Figure 5 As shown, the step of using the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input for decoding to generate the target text sequence specifically includes:

[0093] Step S2041, concatenate the bit encoding of the first segment of speech text corresponding to the target language text with the speech embedding vector sequence and input them into the decoder of the ByT5 model.

[0094] Step S2042, the decoder of the ByT5 model obtains the bit code of the second segment of speech text corresponding to the target language text after decoding.

[0095] Step S2043, after concatenating the bit codes of the target language text obtained after each decoding, decode it through the decoder of the ByT5 model until the end time point of the triggered voice segment is reached, and obtain the generated target text sequence of the input voice.

[0096] Among them, the step of splicing the bit code of the first paragraph of phonetic text corresponding to the target language text with the speech embedding vector sequence and inputting them into the decoder of the ByT5 model includes: using the ET flag in the bit code of the first paragraph of phonetic text corresponding to the target language text as the ST flag of the second paragraph of phonetic text, and then inputting the sequence spliced ​​by the ST flag, the SOB flag, the ORI-LAN flag, and the TAR-LAN flag into the decoder of the ByT5 model.

[0097] In this embodiment, after the step of using the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input for decoding to generate the target text sequence, the step further includes:

[0098] The target text sequence is subjected to a grammar check through a preset grammar library. If a grammatical error is detected, it will be prompted that the target text sequence has a grammatical error, and the grammar library will be used to correct the target text sequence with errors, that is, the target text sequence is reordered according to the standard grammar of the grammar library, etc.

[0099] The grammar library in this embodiment can be selected from the existing technology. For example, different grammar libraries with different targets can be selected for different languages ​​in the translation process.

[0100] The present application processes the input speech to obtain a speech embedding vector sequence, and then extracts the features of the speech embedding vector sequence based on the encoder of the pre-trained ByT5 model to obtain the bit encoding of the speech embedding vector sequence; then the bit encoding of the speech embedding vector sequence is input into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure after decoding; finally, the bit encoding sequence of the target language text output by the decoder of the ByT5 model is used as the next input for decoding to generate a target text sequence, which not only has a stronger multi-language processing capability and effectively improves the robustness of the model, but also the bit encoding data representation method greatly reduces the model size, reduces the computational complexity of the model, and successfully realizes the cross-modal conversion of information from audio to text.

[0101] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can be connected by wired connection or wireless connection. It should be noted that the above wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMA multi-language automatic speech recognition connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0102] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0103] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0104] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0105] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0106] Further references Figure 6 , as a response to the above Figure 2 The present application provides an embodiment of a multilingual automatic speech recognition device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0107] like Figure 6 As shown, the multilingual automatic speech recognition device 600 described in this embodiment includes: a processing module 601, an extraction module 602, a coding and decoding module 603 and a generation module 604. Among them:

[0108] The processing module 601 is used to process the input speech to obtain a speech embedding vector sequence.

[0109] The extraction module 602 is used to extract the features of the speech embedding vector sequence based on the encoder of the pre-trained ByT5 model to obtain the bit encoding of the speech embedding vector sequence.

[0110] The encoding and decoding module 603 is used to input the bit encoding of the speech embedding vector sequence into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure after decoding.

[0111] The generation module 604 is used to decode the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input to generate the target text.

[0112] The present application processes the input speech through the processing module 601 to obtain a speech embedding vector sequence, and then extracts the features of the speech embedding vector sequence based on the encoder of the pre-trained ByT5 model through the extraction module 602 to obtain the bit encoding of the speech embedding vector sequence; then the encoding and decoding module 603 is used to input the bit encoding of the speech embedding vector sequence into the decoder of the ByT5 model according to the preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to the preset output data structure after decoding; finally, the generation module 604 is used to decode the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input to generate the target text sequence, which not only has a stronger multi-language processing capability and effectively improves the robustness of the model, but also the bit encoding data representation method greatly reduces the model size and reduces the calculation of the model.

[0113] To solve the above technical problems, the present application also provides a computer device. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0114] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 7 with components 61-63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0115] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0116] The memory 71 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D multilingual automatic speech recognition memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 71 can also be an external storage device of the computer device 7, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 7. Of course, the memory 71 can also include both the internal storage unit of the computer device 7 and its external storage device. In this embodiment, the memory 71 is generally used to store the operating system and various application software installed on the computer device 7, such as computer-readable instructions of the multilingual automatic speech recognition method, etc. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or are to be output.

[0117] The processor 72 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 72 is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to run the computer-readable instructions stored in the memory 71 or process data, such as computer-readable instructions for running the multilingual automatic speech recognition method.

[0118] The network interface 73 may include a wireless network interface or a wired network interface. The network interface 73 is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0119] The present application processes the input speech to obtain a speech embedding vector sequence, and then extracts the features of the speech embedding vector sequence based on the encoder of the pre-trained ByT5 model to obtain the bit encoding of the speech embedding vector sequence; then the bit encoding of the speech embedding vector sequence is input into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure after decoding; finally, the bit encoding sequence of the target language text output by the decoder of the ByT5 model is used as the next input for decoding to generate a target text sequence, which not only has a stronger multi-language processing capability and effectively improves the robustness of the model, but also the bit encoding data representation method greatly reduces the model size, reduces the computational complexity of the model, and successfully realizes the cross-modal conversion of information from audio to text.

[0120] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the multilingual automatic speech recognition method as described above.

[0121] The present application processes the input speech to obtain a speech embedding vector sequence, and then extracts the features of the speech embedding vector sequence based on the encoder of the pre-trained ByT5 model to obtain the bit encoding of the speech embedding vector sequence; then the bit encoding of the speech embedding vector sequence is input into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure after decoding; finally, the bit encoding sequence of the target language text output by the decoder of the ByT5 model is used as the next input for decoding to generate a target text sequence, which not only has a stronger multi-language processing capability and effectively improves the robustness of the model, but also the bit encoding data representation method greatly reduces the model size, reduces the computational complexity of the model, and successfully realizes the cross-modal conversion of information from audio to text.

[0122] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0124] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A multilingual automatic speech recognition method, characterized in that: The steps include: Process the input speech to obtain a speech embedding vector sequence; The encoder based on the pre-trained ByT5 model extracts the features of the speech embedding vector sequence and obtains the bit encoding of the speech embedding vector sequence; Inputting the bit encoding of the speech embedding vector sequence into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure after decoding; The bit encoding sequence of the target language text output by the decoder of the ByT5 model is used as the next input for decoding to generate a target text sequence.

2. The multilingual automatic speech recognition method according to claim 1, characterized in that: The step of processing the input speech to obtain a speech embedding vector sequence comprises: Convert the input speech into the corresponding Mel spectrum; One-dimensional convolution and GELU activation function are used to extract features from the Mel spectrum, and the extracted features are added to the sinusoidal position code to obtain the speech embedding vector sequence.

3. The multilingual automatic speech recognition method according to claim 1, characterized in that: The preset input data structure is specifically: Among them, SOB represents the sequence start point, ORI-LAN represents the language corresponding to the input speech, TAR-LAN represents the language of the target generated text, ST represents the starting time point of the speech segment, Bytes Seq represents the bit code corresponding to the single word, and ET represents the end time point of the speech segment; The step of outputting the bit encoding of the target language text according to the preset output data structure after decoding by the decoder of the ByT5 model specifically includes: Enter the SOB flag, ORI-LAN flag, and TAR-LAN flag; The decoder of the ByT5 model predicts the starting time ST of the first speech segment; The sequence of the SOB flag, ORI-LAN flag, TAR-LAN flag and ST flag is input into the decoder of the ByT5 model to obtain the bit encoding of the first segment of speech text corresponding to the target language text.

4. The multilingual automatic speech recognition method according to claim 3, characterized in that: The data structure of the bit encoding of the first segment of speech text corresponding to the target language text is specifically: Among them, ORI-LAN represents the language corresponding to the input speech, TAR-LAN represents the language of the target generated text, ST represents the starting time point of the speech segment, Bytes-Seq represents the bit encoding of the target language text, and ET represents the ending time point of the speech segment; The step of using the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input for decoding to generate the target text sequence specifically includes: The bit encoding of the first speech text corresponding to the target language text is concatenated with the speech embedding vector sequence and input into the decoder of the ByT5 model; The decoder of the ByT5 model obtains the bit encoding of the second speech text corresponding to the target language text after decoding; After each decoding, the bit codes of the speech text corresponding to the target language text are concatenated and then decoded by the decoder of the ByT5 model until the end time point of the triggered speech segment is reached, thereby obtaining the generated target text sequence of the input speech.

5. The multilingual automatic speech recognition method according to claim 1, characterized in that: After the step of using the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input for decoding to generate the target text sequence, the method further includes: Performing grammar checking on the target text sequence using a preset grammar library; If a grammatical error is detected, it will be prompted that the target text sequence has a grammatical error; The grammar library is used to correct the target text sequence with errors.

6. The multilingual automatic speech recognition method according to claim 4, characterized in that: The step of splicing the bit encoding of the first segment of speech text corresponding to the target language text with the speech embedding vector sequence and inputting the resultant into the decoder of the ByT5 model comprises: The ET flag in the bit encoding of the target language text corresponding to the first segment of speech text is used as the ST flag of the second segment of speech text, and then the sequence of the ST flag, SOB flag, ORI-LAN flag and TAR-LAN flag is input into the decoder of the ByT5 model.

7. The multilingual automatic speech recognition method according to claim 1, characterized in that: The training steps of the pre-trained ByT5 model include: Acquire a multilingual speech-text dataset, and divide the multilingual speech-text dataset into a training set and a test set of a certain proportion; Select the ByT5 model, which includes a ByT5 encoder and a ByT5 decoder; A certain proportion of the training set is used to train the ByT5 model until the test set tests that the ByT5 model meets the set standards.

8. A multilingual automatic speech recognition device, characterized in that: include: A processing module, used to process the input speech to obtain a speech embedding vector sequence; An extraction module, used to extract features of a speech embedding vector sequence based on an encoder of a pre-trained ByT5 model, and obtain a bit encoding of the speech embedding vector sequence; A coding and decoding module, used for inputting the bit coding of the speech embedding vector sequence into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit coding of the target language text according to a preset output data structure after decoding; A generation module is used to decode the bit encoding sequence of the target language text output by the decoder of the ByT5 model as the next input to generate the target text.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the multilingual automatic speech recognition method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the multilingual automatic speech recognition method according to any one of claims 1 to 7 are implemented.

Images