Multilingual automatic speech recognition method and device, computer device and storage medium

Abstract

The embodiment of the application belongs to the field of artificial intelligence, and relates to a multilingual automatic speech recognition method, which comprises processing input speech to obtain a speech embedding vector sequence; extracting features of the speech embedding vector sequence based on an encoder of a pre-trained ByT5 model to obtain bit encoding of the speech embedding vector sequence; inputting the bit encoding of the speech embedding vector sequence to a decoder of the ByT5 model according to a preset input data structure, and outputting bit encoding of target language characters according to a preset output data structure after decoding by the decoder of the ByT5 model; and inputting the bit encoding of the target language characters output by the decoder of the ByT5 model into a sequence as input for next decoding to generate a target text sequence. The application also provides a multilingual automatic speech recognition device, a computer device and a storage medium. The application successfully realizes cross-modal conversion of information from audio to text.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to multilingual automatic speech recognition methods, devices, computer equipment, and storage media. Background Technology

[0002] Automatic Speech Recognition (ASR) is a technology that converts speech signals into text. It is widely used in various fields, including but not limited to voice assistants, voice search, voice translation, and voice commands. ASR technology greatly enhances the natural interaction between humans and computers, thus significantly optimizing the user experience of business voice assistants. However, current ASR technologies often exhibit inconsistent performance in handling multiple languages ​​or accents in real-world applications, thus typically requiring training for a single language to obtain a practical ASR system. Furthermore, the text generation process in ASR generally uses a token-based language model decoder. Token-based methods require encoding and decoding based on a dictionary, thus being limited by the dictionary and unable to handle words not present in it. Simultaneously, in multilingual scenarios, the required dictionary size is enormous, leading to a significant increase in model size and computational cost, and also introducing label noise. Therefore, it is indeed necessary to develop a multilingual automatic speech recognition method. Summary of the Invention

[0003] The purpose of this application is to provide a multilingual automatic speech recognition method, apparatus, computer device, and storage medium to solve the problem of token-level models being limited by dictionaries.

[0004] To address the aforementioned technical problems, this application provides a multilingual automatic speech recognition method, employing the following technical solution:

[0005] A multilingual automatic speech recognition method includes the following steps:

[0006] The input speech is processed to obtain a speech embedding vector sequence;

[0007] The encoder based on the pre-trained ByT5 model extracts features from the speech embedding vector sequence to obtain the bit encoding of the speech embedding vector sequence.

[0008] The bit encoding of the speech embedding vector sequence is input to the decoder of the ByT5 model according to a preset input data structure. After decoding, the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure.

[0009] The bit-encoded sequence of the target language text output by the decoder of the ByT5 model is used as the input for the next decoding to generate the target text sequence.

[0010] Furthermore, the step of processing the input speech to obtain a speech embedding vector sequence includes:

[0011] Convert the input speech into the corresponding Mel spectrum;

[0012] Features are extracted from the Mel spectrum using one-dimensional convolution and the GELU activation function, and the extracted features are added to the sinusoidal positional code to obtain the speech embedding vector sequence.

[0013] Furthermore, the preset input data structure is specifically as follows:

[0014]

[0015] Where SOB represents the start point of the sequence, ORI-LAN represents the language of the input speech, TAR-LAN represents the language of the target generated text, ST represents the start time point of the speech segment, Bytes Seq represents the bit encoding of the single word, and ET represents the end time point of the speech segment.

[0016] The steps of the ByT5 model's decoder outputting the bit encoding of the target language text according to a preset output data structure specifically include:

[0017] Enter the SOB flag, ORI-LAN flag, and TAR-LAN flag;

[0018] The decoder of the ByT5 model predicts the start time ST of the first segment of speech.

[0019] The sequence of concatenated SOB flag, ORI-LAN flag, TAR-LAN flag and ST flag is input into the decoder of the ByT5 model to obtain the bit encoding of the target language text corresponding to the first speech text.

[0020] Furthermore, the specific data structure of the bit encoding of the target language text corresponding to the first segment of speech is as follows:

[0021]

[0022] Wherein, ORI-LAN represents the language of the input speech, TAR-LAN represents the language of the target generated text, ST represents the start time of the speech segment, Bytes-Seq represents the bit encoding of the target language text, and ET represents the end time of the speech segment.

[0023] The step of using the bit-encoded sequence of the target language text output by the decoder of the ByT5 model as the input for the next decoding to generate the target text sequence specifically includes:

[0024] The bit encoding of the target language text corresponding to the first speech text is concatenated with the speech embedding vector sequence and then input into the decoder of the ByT5 model;

[0025] The decoder of the ByT5 model obtains the bit encoding of the target language text corresponding to the second segment of speech text after decoding;

[0026] After each decoding, the bit encodings of the target language text corresponding to the speech text are concatenated and then decoded by the decoder of the ByT5 model until the end time point of the speech segment is triggered, thus obtaining the generated target text sequence of the input speech.

[0027] Furthermore, after the step of using the bit-encoded sequence of the target language text output by the decoder of the ByT5 model as the input for the next decoding to generate the target text sequence, the method further includes:

[0028] The target text sequence is subjected to grammar checks using a preset grammar library;

[0029] If a syntax error is detected, a message will be displayed indicating that the target text sequence contains a syntax error.

[0030] The target text sequence containing errors is corrected using a syntax library.

[0031] Furthermore, the step of concatenating the bit encoding of the target language text corresponding to the first speech text with the speech embedding vector sequence and then inputting it into the decoder of the ByT5 model includes:

[0032] The ET flag in the bit encoding of the target language text corresponding to the first speech text is used as the ST flag in the second speech text. The sequence concatenated with the ST flag, SOB flag, ORI-LAN flag, and TAR-LAN flag is then input into the decoder of the ByT5 model.

[0033] Furthermore, the training steps of the pre-trained ByT5 model include:

[0034] Obtain a multilingual speech-text dataset and divide the multilingual speech-text dataset into a certain proportion of training set and test set;

[0035] Select the ByT5 model, which includes a ByT5 encoder and a ByT5 decoder;

[0036] The ByT5 model is trained using a certain proportion of the training set until the ByT5 model meets the set criteria on the test set.

[0037] To address the aforementioned technical problems, this application also provides a multilingual automatic speech recognition device, which employs the following technical solution:

[0038] A multilingual automatic speech recognition device, comprising:

[0039] The processing module is used to process the input speech to obtain a speech embedding vector sequence;

[0040] The extraction module is used to extract features of the speech embedding vector sequence from the encoder based on the pre-trained ByT5 model, and obtain the bit encoding of the speech embedding vector sequence.

[0041] The encoding and decoding module is used to input the bit encoding of the speech embedding vector sequence into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure after decoding.

[0042] The generation module is used to take the bit-encoded sequence of the target language text output by the decoder of the ByT5 model as the input for the next decoding to generate the target text.

[0043] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0044] A computer device includes a memory and a processor, the memory storing computer-readable instructions, the processor executing the computer-readable instructions to implement the steps of the multilingual automatic speech recognition method as described above.

[0045] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0046] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the multilingual automatic speech recognition method described above.

[0047] Compared with the prior art, the embodiments of this application have the following advantages: This application processes the input speech to obtain a speech embedding vector sequence, and then extracts the features of the speech embedding vector sequence based on the encoder of the pre-trained ByT5 model to obtain the bit encoding of the speech embedding vector sequence; then the bit encoding of the speech embedding vector sequence is input to the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure; finally, the bit encoding of the target language text output by the decoder of the ByT5 model is used as the input for the next decoding to generate the target text sequence. This not only has stronger multilingual processing capabilities and effectively improves the robustness of the model, but also the bit encoding data representation method greatly reduces the model size and reduces the computational complexity of the model, successfully realizing cross-modal conversion of information from audio to text. Attached Figure Description

[0048] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0050] Figure 2 A flowchart of an embodiment of the multilingual automatic speech recognition method according to this application;

[0051] Figure 3 This is a schematic diagram of the encoder and decoder of the ByT5 model in the embodiments of this application.

[0052] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S203;

[0053] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S204;

[0054] Figure 6 This is a schematic diagram of a structure of an embodiment of the multilingual automatic speech recognition device according to this application;

[0055] Figure 7 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0057] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0058] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0059] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0060] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0061] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Eperts Group Audio Layer III), MP4 players (Moving Picture Eperts Group Audio Layer IV), laptops, and desktop computers, etc.

[0062] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0063] It should be noted that the multilingual automatic speech recognition method provided in this application embodiment is generally executed by a server, and correspondingly, the multilingual automatic speech recognition device is generally installed in a terminal device.

[0064] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0065] Continue to refer to Figure 2 A flowchart of an embodiment of a multilingual automatic speech recognition method according to this application is shown. The multilingual automatic speech recognition method includes the following steps:

[0066] Step S201: Process the input speech to obtain a speech embedding vector sequence.

[0067] In this embodiment, the steps of processing the input speech to obtain a speech embedding vector sequence include:

[0068] First, the input speech is converted into the corresponding Mel spectrum; then, one-dimensional convolution and GELU activation function are used to extract features from the Mel spectrum, and the extracted features are added to the sinusoidal position code to obtain the speech embedding vector sequence.

[0069] The steps for converting input speech into the corresponding Mel spectrum include: dividing the speech signal into short time windows, typically 20 to 30 milliseconds in length, and applying a window function Q within each window to reduce spectral leakage; for each window, calculating FFTQ to obtain the signal's spectrum, where the size of the spectrum is half the number of FFT points; converting the spectral values ​​into frequencies on the Mel frequency scale, which can be achieved by applying a Mel filter bank; taking the logarithm of the Mel filter output to obtain Mel cepstral coefficients (MFCs); and for a specified time interval, extracting the Mel cepstral coefficients within the corresponding window to obtain the corresponding Mel spectrum.

[0070] Step S202: The encoder based on the pre-trained ByT5 model extracts features of the speech embedding vector sequence to obtain the bit encoding of the speech embedding vector sequence.

[0071] In this embodiment, since text is stored in bytes in a computer, any word or phrase can be represented by bytes or byte sequences. Therefore, the ByT5 architecture, which uses byte encoding to represent words, demonstrates outstanding superiority in solving multilingual and unfamiliar word problems. Furthermore, because the ByT5 model is a byte-level model, only 256 embeddings are needed to represent any word in all languages. Compared to token-level models, this significantly reduces model size and computational cost, and also eliminates the complex modifications required when the model needs to learn a new language.

[0072] In this embodiment, the pre-trained ByT5 model is specifically trained using a multilingual speech-text dataset. The multilingual speech-text dataset can be any existing dataset.

[0073] Specifically, the training steps for the pre-trained ByT5 model include:

[0074] Obtain a multilingual speech-text dataset and divide the multilingual speech-text dataset into a certain proportion of training set and test set;

[0075] Select the ByT5 model, which includes a ByT5 encoder and a ByT5 decoder;

[0076] The ByT5 model is trained using a certain proportion of the training set until the ByT5 model meets the set criteria on the test set.

[0077] In this embodiment, see Figure 3As shown, the ByT5 model's encoder consists of functional blocks comprising a self-attention mechanism, a multilayer perceptron, and residual connections. Multiple such functional blocks connected sequentially form the ByT5 encoder structure. The self-attention mechanism extracts relationships between positions to improve the model's understanding of context and enables it to focus on important features in the input.

[0078] Step S203: The bit encoding of the speech embedding vector sequence is input to the decoder of the ByT5 model according to a preset input data structure. After decoding, the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure.

[0079] In this embodiment, the preset input data structure is specifically as follows:

[0080]

[0081] Where SOB represents the start point of the sequence, ORI-LAN represents the language of the input speech, TAR-LAN represents the language of the target generated text, ST represents the start time point of the speech segment, Bytes Seq represents the bit encoding of the single word, and ET represents the end time point of the speech segment.

[0082] The ORI-LAN and TAR-LAN flags enable controllability of the language of the generated text, while the ST and ET flags can effectively locate the speech start time in the audio, allowing the decoder to more accurately align with the speech during the text generation process.

[0083] See Figure 3 As shown, the decoder of the ByT5 model is also composed of multiple identical functional blocks connected together. These decoder functional blocks consist of a self-attention mechanism, a cross-attention mechanism, a multilayer perceptron, and residual connections. The cross-attention mechanism requires simultaneous input from both the encoder's output and the decoder's input. Unlike the encoder, which processes the input sequence in parallel, the decoder inputs and outputs sequentially from left to right.

[0084] Among them, such as Figure 4 As shown, the steps of the ByT5 model's decoder outputting the bit encoding of the target language text according to a preset output data structure after decoding specifically include:

[0085] Step S2031: Input the SOB flag, ORI-LAN flag, and TAR-LAN flag to indicate that the conversion has started, and specify the source language and target language for the conversion.

[0086] In step S2032, the decoder of the ByT5 model predicts the start time ST of the first speech segment.

[0087] Step S2033: Input the sequence of concatenated SOB flag, ORI-LAN flag, TAR-LAN flag and ST flag into the decoder of the ByT5 model to obtain the bit encoding of the target language text corresponding to the first speech text.

[0088] Specifically, the bit-encoded data structure of the target language text corresponding to the first segment of speech is as follows:

[0089]

[0090] Wherein, ORI-LAN represents the language of the input speech, TAR-LAN represents the language of the target generated text, ST represents the start time of the speech segment, Bytes-Seq represents the bit encoding of the target language text, and ET represents the end time of the speech segment.

[0091] Step S204: The bit-encoded sequence of the target language text output by the decoder of the ByT5 model is used as the input for the next decoding to generate the target text sequence.

[0092] In this embodiment, as Figure 5 As shown, the step of using the bit-encoded sequence of the target language text output by the decoder of the ByT5 model as the input for the next decoding to generate the target text sequence specifically includes:

[0093] Step S2041: The bit encoding of the target language text corresponding to the first speech text is concatenated with the speech embedding vector sequence and then input into the decoder of the ByT5 model.

[0094] In step S2042, the decoder of the ByT5 model decodes the second segment of speech text to obtain the bit encoding of the target language text.

[0095] Step S2043: After each decoding, the bit encoding of the target language text corresponding to the speech text is concatenated and then decoded by the decoder of the ByT5 model until the end time point of the speech segment is triggered, so as to obtain the generated target text sequence of the input speech.

[0096] The step of concatenating the bit encoding of the target language text corresponding to the first speech text with the speech embedding vector sequence and then inputting it into the decoder of the ByT5 model includes: using the ET flag bit in the bit encoding of the target language text corresponding to the first speech text as the ST flag bit of the second speech text, and then inputting the sequence concatenated with the ST flag bit, SOB flag bit, ORI-LAN flag bit, and TAR-LAN flag bit into the decoder of the ByT5 model.

[0097] In this embodiment, after the step of using the bit-encoded sequence of the target language text output by the decoder of the ByT5 model as the input for the next decoding to generate the target text sequence, the method further includes:

[0098] The target text sequence is subjected to grammar checks using a preset grammar library. If a grammar error is detected, a message will be displayed indicating that the target text sequence contains a grammar error. The grammar library will then be used to correct the target text sequence with errors, such as reordering the target text sequence according to the standard grammar of the grammar library.

[0099] The grammar library in this embodiment can be selected from existing technologies. For example, different target grammar libraries can be selected for languages ​​with different translation processes.

[0100] This application processes the input speech to obtain a speech embedding vector sequence, then extracts features from the speech embedding vector sequence based on the encoder of a pre-trained ByT5 model to obtain the bit encoding of the speech embedding vector sequence. The bit encoding of the speech embedding vector sequence is then input into the decoder of the ByT5 model according to a preset input data structure. After decoding, the ByT5 model decoder outputs the bit encoding of the target language text according to a preset output data structure. Finally, the bit encoding of the target language text output by the ByT5 model decoder is used as the next input for decoding to generate the target text sequence. This not only provides stronger multilingual processing capabilities and effectively improves the robustness of the model, but also significantly reduces the model size and computational complexity by using a bit encoding data representation method, successfully achieving cross-modal conversion of information from audio to text.

[0101] In this embodiment, the multilingual automatic speech recognition method operates on an electronic device (e.g., Figure 1 The server shown can be connected via wired or wireless means. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMA multilingual automatic voice recognition connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future wireless connection methods.

[0102] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0103] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0105] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0106] Further reference Figure 6 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a multilingual automatic speech recognition device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0107] like Figure 6 As shown, the multilingual automatic speech recognition device 600 described in this embodiment includes: a processing module 601, an extraction module 602, an encoding / decoding module 603, and a generation module 604. Wherein:

[0108] The processing module 601 is used to process the input speech to obtain a speech embedding vector sequence.

[0109] The extraction module 602 is used to extract features of the speech embedding vector sequence from the encoder based on the pre-trained ByT5 model, and obtain the bit encoding of the speech embedding vector sequence.

[0110] The encoding / decoding module 603 is used to input the bit encoding of the speech embedding vector sequence into the decoder of the ByT5 model according to a preset input data structure. After decoding, the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure.

[0111] The generation module 604 is used to take the bit-encoded sequence of the target language text output by the decoder of the ByT5 model as the input for the next decoding to generate the target text.

[0112] This application processes the input speech using a processing module 601 to obtain a speech embedding vector sequence. Then, an extraction module 602 extracts features from the speech embedding vector sequence based on the encoder of a pre-trained ByT5 model, obtaining the bit encoding of the speech embedding vector sequence. Next, an encoding / decoding module 603 inputs the bit encoding of the speech embedding vector sequence into the ByT5 model's decoder according to a preset input data structure. The ByT5 model's decoder then outputs the bit encoding of the target language text according to a preset output data structure. Finally, a generation module 604 uses the bit encoding of the target language text output by the ByT5 model's decoder to form a sequence, which is then used as the next input for decoding to generate the target text sequence. This not only provides stronger multilingual processing capabilities and effectively improves the model's robustness, but also significantly reduces the model size and computational burden due to the bit encoding data representation method.

[0113] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 7 , Figure 7 This is a basic structural block diagram of the computer device in this embodiment.

[0114] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are interconnected via a system bus. It should be noted that only the computer device 7 with components 61-63 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0115] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0116] The memory 71 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or SD multilingual automatic speech recognition memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 71 may be an internal storage unit of the computer device 7, such as the hard disk or memory of the computer device 7. In other embodiments, the memory 71 may also be an external storage device of the computer device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 7. Of course, the memory 71 may also include both the internal storage unit and its external storage device of the computer device 7. In this embodiment, the memory 71 is typically used to store the operating system and various application software installed on the computer device 7, such as computer-readable instructions for multilingual automatic speech recognition methods. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or will be output.

[0117] In some embodiments, the processor 72 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 72 is typically used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to execute computer-readable instructions stored in the memory 71 or to process data, such as executing computer-readable instructions for the multilingual automatic speech recognition method.

[0118] The network interface 73 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 7 and other electronic devices.

[0119] This application processes the input speech to obtain a speech embedding vector sequence, then extracts features from the speech embedding vector sequence based on the encoder of a pre-trained ByT5 model to obtain the bit encoding of the speech embedding vector sequence. The bit encoding of the speech embedding vector sequence is then input into the decoder of the ByT5 model according to a preset input data structure. After decoding, the ByT5 model decoder outputs the bit encoding of the target language text according to a preset output data structure. Finally, the bit encoding of the target language text output by the ByT5 model decoder is used as the next input for decoding to generate the target text sequence. This not only provides stronger multilingual processing capabilities and effectively improves the robustness of the model, but also significantly reduces the model size and computational complexity by using a bit encoding data representation method, successfully achieving cross-modal conversion of information from audio to text.

[0120] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the multilingual automatic speech recognition method described above.

[0121] This application processes the input speech to obtain a speech embedding vector sequence, then extracts features from the speech embedding vector sequence based on the encoder of a pre-trained ByT5 model to obtain the bit encoding of the speech embedding vector sequence. The bit encoding of the speech embedding vector sequence is then input into the decoder of the ByT5 model according to a preset input data structure. After decoding, the ByT5 model decoder outputs the bit encoding of the target language text according to a preset output data structure. Finally, the bit encoding of the target language text output by the ByT5 model decoder is used as the next input for decoding to generate the target text sequence. This not only provides stronger multilingual processing capabilities and effectively improves the robustness of the model, but also significantly reduces the model size and computational complexity by using a bit encoding data representation method, successfully achieving cross-modal conversion of information from audio to text.

[0122] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0124] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A multilingual automatic speech recognition method, characterized in that, Includes the following steps: The input speech is processed to obtain a speech embedding vector sequence; The encoder based on the pre-trained ByT5 model extracts features from the speech embedding vector sequence to obtain the bit encoding of the speech embedding vector sequence. The bit encoding of the speech embedding vector sequence is input to the decoder of the ByT5 model according to a preset input data structure. After decoding, the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure. The bit-encoded sequence of the target language text output by the decoder of the ByT5 model is used as the input for the next decoding to generate the target text sequence. The preset input data structure is specifically as follows: Where SOB represents the start point of the sequence, ORI-LAN represents the language of the input speech, TAR-LAN represents the language of the target generated text, ST represents the start time point of the speech segment, Bytes Seq represents the bit encoding of the single word, and ET represents the end time point of the speech segment. The steps of the ByT5 model's decoder outputting the bit encoding of the target language text according to a preset output data structure specifically include: Enter the SOB flag, ORI-LAN flag, and TAR-LAN flag; The decoder of the ByT5 model predicts the start time of the first segment of speech. The sequence of concatenated SOB flag, ORI-LAN flag, TAR-LAN flag and ST flag is input into the decoder of the ByT5 model to obtain the bit encoding of the target language text corresponding to the first speech text. The specific bit-encoded data structure of the target language text corresponding to the first segment of speech is as follows: Wherein, ORI-LAN represents the language of the input speech, TAR-LAN represents the language of the target generated text, ST represents the start time of the speech segment, Bytes-Seq represents the bit encoding of the target language text, and ET represents the end time of the speech segment. The step of using the bit-encoded sequence of the target language text output by the decoder of the ByT5 model as the input for the next decoding to generate the target text sequence specifically includes: The bit encoding of the target language text corresponding to the first speech text is concatenated with the speech embedding vector sequence and then input into the decoder of the ByT5 model; The decoder of the ByT5 model obtains the bit encoding of the target language text corresponding to the second segment of speech text after decoding; After each decoding, the bit encodings of the target language text corresponding to the speech text are concatenated and then decoded by the decoder of the ByT5 model until the end time point of the speech segment is triggered, thus obtaining the generated target text sequence of the input speech.

2. The multilingual automatic speech recognition method according to claim 1, characterized in that, The step of processing the input speech to obtain the speech embedding vector sequence includes: Convert the input speech into the corresponding Mel spectrum; Features are extracted from the Mel spectrum using one-dimensional convolution and the GELU activation function, and the extracted features are added to the sinusoidal positional code to obtain the speech embedding vector sequence.

3. The multilingual automatic speech recognition method according to claim 1, characterized in that, After the step of using the bit-encoded sequence of the target language text output by the decoder of the ByT5 model as the input for the next decoding to generate the target text sequence, the method further includes: The target text sequence is subjected to grammar checks using a preset grammar library; If a syntax error is detected, a message will be displayed indicating that the target text sequence contains a syntax error. The target text sequence containing errors is corrected using a syntax library.

4. The multilingual automatic speech recognition method according to claim 2, characterized in that, The step of concatenating the bit encoding of the target language text corresponding to the first speech text with the speech embedding vector sequence and then inputting it into the decoder of the ByT5 model includes: The ET flag in the bit encoding of the target language text corresponding to the first speech text is used as the ST flag in the second speech text. The sequence concatenated with the ST flag, SOB flag, ORI-LAN flag, and TAR-LAN flag is then input into the decoder of the ByT5 model.

5. The multilingual automatic speech recognition method according to claim 1, characterized in that, The training steps for the pre-trained ByT5 model include: Obtain a multilingual speech-text dataset and divide the multilingual speech-text dataset into a certain proportion of training set and test set; Select the ByT5 model, which includes a ByT5 encoder and a ByT5 decoder; The ByT5 model is trained using a certain proportion of the training set until the ByT5 model meets the set criteria on the test set.

6. A multilingual automatic speech recognition device, characterized in that, include: The processing module is used to process the input speech to obtain a speech embedding vector sequence; The extraction module is used to extract features of the speech embedding vector sequence from the encoder based on the pre-trained ByT5 model, and obtain the bit encoding of the speech embedding vector sequence. The encoding and decoding module is used to input the bit encoding of the speech embedding vector sequence into the decoder of the ByT5 model according to a preset input data structure, and the decoder of the ByT5 model outputs the bit encoding of the target language text according to a preset output data structure after decoding. The generation module is used to take the bit-encoded sequence of the target language text output by the decoder of the ByT5 model as the input for the next decoding to generate the target text. The preset input data structure is specifically as follows: Where SOB represents the start point of the sequence, ORI-LAN represents the language of the input speech, TAR-LAN represents the language of the target generated text, ST represents the start time point of the speech segment, Bytes Seq represents the bit encoding of the single word, and ET represents the end time point of the speech segment. The encoding / decoding module is specifically used for: Enter the SOB flag, ORI-LAN flag, and TAR-LAN flag; The decoder of the ByT5 model predicts the start time of the first segment of speech. The sequence of concatenated SOB flag, ORI-LAN flag, TAR-LAN flag and ST flag is input into the decoder of the ByT5 model to obtain the bit encoding of the target language text corresponding to the first speech text. The specific bit-encoded data structure of the target language text corresponding to the first segment of speech is as follows: Wherein, ORI-LAN represents the language of the input speech, TAR-LAN represents the language of the target generated text, ST represents the start time of the speech segment, Bytes-Seq represents the bit encoding of the target language text, and ET represents the end time of the speech segment. The generation module is specifically used for: The bit encoding of the target language text corresponding to the first speech text is concatenated with the speech embedding vector sequence and then input into the decoder of the ByT5 model; The decoder of the ByT5 model obtains the bit encoding of the target language text corresponding to the second segment of speech text after decoding; After each decoding, the bit encodings of the target language text corresponding to the speech text are concatenated and then decoded by the decoder of the ByT5 model until the end time point of the speech segment is triggered, thus obtaining the generated target text sequence of the input speech.

7. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the multilingual automatic speech recognition method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the multilingual automatic speech recognition method as described in any one of claims 1 to 5.

Images