Multilingual automatic speech recognition method and system, storage medium and electronic equipment

Abstract

The invention provides a multilingual automatic speech recognition method and system, a storage medium and electronic equipment. The multilingual automatic speech recognition method comprises the steps of obtaining word table feature vectors of word tables of multiple languages; obtaining language identifier features and identification identifier features; acquiring an acoustic feature matrix of the to-be-recognized voice; obtaining an output feature matrix based on the language identifier features, the acoustic feature matrix and the identification identifier features; obtaining a language feature vector based on the output feature matrix; determining a word table of a language corresponding to the to-be-recognized voice based on the language feature vector and the word table feature vector; and acquiring the identification character and the identification indicator based on the vector corresponding to the identification identifier feature in the output feature matrix, the vector corresponding to the acoustic feature matrix and the word table of the corresponding language until the next identification indicator indicates that the identification is stopped. According to the multi-language automatic speech recognition method and system, the storage medium and the electronic equipment, automatic recognition can be achieved for speech of multiple languages, and the method is rapid, efficient and high in practicability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning, and particularly relates to a multi-language automatic speech recognition method and system, a storage medium and an electronic device. BACKGROUND

[0002] Automatic speech recognition technology (ASR) is a technology for converting human speech into text, and its goal is to convert the lexical content in human speech into computer-readable input, such as key presses, binary encodings, or character sequences. In the prior art, a computer system usually uses various artificial intelligence and machine learning algorithms to convert human speech into text. Currently, ASR recognition is widely used in many fields, including telephone voice assistants, speech-to-text, voice search, etc.

[0003] However, the existing ASR recognition algorithm only performs speech recognition for a specific language at a time. Different models need to be trained for different languages, which leads to cumbersome operation, low efficiency, and high resource consumption. SUMMARY

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a multi-language automatic speech recognition method, system, storage medium and electronic device, which can automatically recognize the speech of multiple languages, is fast and efficient, and has strong practicality.

[0005] In a first aspect, the present application provides a multi-language automatic speech recognition method, comprising the following steps: obtaining a plurality of language word tables and obtaining word table feature vectors of the word tables; setting a language identifier and an identification identifier, and obtaining language identifier features of the language identifier and identification identifier features of the identification identifier; obtaining a speech to be recognized, and obtaining an acoustic feature matrix of the speech to be recognized; constructing an input feature matrix based on the language identifier features, the acoustic feature matrix and the identification identifier features, and inputting the input feature matrix into a large language model to obtain an output feature matrix; obtaining a language feature vector based on the vector corresponding to the language identifier features in the output feature matrix; determining the word table corresponding to the language of the speech to be recognized based on the language feature vector and the word table feature vector; obtaining an identification character and an identification indicator based on the vector corresponding to the identification identifier features in the output feature matrix, the vector corresponding to the acoustic feature matrix and the word table corresponding to the language; when the identification indicator indicates to continue identification, obtaining an updated input feature matrix based on the input feature matrix and all obtained identification characters, and obtaining a next identification character and a next identification indicator based on the updated input feature matrix, until the next identification indicator indicates to stop identification.

[0006] In an implementation form of the first aspect, the obtaining of the word list of each language and the obtaining of the word list feature vector of the word list comprises the following steps:

[0007] The word list of each language is obtained.

[0008] For each language word list, all the words in the language word list are spliced to obtain a spliced word list.

[0009] The word list feature of the spliced word list is extracted based on a BERT model.

[0010] The word list feature is normalized based on a sigmoid function to obtain the word list feature vector.

[0011] In an implementation form of the first aspect, the obtaining of the language identifier feature of the language identifier and the recognition identifier feature of the recognition identifier comprises the following steps:

[0012] The language identifier is input into a word segmenter and a multi-layer perceptron in sequence to obtain the language identifier feature.

[0013] The recognition identifier is input into a word segmenter and a multi-layer perceptron in sequence to obtain the recognition identifier feature.

[0014] In an implementation form of the first aspect, the acoustic feature matrix of the speech to be recognized is obtained based on a wav2vec model.

[0015] In an implementation form of the first aspect, the obtaining of the language feature vector based on the vector corresponding to the language identifier feature in the output feature matrix comprises the following steps:

[0016] The vector corresponding to the language identifier feature in the output feature matrix is obtained.

[0017] The vector is input into a multi-layer perceptron and a sigmoid function in sequence to obtain the language feature vector.

[0018] In an implementation form of the first aspect, the language word list corresponding to the language of the speech to be recognized is determined based on the similarity of the language feature vector and the word list feature vector, comprising the following steps:

[0019] The similarity of the language feature vector and each word list feature vector is calculated.

[0020] The word list corresponding to the word list feature vector with the largest similarity is selected as the language word list corresponding to the language of the speech to be recognized.

[0021] In an implementation form of the first aspect, obtaining the recognized character and the recognized indicator based on the vector corresponding to the recognized identifier feature in the output feature matrix, the vector corresponding to the acoustic feature matrix, and the word list of the corresponding language comprises the following steps:

[0022] passing the vector corresponding to the recognized identifier feature in the output feature matrix through two multi-layer perceptrons respectively to obtain a first initial cluster center point vector and a second initial cluster center point vector;

[0023] taking the vector corresponding to the acoustic feature matrix in the output feature matrix as a vector to be clustered;

[0024] performing kmeans clustering based on the first cluster center point, the second cluster center point, and the vector to be clustered to obtain a first cluster center point vector and a second cluster center point vector;

[0025] inputting the first cluster center point vector into a multi-layer perceptron and a softmax function in sequence to obtain a probability of each word in the word list of the corresponding language, and selecting a word with the maximum probability as the recognized character;

[0026] inputting the second cluster center point vector into a multi-layer perceptron and a softmax function in sequence to obtain the recognized indicator; when the recognized indicator is greater than a preset value, it indicates that the recognition is continued; when the recognized indicator is not greater than the preset value, it indicates that the recognition is stopped.

[0027] In a second aspect, the present application provides a multi-language automatic speech recognition system, which comprises a first acquisition module, a second acquisition module, a third acquisition module, a processing module, a fourth acquisition module, a determination module, a recognition module, and an iteration module;

[0028] The first acquisition module is configured to acquire word lists of multiple languages and acquire word list feature vectors of the word lists.

[0029] The second acquisition module is configured to set a language identifier and a recognized identifier, and acquire a language identifier feature of the language identifier and a recognized identifier feature of the recognized identifier.

[0030] The third acquisition module is configured to acquire a speech to be recognized and acquire an acoustic feature matrix of the speech to be recognized.

[0031] The processing module is configured to construct an input feature matrix based on the language identifier feature, the acoustic feature matrix, and the recognized identifier feature, and input the input feature matrix into a large language model to obtain an output feature matrix.

[0032] The fourth acquisition module is configured to obtain a language feature vector based on a vector corresponding to the language identifier feature in the output feature matrix.

[0033] The determining module is configured to determine a vocabulary corresponding to the language of the to-be-recognized speech based on the language feature vector and the vocabulary feature vector.

[0034] The recognizing module is configured to obtain a recognized character and a recognition indicator based on a vector corresponding to the recognition indicator feature in the output feature matrix, a vector corresponding to the acoustic feature matrix, and the vocabulary corresponding to the language.

[0035] The iteration module is configured to obtain an updated input feature matrix based on the input feature matrix and all obtained recognized characters when the recognition indicator indicates to continue recognizing, and obtain a next recognized character and a next recognition indicator based on the updated input feature matrix until the next recognition indicator indicates to stop recognizing.

[0036] In a third aspect, the present application provides an electronic device, comprising a processor and a memory.

[0037] The memory is configured to store a computer program.

[0038] The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the multi-language automatic speech recognition method described above.

[0039] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by an electronic device to implement the multi-language automatic speech recognition method described above.

[0040] As described above, the multi-language automatic speech recognition method, system, storage medium and electronic device of the present application have the following beneficial effects:

[0041] (1) The automatic recognition can be realized for the speech of multiple languages, and the resource consumption of the system is effectively reduced;

[0042] (2) Fast speed, high accuracy and strong practicability;

[0043] (3) High degree of intelligence, greatly improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A scene schematic diagram of an embodiment of the electronic device of the present application is shown;

[0045] Figure 2 A flowchart of an embodiment of the multi-language automatic speech recognition method of the present application is shown;

[0046] Figure 3 An architecture schematic diagram of an embodiment of the multi-language automatic speech recognition method of the present application is shown.

[0047] Figure 4 FIG. 1 shows a structural schematic diagram of a multilingual automatic speech recognition system according to an embodiment of the present application;

[0048] Figure 5 FIG. 2 shows a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] The present application is described below by way of specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure herein. The present application can also be implemented or applied in other different specific embodiments, and various modifications or changes can be made to the details herein based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0050] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of the present application, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component can be arbitrarily changed in terms of shape, number and proportion, and the layout pattern of the components can also be more complex.

[0051] The following embodiments of the present application provide a multilingual automatic speech recognition method, which can be applied to an electronic device as shown in Figure 1 The electronic device described in the present application can include a mobile phone 11, a tablet computer 12, a notebook computer 13, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. with wireless charging function, and the specific type of the electronic device is not limited in the embodiments of the present application.

[0052] For example, the electronic device can be a station (STATION, STA) in a WLAN with wireless charging capability, a cellular phone with wireless charging capability, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless charging capability, a computing device or other processing device, a computer, a laptop, a handheld communications device, a handheld computing device, and / or other devices for communicating over a wireless system and next generation communication systems, e.g., a mobile terminal in a 5G network, a mobile terminal in a future evolved Public Land Mobile Network (PLMN), or a mobile terminal in a future evolved Non-Terrestrial Network (NTN), etc.

[0053] For example, the electronic device can communicate with a network and other devices via wireless communication. The wireless communication can use any communication standards or protocols, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou navigation satellite system (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0054] The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0055] As shown in Figure 2 and Figure 3 In an embodiment, the multilingual automatic speech recognition method of the present application comprises steps S1-S8.

[0056] Step S1, obtain a plurality of language word tables, and obtain word table feature vectors of the word tables.

[0057] Specifically, corresponding word tables are provided for N different languages, such as Chinese word tables, English word tables, etc. For each language word table, all the words in the language word table are spliced to obtain a spliced word table. Then, the word table features of the spliced word table are extracted based on the BERT model, and the word table features are normalized based on the sigmoid function to obtain the word table feature vectors. Wherein, N word tables correspond to N word table feature vectors.

[0058] Step S2, set a language identifier and an identification identifier, and obtain language identifier features of the language identifier and identification identifier features of the identification identifier.

[0059] Specifically, the language identifier <lang>and identifying an identifier <asr>wherein <lang>to indicate that what follows is speech in a certain language, <asr>The content behind it is speech recognition. The language identifier is input into a word segmenter and a multilayer perceptron (MLP) in turn, and the language identifier features are obtained. The recognition identifier is input into a word segmenter and a multilayer perceptron in turn, and the recognition identifier features are obtained.

[0060] Step S3, obtaining the speech to be recognized, and obtaining the acoustic feature matrix of the speech to be recognized.

[0061] Specifically, the acoustic feature matrix of the speech to be recognized is obtained based on a wav2vec model. The wav2vec model is an unsupervised speech pre-training model that can be migrated to speech downstream tasks and used for speech feature extraction.

[0062] Step S4, constructing an input feature matrix based on the language identifier features, the acoustic feature matrix and the recognition identifier features, and inputting the input feature matrix into a large language model to obtain an output feature matrix.

[0063] Specifically, a large language model (LLM) is an artificial intelligence model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. LLMs are characterized by their size, containing billions of parameters that help them learn complex patterns in language data. These models are usually based on deep learning architectures such as transformers, which help them achieve impressive performance on various NLP tasks.

[0064] The language identifier features, the acoustic feature matrix and the recognition identifier features are connected in turn to form the input feature matrix. The input feature matrix is input into the large language model to obtain the output feature matrix.

[0065] Step S5, obtaining a language feature vector based on the vector corresponding to the language identifier features in the output feature matrix.

[0066] Specifically, in the output feature matrix, the language identifier <lang>The vectors in the corresponding positions are sequentially input into the multilayer perceptron and the sigmoid function, and the language feature vector can be obtained.

[0067] In step S6, the word table corresponding to the language of the speech to be recognized is determined based on the language feature vector and the word table feature vector.

[0068] Specifically, the similarity, such as the cosine similarity, between the language feature vector and each word table feature vector is first calculated. Then, the word table feature vector with the maximum similarity is selected as the word table corresponding to the language of the speech to be recognized.

[0069] In step S7, the recognized character and the recognized indicator are obtained based on the vector corresponding to the recognized indicator feature in the output feature matrix, the vector corresponding to the acoustic feature matrix, and the word table corresponding to the corresponding language.

[0070] Specifically, the recognized character and the recognized indicator are obtained based on the vector corresponding to the recognized indicator feature in the output feature matrix, the vector corresponding to the acoustic feature matrix, and the word table corresponding to the corresponding language, including the following steps:

[0071] 71) The vectors in the positions corresponding to the recognized indicator feature in the output feature matrix are respectively input into two multilayer perceptrons to obtain a first initial cluster center point vector and a second initial cluster center point vector. The first initial cluster center point vector and the second initial cluster center point vector are used as the initial center point vectors of two clusters.

[0072] 72) The vector corresponding to the acoustic feature matrix in the output feature matrix is used as a vector to be clustered.

[0073] 73) The kmeans clustering is performed based on the first cluster center point, the second cluster center point, and the vector to be clustered to obtain a first cluster center point vector and a second cluster center point vector.

[0074] Among them, according to the first cluster center point, the second cluster center point, and the vector to be clustered, the kmeans clustering algorithm is used to divide the vector to be clustered into two clusters, so as to obtain the final first cluster center point vector and the second cluster center point vector.

[0075] 74) The first cluster center point vector is sequentially input into the multilayer perceptron and the softmax function to obtain the probability of each word in the word table corresponding to the corresponding language, and the word with the maximum probability is selected as the recognized character.

[0076] 75) inputting the second cluster center point vector into a multi-layer perception machine and a softmax function in sequence to obtain the recognition indicator; when the recognition indicator is greater than a preset value, it indicates to continue recognition; when the recognition indicator is not greater than the preset value, it indicates to stop recognition.

[0077] wherein the score of the recognition indicator is between 0 and 1. If greater than 0.5, continue recognition, otherwise stop recognition.

[0078] Step S8, when the recognition indicator indicates to continue recognition, an updated input feature matrix is obtained based on the input feature matrix and all obtained recognition characters, and a next recognition character and a next recognition indicator are obtained based on the updated input feature matrix, until the next recognition indicator indicates to stop recognition.

[0079] Specifically, a first recognition character and a first recognition indicator are obtained based on the vector corresponding to the recognition identifier feature in the output feature matrix, the vector corresponding to the acoustic feature matrix and the word list of the corresponding language. When the first recognition indicator indicates to continue recognition, the first recognition identifier is appended after the input feature vector to form an updated input feature vector, and then steps S4-S6 are executed again to obtain a second recognition character and a second recognition indicator. When the second recognition indicator indicates to continue recognition, the second recognition character is appended after the updated input feature vector to form a further updated input feature vector, i.e. the language identifier feature + the acoustic feature matrix + the recognition identifier feature + the first recognition character + the second recognition character. Steps S4-S6 are executed again based on the updated input feature vector until the obtained recognition indicator indicates to stop recognition. At this time, the obtained recognition characters are sequentially connected in order, which is the recognition text of the to-be-recognized speech.

[0080] The protection scope of the multilingual automatic speech recognition method according to the embodiments of the present application is not limited to the execution order of the steps listed in the embodiments, and any scheme realized by adding, replacing or substituting steps according to the principle of the present application is included in the protection scope of the present application.

[0081] The embodiments of the present application also provide a multilingual automatic speech recognition system which can realize the multilingual automatic speech recognition method according to the present application, but the implementation device of the multilingual automatic speech recognition system according to the present application includes but is not limited to the structure of the multilingual automatic speech recognition system listed in the embodiments, and any structure deformation and substitution according to the principle of the present application is included in the protection scope of the present application.

[0082] As Figure 4 As shown, in an embodiment, the multilingual automatic speech recognition system of the present application comprises a first obtaining module 41, a second obtaining module 42, a third obtaining module 43, a processing module 44, a fourth obtaining module 45, a determining module 46, a recognition module 47 and an iteration module 48.

[0083] The first obtaining module 41 is configured to obtain a plurality of language tables and obtain language table feature vectors of the language tables.

[0084] The second obtaining module 42 is configured to set a language identifier and a recognition identifier, and obtain language identifier features of the language identifier and recognition identifier features of the recognition identifier.

[0085] The third obtaining module 43 is configured to obtain a speech to be recognized and obtain an acoustic feature matrix of the speech to be recognized.

[0086] The processing module 44 is connected to the second obtaining module 42 and the third obtaining module 43, and is configured to construct an input feature matrix based on the language identifier features, the acoustic feature matrix and the recognition identifier features, input the input feature matrix into a large language model, and obtain an output feature matrix.

[0087] The fourth obtaining module 45 is connected to the processing module 44, and is configured to obtain a language feature vector based on a vector corresponding to the language identifier features in the output feature matrix.

[0088] The determining module 46 is connected to the first obtaining module 41 and the fourth obtaining module 45, and is configured to determine a language table corresponding to the speech to be recognized based on the language feature vector and the language table feature vectors.

[0089] The recognition module 47 is connected to the processing module 44 and the determining module 46, and is configured to obtain a recognized character and a recognition indicator based on a vector corresponding to the recognition identifier features in the output feature matrix, a vector corresponding to the acoustic feature matrix and the language table corresponding to the speech to be recognized.

[0090] The iteration module 48 is connected to the recognition module 47, and is configured to, when the recognition indicator indicates to continue recognition, obtain an updated input feature matrix based on the input feature matrix and all recognized characters obtained, and obtain a next recognized character and a next recognition indicator based on the updated input feature matrix, until the next recognition indicator indicates to stop recognition.

[0091] The first obtaining module 41, the second obtaining module 42, the third obtaining module 43, the processing module 44, the fourth obtaining module 45, the determining module 46, the identifying module 47, and the iteration module 48 correspond to the steps of the multi-lingual automatic speech recognition method one by one in structure and principle, and thus will not be described here.

[0092] In several embodiments provided by the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the modules / units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules / units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or modules / units, and can be electrical, mechanical or other forms.

[0093] The modules / units described as separate components can or can not be physically separate, and the components shown as modules / units can or can not be physical modules, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules / units can be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in each embodiment of the present application can be integrated into a processing module, or each module / unit can be physically separate, or two or more modules / units can be integrated into one module / unit.

[0094] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0095] The embodiments of the present application further provide a computer readable storage medium. Those skilled in the art can understand that all or part of the steps of the method described above can be instructed by a program to complete the processor, and the program can be stored in a computer readable storage medium. The storage medium is a non-transitory medium, for example, a random access memory, a read-only memory, a flash memory, a hard disk, a solid state disk, a magnetic tape, a floppy disk, an optical disc and any combination thereof. The storage medium can be any available medium accessible by a computer or a data storage device such as a server, a data center and the like, which includes one or more available medium sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)) or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0096] The embodiments of the present application further provide an electronic device. The electronic device comprises a processor and a memory.

[0097] The memory is used to store a computer program.

[0098] The memory comprises a ROM, a RAM, a disk, a U disk, a memory card or an optical disc and the like various medium capable of storing program codes.

[0099] The processor is connected with the memory, and is used to execute the computer program stored in the memory, so that the electronic device executes the multi-lingual automatic speech recognition method described above.

[0100] Preferably, the processor can be a general processor, including a central processing unit (CPU), a network processor (NP) and the like; and can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0101] As Figure 5 As shown, the electronic device of the present application is in the form of a general- purpose computing device. The components of the electronic device can include, but are not limited to, one or more processors or processing units 51, a main memory 52, and a bus 53 that couples various system components including the memory 52 to the processing unit 51.

[0102] The bus 53 represents one or more of any of several bus structures, including a memory bus or memory controller, a peripheral bus, a graphics accelerator bus, a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0103] The electronic device typically includes a variety of computer system readable media. Such media can be any available media that is accessible by the electronic device and includes both volatile and non- volatile media, removable and non-removable media.

[0104] The memory 52 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 521 and / or cache memory 522. The electronic device can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 523 can be provided for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 5 Not shown, a removable / non-removable interface can also be coupled to the bus 53 through which a removable storage unit 524 is attached and detached, such as a magnetic disk, optical disk or Universal Serial Bus (USB) stick, for example. It will be appreciated that the storage devices 523, 524 have memory storage capacities which can be other than that shown, including capacities of a few terabytes, or even more. It is also appreciated that the storage devices 523, 524 described and depicted are only examples and that other types of storage devices can be used including optical drives such as CD-ROM, DVD-ROM players, Blu-ray players, and the like, without departing from the scope of the present application. Figure 5 Not shown, a removable / non-removable interface can also be coupled to the bus 53 through which a removable storage unit 524 is attached and detached, such as a magnetic disk, optical disk or Universal Serial Bus (USB) stick, for example. It will be appreciated that the storage devices 523, 524 have memory storage capacities which can be other than that shown, including capacities of a few terabytes, or even more. It is also appreciated that the storage devices 523, 524 described and depicted are only examples and that other types of storage devices can be used including optical drives such as CD-ROM, DVD-ROM players, Blu-ray players, and the like, without departing from the scope of the present application.

[0105] A program / utility 524 having a set (at least one) of program modules 5241 can be stored in, for example, memory 52 by way of example, such program modules 5241 include an operating system, one or more application programs, other program modules, and program data, each of which or a combination can include implementation of a network environment. The program modules 5241 typically carry out the functions and / or methodologies of the embodiments described herein.

[0106] The electronic device can also communicate with one or more external devices such as a keyboard or a pointing device, displays, etc.; other devices that enable a user to interact with the electronic device; and / or any devices (e.g., network card, modem, etc.) that enable the electronic device to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface 54. Still yet, the electronic device can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the Internet through network adapter 55. As Figure 5 illustrated, network adapter 55 communicates with the other components of the electronic device via bus 53. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with the electronic device. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0107] The foregoing embodiments are merely exemplary and are not used in a limiting sense. As such, those skilled in the art will readily appreciate that variations of the embodiments described herein that do not depart from the spirit and scope of the application are intended to be encompassed by the claims. Accordingly, any equivalent modifications or changes which do not depart from the spirit and scope of the application are intended to be included in the claims.< / lang> < / asr> < / lang> < / asr> < / lang>

Claims

1. A multilingual automatic speech recognition method, characterized in that, The method includes the following steps: Obtain character lists for multiple languages, and obtain the character list feature vectors of the character lists; Set a language identifier and an identification identifier, and obtain the language identifier features of the language identifier and the identification identifier features of the identification identifier; Obtain the speech to be recognized, and obtain the acoustic feature matrix of the speech to be recognized; An input feature matrix is ​​constructed based on the language identifier features, the acoustic feature matrix, and the identification identifier features. The input feature matrix is ​​then input into a large language model to obtain an output feature matrix. The language feature vector is obtained based on the vector corresponding to the language identifier feature in the output feature matrix; The character list of the language corresponding to the speech to be identified is determined based on the language feature vector and the character list feature vector; Based on the vector corresponding to the identification identifier feature in the output feature matrix, the vector corresponding to the acoustic feature matrix, and the character list of the corresponding language, the identification character and identification indicator are obtained; When the recognition indicator indicates that recognition should continue, an updated input feature matrix is ​​obtained based on the input feature matrix and all the previously acquired recognition characters. The next recognition character and the next recognition indicator are then obtained based on the updated input feature matrix, until the next recognition indicator indicates that recognition should stop.

2. The multilingual automatic speech recognition method according to claim 1, characterized in that: Obtaining character lists for multiple languages ​​and then obtaining the character list feature vectors of those character lists includes the following steps: Obtain the corresponding character set for each language; For each language character list, concatenate all the characters in the language character list to obtain a concatenated character list; The character table features of the concatenated character table are extracted based on the BERT model; The character table features are normalized using the sigmoid function to obtain the character table feature vector.

3. The multilingual automatic speech recognition method according to claim 1, characterized in that: Obtaining the language identifier features of the language identifier and the identification identifier features of the identification identifier includes the following steps: The language identifier is sequentially input into the word segmenter and the multilayer perceptron to obtain the language identifier features; The identification identifier is sequentially input into the word segmenter and the multilayer perceptron to obtain the features of the identification identifier.

4. The multilingual automatic speech recognition method according to claim 1, characterized in that: The acoustic feature matrix of the speech to be recognized is obtained based on the wav2vec model.

5. The multilingual automatic speech recognition method according to claim 1, characterized in that: Obtaining the language feature vector based on the vector corresponding to the language identifier feature in the output feature matrix includes the following steps: Obtain the vector corresponding to the language identifier feature in the output feature matrix; The vector is sequentially input into a multilayer perceptron and a sigmoid function to obtain the language feature vector.

6. The multilingual automatic speech recognition method according to claim 1, characterized in that: Determining the vocabulary of the language corresponding to the speech to be identified based on the similarity between the language feature vector and the vocabulary feature vector includes the following steps: Calculate the similarity between the language feature vector and the feature vector of each character table; The character list corresponding to the character list feature vector with the highest similarity is selected as the character list of the language corresponding to the speech to be identified.

7. The multilingual automatic speech recognition method according to claim 1, characterized in that: Obtaining the recognition character and recognition indicator based on the vector corresponding to the recognition identifier feature in the output feature matrix, the vector corresponding to the acoustic feature matrix, and the vocabulary of the corresponding language includes the following steps: The vectors corresponding to the identification identifier features in the output feature matrix are passed through two multilayer perceptrons to obtain the first initial cluster center point vector and the second initial cluster center point vector. The vector corresponding to the acoustic feature matrix in the output feature matrix is ​​used as the clustering vector; Based on the first cluster center point, the second cluster center point, and the vector to be clustered, kmeans clustering is performed to obtain the first cluster center point vector and the second cluster center point vector. The first cluster center point vector is sequentially input into a multilayer perceptron and a softmax function to obtain the probability of each character in the character table of the corresponding language, and the character with the highest probability is selected as the recognition character. The second cluster center point vector is sequentially input into a multilayer perceptron and a softmax function to obtain the recognition indicator; when the recognition indicator is greater than a preset value, recognition continues; when the recognition indicator is not greater than the preset value, recognition stops.

8. A multilingual automatic speech recognition system, characterized in that, The system includes a first acquisition module, a second acquisition module, a third acquisition module, a processing module, a fourth acquisition module, a determination module, an identification module, and an iteration module; The first acquisition module is used to acquire character lists for multiple languages ​​and to acquire character list feature vectors of the character lists; The second acquisition module is used to set the language identifier and the identification identifier, and to acquire the language identifier features of the language identifier and the identification identifier features of the identification identifier; The third acquisition module is used to acquire the speech to be recognized and to acquire the acoustic feature matrix of the speech to be recognized; The processing module is used to construct an input feature matrix based on the language identifier features, the acoustic feature matrix, and the identification identifier features, and input the input feature matrix into a large language model to obtain an output feature matrix; The fourth acquisition module is used to acquire the language feature vector based on the vector corresponding to the language identifier feature in the output feature matrix; The determining module is used to determine the character list of the language corresponding to the speech to be identified based on the language feature vector and the character list feature vector; The recognition module is used to obtain recognition characters and recognition indicators based on the vector corresponding to the recognition identifier features in the output feature matrix, the vector corresponding to the acoustic feature matrix, and the character list of the corresponding language; The iterative module is used to obtain an updated input feature matrix based on the input feature matrix and all the acquired recognition characters when the recognition indicator indicates that recognition should continue, and to obtain the next recognition character and the next recognition indicator based on the updated input feature matrix, until the next recognition indicator indicates that recognition should stop.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to cause the electronic device to perform the multilingual automatic speech recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the multilingual automatic speech recognition method as described in any one of claims 1 to 7.

Images