System and method for multi-spoken language detection

Abstract

A method includes: using at least one processor to perform feature extraction of input audio data to identify extracted features associated with the input audio data. The method also includes: using the at least one processor to detect a language associated with each of a plurality of portions of the input audio data by processing the extracted features using a plurality of language models, wherein each of the plurality of language models is associated with a different language. In addition, the method includes: using the at least one processor to direct each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on the language associated with each portion of the input audio data.

Description

Technical Field

[0001] The present disclosure generally relates to automatic speech recognition (ASR) systems. More specifically, the present disclosure relates to a system and method for multi-spoken language detection. Background Art

[0002] Speech-based interfaces are increasingly being used as a mechanism for supporting interaction between humans and machines. These types of interfaces typically include an automatic speech recognition (ASR) system designed to convert speech into text. The text can then be displayed, transmitted to other users, further processed or used to perform one or more functions, or used in any other suitable manner. This type of functionality is common in various applications such as speech-based digital personal assistants on mobile electronic devices or other electronic devices. Summary of the invention

[0003] Technical Solution

[0004] The present disclosure provides a system and method for multi-spoken language detection.

[0005] In a first embodiment, a method includes: using at least one processor to perform feature extraction of input audio data to identify extracted features associated with the input audio data. The method also includes: using the at least one processor to detect a language associated with each of a plurality of portions of the input audio data by processing the extracted features using a plurality of language models, wherein each of the plurality of language models is associated with a different language. In addition, the method includes: using the at least one processor, directing each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on the language associated with each portion of the input audio data.

[0006] In a second embodiment, an electronic device includes at least one memory and at least one processor operably coupled to the at least one memory. The at least one processor is configured to perform feature extraction of input audio data to identify extracted features associated with the input audio data. The at least one processor is further configured to detect a language associated with each of a plurality of portions of the input audio data by processing the extracted features using a plurality of language models, wherein each of the plurality of language models is associated with a different language. The at least one processor is further configured to direct each portion of the input audio data to one of a plurality of ASR models based on the language associated with each portion of the input audio data.

[0007] In a third embodiment, a non-transitory machine-readable medium includes instructions, wherein the instructions, when executed, cause at least one processor of an electronic device to perform the following operations: perform feature extraction of input audio data to identify extracted features associated with the input audio data. The medium also includes instructions, wherein the instructions, when executed, cause the at least one processor to perform the following operations: detect a language associated with each of a plurality of portions of the input audio data by processing the extracted features using a plurality of language models, wherein each of the plurality of language models is associated with a different language. The medium also includes instructions, wherein the instructions, when executed, cause the at least one processor to perform the following operations: direct each portion of the input audio data to one of a plurality of ASR models based on the language associated with each portion of the input audio data.

[0008] Other technical features may be apparent to those skilled in the art from the following drawings, descriptions and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0010] Figure 1 shows an example network configuration according to the present disclosure;

[0011] Figure 2 An example architecture for multi-speaking detection according to the present disclosure is shown;

[0012] Figure 3 A more specific example architecture for multi-spoken language detection according to the present disclosure is shown;

[0013] Figure 4 illustrates an example architecture for supporting recognition of translations between multiple languages ​​according to the present disclosure; and

[0014] Figure 5 An example method for multi-spoken language detection according to the present disclosure is shown. DETAILED DESCRIPTION

[0015] Before proceeding to the following detailed description, it may be advantageous to set forth definitions of specific words and phrases used throughout this patent document. The terms "send," "receive," and "transmit," and their derivatives, encompass both direct and indirect communications. The terms "include," "comprise," and their derivatives, mean including, but not limited to. The term "or" is inclusive, meaning and / or. The phrase "associated with," and its derivatives, means including, included within, interconnected with, containing, contained within, connected to or connected with, coupled to or coupled with, communicable with, collaborate with, intertwined, juxtaposed, proximate to, bound to or bound with, having, having the nature of, and or having a relationship with, etc.

[0016] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed from a computer-readable program code and implemented in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or a part thereof that are suitable for being implemented in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disc (CD), a digital video disc (DVD) or any other type of memory. "Non-transitory" computer-readable media does not include wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and rewrite data later, such as rewritable optical discs or erasable memory devices.

[0017] As used herein, terms and phrases such as "having", "may have", "include" or "may include" a feature (such as a number, function, operation or component such as a part) indicate the presence of the feature and do not exclude the presence of other features. In addition, as used herein, the phrases "A or B", "at least one of A and / or B" or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B" and "at least one of A or B" may indicate all of the following: (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B. In addition, as used herein, the terms "first" and "second" may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate user devices that are different from each other, regardless of the order or importance of the devices. Without departing from the scope of the present disclosure, a first component may be represented as a second component, and vice versa.

[0018] It will be understood that when an element (such as a first element) is referred to as being (operably or communicatively) “coupled to / coupled to” another element (such as a second element) or “connected to / connected to” another element (such as a second element), it may be coupled to or connected to / coupled to or connected to the other element directly or via a third element. Conversely, it will be understood that when an element (such as a first element) is referred to as being “directly coupled to” / “directly coupled to” another element (such as a second element) or “directly connected to” / “directly connected to” another element (such as a second element), no other elements (such as a third element) are interposed between the element and the other element.

[0019] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for," "capable of," "designed to," "suitable for," "made to," or "capable of," depending on the circumstances. The phrase "configured (or set) to" does not essentially mean "specially designed in hardware to." Rather, the phrase "configured to" may mean that a device is capable of performing an operation together with another device or component. For example, the phrase "a processor configured (or set) to perform A, B, and C" may mean a general-purpose processor (such as a CPU or an application processor) that can perform operations by executing one or more software programs stored in a memory device, or a dedicated processor (such as an embedded processor) for performing operations.

[0020] The terms and phrases used herein are only used to describe some embodiments of the present disclosure, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that, unless the context clearly indicates otherwise, the singular form includes plural references. All terms and phrases used herein (including technical terms and phrases and scientific terms and phrases) have the same meaning as the meaning generally understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It will also be understood that terms and phrases (such as those defined in a common dictionary and phrases) should be interpreted as having the same meaning as the meaning in the context of the relevant field, and will not be interpreted with an idealized or overly formal meaning, unless clearly defined here. In some cases, the terms and phrases defined herein may be interpreted as not including embodiments of the present disclosure.

[0021] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head mounted device (HMD), electronic clothing, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washing machine, a dryer, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (such as SAMSUNG HOMESYNC, APPLETV, or GOOGLE TV), a smart speaker or a speaker with an integrated digital assistant (such as SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a game console (such as XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a camera, or at least one of an electronic photo frame. Still other examples of electronic devices include various medical devices (such as various portable medical measuring devices (such as blood sugar measuring devices, heart rate measuring devices, or body temperature measuring devices), magnetic resonance angiography (MRA) devices, magnetic resonance imaging (MRI) devices, computed tomography (CT) devices, imaging devices, or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDR), flight data recorders (FDR), car infotainment devices, navigation electronic devices (such as navigation navigation devices or gyrocompasses), avionics equipment, security devices, vehicle head units, industrial or home robots, automated teller machines (ATMs), point of sale (POS) devices, or at least one of Internet of Things (IoT) devices (such as light bulbs, various sensors, electric or gas meters, sprinklers, fire alarms, thermostats, street lights, ovens, fitness equipment, hot water tanks, heaters, or boilers). Other examples of electronic devices include at least a part of a piece of furniture or a building / construction, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). Note that according to various embodiments of the present disclosure, the electronic device may be one or a combination of the devices listed above. According to some embodiments of the present disclosure, the electronic device may be a flexible electronic device. The electronic devices disclosed herein are not limited to the devices listed above, and may include new electronic devices according to the development of technology.

[0022] In the following description, an electronic device is described with reference to the accompanying drawings according to various embodiments of the present disclosure. As used herein, the term "user" may refer to a person or another device (such as an artificial intelligence electronic device) using the electronic device.

[0023] Definitions for additional specific words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0024] Nothing in this application should be interpreted as implying that any particular element, step, or function is an essential element that must be included in the scope of the claims. The scope of the patent subject matter is limited only by the claims. Any other terms used in the claims, including but not limited to "mechanism", "module", "device", "unit", "component", "element", "member", "device", "machine", "system", "processor" or "controller", are understood by the applicant to refer to structures known to technicians in the relevant field.

[0025] The following discussion is described with reference to the accompanying drawings. Figures 1 to 5 As well as various embodiments of the present disclosure. However, it should be understood that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or replacements thereof also belong to the scope of the present disclosure. Throughout this specification and the accompanying drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0026] As mentioned above, voice-based interfaces are more and more often used to support the interaction between people and machines. These types of interfaces usually include an automatic speech recognition (ASR) system that converts speech to text. In an ASR system, determining the language that the user is speaking is usually a prerequisite for performing automatic speech recognition operations. However, there are many technical challenges when performing automatic speech recognition, such as overlapping phonemes between different languages, overlapping characteristics of different speakers (such as native and non-native speakers or male and female speakers) using the same language, and matching of wide phoneme distributions between different languages. In some existing methods, such as during the setting of a mobile electronic device or other device, the user is simply given the option of selecting a specific language to be used. In these methods, the user is usually restricted to using the specific language, and the user cannot usually switch between different languages ​​(at least not very easily and usually cannot without repeating the setting process).

[0027] The present disclosure provides a technology for multi-spoken language detection, wherein the technology for multi-spoken language detection enables automatic selection and use of one or more language-specific ASR models to process input in other aspects. As described in more detail below, a device receives input data, such as audio data associated with a user's speech. The device performs feature extraction to identify features associated with the input data, such as by identifying features of phonemes from the input data. Multiple language models (also referred to as language-specific acoustic models) are used to process the extracted features to identify the language associated with each of the multiple parts of the input data. Depending on the input data, all parts of the input data may be associated with a single language, or different parts of the input data may be associated with different languages. Each part of the input data may be routed to a language-specific ASR model for further processing based on the identified language of the part, and each ASR model is adjusted (trained) for a specific language. In some embodiments, since the recognition of the language used in the input data may involve fewer features than the recognition of the actual words or phrases contained in the input data, the language model may be obtained from the ASR model and may process fewer features than the ASR model.

[0028] In this way, since one or more languages ​​associated with the input audio data can be automatically identified, these technologies enable electronic devices or other devices to receive and process the input audio data more effectively. In addition, these technologies enable electronic devices or other devices to seamlessly switch between processing different languages ​​without requiring the user to select different languages ​​or to re-execute the device setup process to select different languages. This may be particularly useful for users who speak multiple languages ​​and may switch between different languages ​​in the same input audio data. In addition, the ability to improve the detection of spoken language can help ASR-based systems provide better responses to user inputs, and can enable the system to serve a wider range of potential user groups.

[0029] As a specific example of this functionality, the accuracy of an ASR system is typically highly dependent on the training of the ASR system and the conditions under which input data is provided to the ASR system for processing. Although the training data used to train an ASR system typically contains pure speech from a particular language, the input data provided to an ASR system during real-world use rarely has the same characteristics. For example, a user who speaks multiple languages ​​may often interact with devices that use different languages. Using the techniques described below, one or more language-specific ASR models may be used for each user input based on the language or languages ​​of that user input. Therefore, in some embodiments, automatic detection of one or more spoken languages ​​associated with a particular input may be performed as part of a preprocessing operation in order to identify the language and use an appropriate ASR model for the identified language.

[0030] Figure 1 An example network configuration 100 according to the present disclosure is shown. Figure 1 As shown in , according to an embodiment of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or an event processing module 180. In some embodiments, the electronic device 101 may not include at least one of the components or may add another component.

[0031] The bus 110 may include circuits for connecting the components 120-180 to each other and transmitting communications (such as control messages and / or data) between the components. The processor 120 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 may control at least one of the other components of the electronic device 101 and / or perform operations or data processing related to communication. As described in more detail below, the processor 120 may analyze the input audio data, automatically identify one or more languages ​​used in the input audio data, and process the input audio data based on the identified language.

[0032] The memory 130 may include a volatile memory and / or a non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or a program 140. The program 140 may include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0033] The kernel 141 may control or manage system resources (such as bus 110, processor 120 or memory 130) for executing operations or functions implemented in other programs (such as middleware 143, API 145 or application 147). The kernel 141 may provide an interface that allows the middleware 143, API 145 or application 147 to access various components of the electronic device 101 to control or manage system resources. The application 147 includes one or more applications for processing input data based on automatic language detection as described below. These functions may be performed by a single application or multiple applications, wherein each application performs one or more of these functions. For example, the middleware 143 may be used as a relay for allowing the API 145 or application 147 to communicate data with the kernel 141. Multiple applications 147 may be provided. The middleware 143 may control a work request received from the application 147, such as by assigning a priority of using the system resources (such as bus 110, processor 120 or memory 130) of the electronic device 101 to at least one of the multiple applications 147. The API 145 is an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 may include at least one interface or function (such as a command) for file control, window control, image processing, or text control.

[0034] The input / output interface 150 may serve as an interface that may, for example, transmit commands or data input from a user or other external devices to other components of the electronic device 101. In addition, the input / output interface 150 may output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0035] The display 160 may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 may display various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hover input using an electronic pen or a body part of the user.

[0036] The communication interface 170 may establish communication between the electronic device 101 and an external electronic device such as the first electronic device 102, the second electronic device 104, or the server 106. For example, the communication interface 170 may be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device.

[0037] Wireless communication may use, for example, at least one of Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM) as a cellular communication protocol. Wired connections may include, for example, at least one of Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), Recommended Standard 232 (RS-232), or Plain Old Telephone Service (POTS). Network 162 or 164 may include at least one communication network, such as a computer network (e.g., a Local Area Network (LAN) or a Wide Area Network (WAN)), the Internet, or a telephone network.

[0038] The first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as a head mounted display (HMD)) on which the electronic device 101 may be mounted. When the electronic device 101 is mounted in an HMD (such as the electronic device 102), the electronic device 101 may detect the installation in the HMD and operate in a virtual reality mode. When the electronic device 101 is mounted in the electronic device 102 (such as the HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without involving a separate network.

[0039] The first external electronic device 102 and the second external electronic device 104 may be devices of the same type or different types as the electronic device 101. According to an embodiment of the present disclosure, the server 106 may include a group of one or more servers. In addition, according to an embodiment of the present disclosure, all or some of the operations performed on the electronic device 101 may be performed on another electronic device or multiple other electronic devices (such as the electronic devices 102 and 104 or the server 106). In addition, according to an embodiment of the present disclosure, when the electronic device 101 should automatically or by request perform a certain function or service, the electronic device 101 may request another device (such as the electronic devices 102 and 104 or the server 106) to perform at least some functions associated with the function or service, rather than performing the function or service alone or in addition. Another electronic device (such as the electronic devices 102 and 104 or the server 106) may perform the requested function or additional function and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the requested function or service by processing the received result as is or additionally. To this end, for example, cloud computing, distributed computing, or client-server computing techniques may be used.

[0040] According to the embodiments of the present disclosure, although Figure 1The electronic device 101 is shown to include a communication interface 170 to communicate with an external electronic device 102 or 104 or a server 106 via networks 162 and 164, but the electronic device 101 can operate independently without a separate communication function. In addition, it is noted that the electronic device 102 or 104 or the server 106 can be implemented using a bus, a processor, a memory, an I / O interface, a display, a communication interface, and an event processing module (or any suitable subset thereof) in the same or similar manner as shown for the electronic device 101.

[0041] The server 106 is operable to drive the electronic device 101 by performing at least one of the operations (or functions) implemented on the electronic device 101. For example, the server 106 may include an event processing server module (not shown) that may support the event processing module 180 implemented in the electronic device 101. The event processing server module may include at least one of the components of the event processing module 180, and perform (or alternatively perform) at least one of the operations (or functions) performed by the event processing module 180. The event processing module 180 may process at least a portion of the information obtained from other elements (such as the processor 120, the memory 130, the input / output interface 150, or the communication interface 170), and may provide it to the user in various ways.

[0042] Although the event processing module 180 is shown as Figure 1 The event processing module 180 is a separate module from the processor 120 in the memory 130, but at least a portion of the event processing module 180 may be included or implemented in the processor 120 or at least one other module, or the entire function of the event processing module 180 may be included or implemented in the processor 120 or another processor as shown. The event processing module 180 may interoperably perform operations according to an embodiment of the present disclosure with at least one program 140 stored in the memory 130.

[0043] although Figure 1 An example of a network configuration 100 is shown, but may be Figure 1 Various changes may be made. For example, network configuration 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0044] Figure 2 An example architecture 200 for multi-speech detection according to the present disclosure is shown. For ease of explanation, Figure 2 The architecture 200 shown in FIG. 2 can be described as using Figure 1 However, the architecture 200 may be used by any suitable device(s) and may be used in any suitable system(s), such as by the server 106 in the network configuration 100.

[0045] like Figure 2 As shown in , the architecture 200 generally operates to receive an audio-based input 202 and generate a text-based output 204 representing the audio-based input 202. The audio-based input 202 may represent any suitable audio information containing one or more spoken words. For example, the audio-based input 202 may represent a digitized (and possibly pre-processed) version of a user's speech that may be captured by a microphone or other component of the electronic device 101 in some cases. The audio-based input 202 may include a word or multiple words spoken by at least one user, and (multiple) users may use a single language or multiple languages ​​when speaking. The text-based output 204 represents the audio-based input 202 converted into text form using automatic speech recognition. Therefore, the architecture 200 can process the audio-based input 202 to identify the language used in the audio-based input 202, and convert the words and phrases contained in the audio-based input 202 into text in the text-based output 204.

[0046] In the architecture 200, an audio-based input 202 is provided to a feature extractor 206, wherein the feature extractor 206 generally operates to process the audio-based input 202 and extract various features from the audio-based input 202. The extracted features may represent any suitable characteristics of the audio-based input 202. In some embodiments, the extracted features include phonemes contained in the audio-based input 202. Phonemes represent units of the speech system of a language, wherein the units are perceived as different sounds in the language. Words in many languages ​​can be represented using a common set of phonemes, although the use of these phonemes varies greatly based on the language. Therefore, the feature extractor 206 can extract phoneme features from the audio-based input 202 to facilitate (i) identifying one or more languages ​​used in the audio-based input 202, and (ii) converting the audio-based input 202 into a text-based output 204 by identifying words and phrases contained in the audio-based input 202.

[0047] The feature extractor 206 can use any suitable technology for extracting phoneme features or other features from the audio-based input 202. For example, the feature extractor 206 can operate based on wavelet decomposition technology or convolutional neural network technology. Various methods for extracting features from audio input data have been developed, and other methods will certainly be developed in the future. The present disclosure is not limited to any particular type of feature extraction technology or any particular embodiment of the feature extractor 206. In addition, the feature extractor 206 can represent the extracted features in any suitable format. In some embodiments, for example, the extracted features can have the form of Mel frequency cepstral coefficients (MFCC), although the extracted features can be defined in any other suitable manner.

[0048] The extracted features of the audio-based input 202 are provided from the feature extractor 206 to a language-specific acoustic model scoring function 208, wherein the language-specific acoustic model scoring function 208 generally operates to identify the likelihood that various portions of the audio-based input 202 are from a particular language. In this example, the scoring function 208 includes or is used in conjunction with a plurality of language models 210 (also referred to as language-specific acoustic models). The language models 210 are associated with different languages ​​(or possibly different dialects of the same language, which are considered different languages ​​herein). In some embodiments, each language model 210 is trained or otherwise configured to calculate the probability that each portion of the audio-based input 202 is from a language for which the language model 210 has been trained or otherwise associated. Thus, for example, a first language model 210 (LM 1) may receive a first portion of an audio-based input 202 and generate a probability that the first portion of the audio-based input 202 is from a first language, a second language model 210 (LM 2) may receive a first portion of the audio-based input 202 and generate a probability that the first portion of the audio-based input 202 is from a second language, and so on. This processing may be performed by each language model 210 to generate a probability that the first portion of the audio-based input 202 is from a respective language, and this may occur in parallel such that all language models 210 process the first portion of the audio-based input 202 simultaneously or in an overlapping manner. Additionally, this processing may be performed for the second portion, the third portion, and other portions of the audio-based input 202. The output from the scoring function 208 represents a collection of probabilities for all portions of the audio-based input 202 across all supported languages.

[0049] The language-specific acoustic model scoring function 208 may support any suitable technique for generating probabilities or otherwise determining the likelihood that various portions of the audio-based input 202 are associated with various languages. For example, in some embodiments, each of the language models 210 may define a plurality of states and include inter-state transitions and intra-state transitions for moving between states. Each state may apply a particular function to an input data segment in order to generate a probability that the input data segment is a particular language. As a specific example, in some cases, the output of each language model 210 may be represented as P(phone k lang-y |x i ), where P(·) represents a specific phoneme phone for a specific language lang-y k A specific segment x present in the audio-based input 202 i Thus, each language model 210 may be used to calculate a probability for each phoneme associated with a particular language, and this may be performed for each portion (segment) of the audio-based input 202 .

[0050] The feature concatenator 212 combines the outputs from the scoring function 208 to generate a feature vector. The feature vector here may be generated in any suitable manner. For example, the feature concatenator 212 may accumulate the probabilities output from the scoring function 208 over time to create a matrix of values, which may then be further processed as described below. However, note that the feature concatenator 212 may combine the outputs from the scoring function 208 to generate a feature vector in any other suitable manner. In addition, in some embodiments, the feature concatenator 212 may combine the outputs from the scoring function 208 with the original extracted features (which may be referred to as "raw" features) generated by the feature extractor 206.

[0051] The feature vectors generated by the feature concatenator 212 are provided to a neural classification model 214, which processes the feature vectors to identify one or more languages ​​used in the audio-based input 202. The neural classification model 214 may use any suitable machine learning technique to process the feature vectors and identify a language based on the feature vectors. For example, the neural classification model 214 may include at least one convolutional neural network and associated layers, such as normalization layers and pooling layers, for processing the feature vectors. Figure 3 An example implementation of the neural classification model 214 described below is shown in FIG. However, it should be noted that Figure 3 The specific implementation of the neural classification model 214 shown in is for illustration only, and the neural classification model 214 may be implemented in any other suitable manner.

[0052] The neural classification model 214 uses the recognized one or more languages ​​associated with the audio-based input 202 to control the ASR engine 216, wherein the ASR engine 216 generally operates to convert the audio-based input 202 (or a specific portion thereof) into text. Here, the ASR engine 216 includes or is used in conjunction with multiple ASR models 218 associated with different languages. The ASR engine 216 can use each ASR model 218 to convert at least a portion of the audio-based input 202 into text in a specific language associated with the ASR model 218. Therefore, if the neural classification model 214 determines that a single specific language is used in the audio-based input 202, the neural classification model 214 can direct the audio-based input 202 to the ASR model 218 associated with the specific language. If the neural classification model 214 determines that multiple specific languages ​​are used in the audio-based input 202, the neural classification model 214 can direct different portions of the audio-based input 202 to different ASR models 218 associated with those specific languages. The ASR engine 216 can then use that specific ASR model 218 or those specific ASR models 218 to generate a text-based output 204 for the audio-based input 202.

[0053] In this way, the architecture 200 is able to automatically identify one or more languages ​​being used in the audio-based input 202, and automatically route the audio-based input 202, or portions thereof, to the appropriate ASR model(s) 218. Among other aspects, this allows the same user or different users of the electronic device 101 to interact with the electronic device 101 using different languages. This also allows the same user to switch between different languages ​​in the same audio-based input 202. In other words, the architecture 200 is able to seamlessly switch between different languages ​​with little or no user input (other than one or more users switching between different languages ​​while speaking).

[0054] Note that the features used by the scoring function 208 to support language identification may include at least some of the same features used by the ASR engine 216 to support language identification (meaning identifying a particular word spoken in a particular language). As a result, the language model 210 used by the scoring function 208 may be similar in form to the ASR model 218 used by the ASR engine 216. However, as described above, the language model 210 used by the scoring function 208 may need to use fewer extracted features for performing language identification than the number of extracted features used by the ASR model 218 to perform language identification. As a specific example, in some cases, the ASR engine 216 may use more than a thousand features to perform language identification, while the language model 210 may use a smaller set of features (such as a hundred or a few hundred features) to perform language identification.

[0055] In some embodiments, to support the use of fewer features in the language model 210, a knowledge distillation function 220 is used in or in conjunction with the architecture 200. The knowledge distillation function 220 operates to take a larger training model (such as the ASR model 218) and train a smaller model (such as the associated language model 210) to mimic the behavior of the larger model or the results obtained from the larger model. In other words, the knowledge distillation function 220 can take an ASR model 218 that has been trained for language recognition of a particular language and generate a smaller language model 210 that is trained for language recognition of the same language. This allows the language model 210 to be generated based on the trained ASR model 218. However, note that the language model 210 can be trained in any other suitable manner. The knowledge distillation function 220 can use any suitable technique for generating a smaller model based on a larger training model.

[0056] The ASR models 218 may also be trained in any suitable manner. For example, each ASR model 218 may be trained by providing known words and phrases for a particular language to the feature extractor 206 and training the ASR model 218 associated with that particular language to recognize the known words and phrases. Various methods have been developed for training models to recognize words and phrases for various languages, and additional methods will certainly be developed in the future. This allows additional languages ​​to be easily added to the architecture 200, such as by training an additional ASR model 218 for the new language and then using the knowledge distillation function 220 to generate a language model 210 for the new language.

[0057] can be used in any suitable manner Figure 2, and described above. For example, an example use case of architecture 200 is in an electronic device 101 or other device used in a home, office, or other environment, where different people speak different languages. The electronic device 101 may receive and process speech from various users using different languages, and provide responses based on the recognized text associated with the user speech. As a specific example, the electronic device 101 may receive a question (such as "What's the weather like?") from a first user and provide a response (such as "It's warm and sunny"). The same electronic device 101 may receive the same question from a second user using a different language (such as Korean), and provide the same response to the second user (although the response may also be in Korean). The same electronic device 101 may also receive the same question from a third user using a different language (such as Hindi), and provide the same response to the third user (although the response may also be in Hindi). The user does not need to perform any setup operations or otherwise actively perform operations for changing the language recognized by the electronic device 101.

[0058] although Figure 2 An example of an architecture 200 for multiple speech detection is shown, but may be used for Figure 2 For example, scoring function 208 may include or be used in conjunction with any suitable number of language models 210, and ASR engine 216 may include or be used in conjunction with any suitable number of ASR models 218.

[0059] Figure 3 A more specific example architecture 300 for multi-spoken language detection according to the present disclosure is shown. Specifically, Figure 3 The architecture 300 shown in FIG. Figure 2 . To this end, Figure 2 The various reference numerals used in Figure 3 200 and 300. For ease of explanation, Figure 3 The architecture 300 shown in FIG. 1 can be described as using Figure 1 However, the architecture 300 may be used by any suitable device(s) and may be used in any suitable system(s), such as by the server 106 in the network configuration 100.

[0060] like Figure 3As shown in , an audio-based input 202 is provided to a feature extractor 206, wherein the feature extractor 206 identifies extracted features from the audio-based input 202. The extracted features are provided to a language-specific acoustic model scoring function 208, wherein the language-specific acoustic model scoring function 208 uses a plurality of language models 210 to identify the probability or other likelihood that different portions of the audio-based input 202 are a specific language. The original extracted features from the audio-based input 202 and the probabilities from the scoring function 208 are provided to a feature concatenator 212.

[0061] The feature concatenator 212 combines the original extracted features and probabilities to form a feature vector. In this particular example, the feature concatenator 212 is implemented using a temporal accumulator that accumulates various information generated by the feature extractor 206 and the scoring function 208 over time. In essence, the feature concatenator 212 stacks the information generated by the feature extractor 206 and the scoring function 208 to form a feature vector, which is then further processed by the neural classification model 214.

[0062] exist Figure 3 In the specific example shown in , information is accumulated by the feature connector 212 in rows 302 and columns 304. Each row 302 is associated with data from a different source. Specifically, the first row 302 in this example includes the original extracted features 306 from the feature extractor 206, the second row 302 in this example includes the probabilities 308 calculated using the first language model 210, and the last row 302 in this example includes the probabilities 310 calculated using the last language model 210. Note that additional rows containing probabilities calculated using additional language models 210 can be similarly accumulated by the feature connector 212. Each column 304 is associated with a different time, and it is associated with a different portion of the audio-based input 202. For example, a first column 304 may include the original extracted features 306 and the probabilities 308 and 310 associated with the first portion of the audio-based input 202 (such as the first 20 millisecond segment), the next column 304 may include the original extracted features 306 and the probabilities 308 and 310 associated with the second portion of the audio-based input 202 (such as the second 20 millisecond segment), and so on. The number of columns 304 stacked or accumulated by the feature concatenator 212 for further processing may vary based on a number of factors. In some cases, the columns 304 may span a specified time period, such as one second, two seconds, three seconds, or five seconds.

[0063] Note that it is optional to include the original extracted features 306 from the feature extractor 206 in the accumulated information provided by the feature concatenator 212. In other words, the feature concatenator 212 may concatenate the probabilities 308, 310 from the language model 210 and exclude the original extracted features 306 from the accumulated information. However, in some cases, it has been determined that including the original extracted features 306 in the feature vector processed by the neural classification model 214 may improve the language recognition of the neural classification model 214.

[0064] The feature vector provided by the feature connector 212 is processed by the neural classification model 214, which in this example includes a temporal convolutional neural network (TCNN) 312. The temporal convolutional neural network 312 applies a convolution function to the information provided by the feature connector 212. More specifically, the temporal convolutional neural network 312 applies the convolution function to the information within the window 314, and the window 314 moves or slides so that the temporal convolutional neural network 312 can apply the convolution function to different subsets of the information from the feature connector 212. In this way, the temporal convolutional neural network 312 can apply the convolution function to the original extracted features 306 provided by the feature connector 212 and the probabilities 308 and 310 in an overlapping manner.

[0065] In some cases, window 314 may have a height that matches the combined height of the original extracted features 306 and all probabilities (including probability 308 and probability 310) provided by feature connector 212, in which case window 314 may slide horizontally but not vertically. However, note that other embodiments may be used for window 314. In addition, window 314 may have a width that is much smaller than the combined width of the original extracted features 306 and all probabilities (including probability 308 and probability 310) provided by feature connector 212. In some embodiments, the width of window 314 may be based on the number of features stacked by feature connector 212. As a specific example, if the information from feature connector 212 spans two hundred sample points horizontally, window 314 may have a width of twenty sample points. However, again, note that other embodiments may be used for window 314.

[0066] The convolution output from the temporal convolutional neural network 312 is provided to a batch normalization function 316, wherein the batch normalization function 316 normalizes the convolution output from the temporal convolutional neural network 312. The batch normalization function 316 may use any suitable technique for normalizing the convolution output from the temporal convolutional neural network 312. The normalized convolution output from the temporal convolutional neural network 312 is then provided to another temporal convolutional neural network 318, wherein the another temporal convolutional neural network 318 may operate in the same or similar manner as the temporal convolutional neural network 312. The temporal convolutional neural network 318 may be followed by another batch normalization function. Note that any suitable number of temporal convolutional neural networks and batch normalization functions may be used here. In some embodiments, for example, the neural classification model 214 may include between three and five temporal convolutional neural networks, and each temporal convolutional neural network may be followed by a batch normalization function.

[0067] The pooling layer 320 processes the convolution outputs (possibly normalized) produced by the final temporal convolutional neural network and selects or combines the convolution outputs to input to the activation function of the neural classification model 214. In this example, the pooling layer 320 combines the convolution outputs from the final temporal convolutional neural network by averaging multiple groups of convolution values ​​and outputting the average convolution value. Various temporal convolutional neural networks, batch normalization functions, and pooling layers are used here to model each language supported by the architecture 300, so that these components effectively score the likelihood of each language being used in the audio-based input 202.

[0068] In this example, a softmax function 322 is used to implement the activation function of the neural classification model 214. The softmax function 322 receives the average convolution value from the pooling layer 320 and operates to calculate a final probability 324, wherein the final probability 324 identifies the probability of each language being associated with at least a portion of the audio-based input 202. Here, the final probability 324 can be represented as P(lang i |x t ), where P(·) represents a specific language lang i A specific time x based on the audio input 202 t For each portion of the audio-based input 202 , the probabilities across all languages ​​may be used to identify the ASR model 218 that should be used to process that portion of the audio-based input 202 .

[0069] Note that in some embodiments, the softmax function 322 may also be configured to take into account the amount of silence in the audio-based input 202 when identifying a language associated with the audio-based input 202. For example, in many cases (even during an active conversation), a large percentage of the audio-based input 202 may be associated with silence (or at least no one is actually speaking). In some cases, the amount of silence in the audio-based input 202 may actually exceed 50% of the entire duration of the audio-based input 202. Therefore, the softmax function 322 may be operable to exclude portions of the audio-based input 202 that are associated with silence. In some embodiments, the softmax function 322 may calculate probabilities for each language while excluding silent portions (also referred to as empty portions) of the audio-based input 202 in the following manner:

[0070]

[0071] Here, X represents the length of the audio-based input 202 from time t0 to time tT, and I represents an identity matrix, where the size of the identity matrix depends on the number of languages ​​associated with non-silent time above a threshold (0.5 in this example, although other thresholds may be used here).

[0072] Based on the probabilities output by the softmax function 322, the audio-based input 202 may be directed in its entirety to a single ASR model 218 when a single language is recognized, or different portions of the audio-based input 202 may be directed to different ASR models 218 when multiple languages ​​are recognized. As a result, inputs in different languages ​​may be automatically routed to the appropriate ASR model 218 for processing, and portions of the same input in different languages ​​may be automatically routed to the appropriate ASR model 218 for processing.

[0073] although Figure 3 A more specific example of an architecture 300 for multiple-speech detection is shown, but may be used for Figure 3 Various changes may be made. For example, the scoring function 208 may include or be used in conjunction with any suitable number of language models 210. In addition, the feature connector 212 may temporally accumulate or otherwise combine any suitable number of inputs over any suitable time period. In addition, the neural classification model 214 may be implemented using any other suitable machine learning architecture or other architecture.

[0074] Figure 4 An example architecture 400 for supporting recognition of conversions between multiple languages ​​according to the present disclosure is shown. For ease of explanation, Figure 4 The architecture 400 shown in FIG. 4 can be described as using Figure 1However, the architecture 400 may be used by any suitable device(s) and may be used in any suitable system(s), such as by the server 106 in the network configuration 100.

[0075] As described above, in some cases, such as when a user switches between languages ​​while speaking, the audio-based input 202 may be associated with multiple languages. In these cases, different parts of the same audio-based input 202 are associated with different languages ​​and can be directed to different ASR models 218 for processing. Here, different parts of the audio-based input 202 can be identified as being associated with different languages ​​using the above-described methods. However, for specific parts of the audio-based input 202 that involve conversions between different languages, it may be more difficult to identify the language to be used. Therefore, in some embodiments, additional processing may be performed to identify when the language being used in the audio-based input 202 changes dynamically.

[0076] like Figure 4 As shown in , the architecture 400 includes a deep sub-network decomposition (DSND) algorithm 402. Here, the DSND algorithm 402 can access a large language model 404 associated with each language. In some cases, for example, the language model 404 can be associated with many languages ​​(such as a universal model). The DSND algorithm 402 supports knowledge distillation in order to take the large language model 404 and generate multiple sub-network models, such as sub-network models 406 and 408.

[0077] Each sub-network model 406 and 408 can be trained to recognize jumps between different sets of languages, and the output of each sub-network model can be considered as a new feature vector for language identification. For example, one sub-network model 406 can be trained to recognize when a speaker switches from English to Korean, and another sub-network model 408 can be trained to recognize when a speaker switches from English to Hindi. Note that these are just examples, and the sub-network models can be trained to recognize when a speaker switches between any suitable languages. This can enable a digital personal assistant or other ASR-based system to more accurately detect when a user dynamically switches between languages, which can enable an ASR-based system to more accurately respond to multilingual input. The DSND algorithm 402 supports any suitable knowledge distillation or other techniques for generating sub-network models based on a larger model.

[0078] The various sub-network models may be used in any suitable manner in the above architectures 200, 300 or other architectures designed according to the present disclosure. Figure 2In the architecture 200, a sub-network model may be used after the neural classification model 214 to help improve the language detection function provided by the neural classification model 214. Figure 3 In the architecture 300 of FIG. 2 , the sub-network model may be used after the last temporal convolutional neural network (and optionally after the last batch normalization function) to again help improve the language detection function provided by the neural classification model 214. However, it should be noted that the sub-network model need not be used in either of the architectures 200 , 300 .

[0079] The functionality provided by the sub-network model may be useful when a user speaks a specific language utterance embedded within an utterance in a different language. For example, a user may provide the following utterance as input to the electronic device 101:

[0080] The Hindi version of “How are you doing” is called What is it in Korean?

[0081] In the above example, various sub-network models may be used to identify transitions between English and Indian languages. This may enable the electronic device 101 or other device to more efficiently route the English portion of the utterance to the English-based ASR model 218 and the Indian-based ASR model 218 to the Indian-based ASR model 218. Of course, the above examples are for illustration only, and any other suitable multilingual utterance involving any number of languages ​​may be processed herein.

[0082] although Figure 4 One example of an architecture 400 for supporting recognition of conversions between multiple languages ​​is shown, but may be used for Figure 4 Various changes may be made. For example, architecture 400 may include any other number of sub-network models. In addition, any other suitable technique may be used to identify or process language transitions in input utterances.

[0083] Figure 5 An example method 500 for multi-spoken language detection according to the present disclosure is shown. For ease of explanation, Figure 5 The method 500 shown in FIG. 5 can be described as comprising Figure 1 However, the method 500 may be performed by any suitable device(s) and may be performed in any suitable system(s), such as by the server 106 in the network configuration 100.

[0084] like Figure 5As shown in , input audio data is received in step 502. This may include, for example, the processor 120 of the electronic device 101 receiving an audio-based input 202, wherein the audio-based input 202 represents a digitized and possibly pre-processed version of a user's speech captured by a microphone of the electronic device 101. The user speech may include one or more words spoken in one or more languages, but at this time (multiple) specific words and (multiple) specific languages ​​are unknown. Features of the input audio data are extracted in step 504. This may include, for example, the processor 120 of the electronic device 101 extracting phonemes or phoneme-based features from the audio-based input 202 using a feature extractor 206. Of course, any other features or additional features of the audio-based input 202 may also or may optionally be extracted here.

[0085] In step 506, the possibility of each of the multiple parts of the input audio data being a specific language is determined. This may include, for example, the processor 120 of the electronic device 101 uses the language model 210 to process the extracted features and determine the probability that each part of the audio-based input 202 is spoken in a specific language. Different parts of the audio-based input 202 can represent different segments here, such as segments spanning a given time period (such as 20 milliseconds). However, different parts of the audio-based input 202 can be identified in any other suitable manner. In step 508, the determined probability and the extracted features of the input audio data (optional) are combined to produce a feature vector. This may include, for example, the processor 120 of the electronic device 101 uses the feature connector 212 to accumulate or otherwise combine the identified probabilities 308, 310 from the language model 210 and the original extracted features 306 from the feature extractor 206 (optional) over time.

[0086] At step 510, the language associated with each portion of the input audio data is detected using the feature vector and the neural classification model. This may include, for example, the processor 120 of the electronic device 101 using the neural classification model 214 to determine the final probability that each portion of the audio-based input 202 is associated with a specific language. As a specific example, this may include the processor 120 of the electronic device 101 using one or more temporal convolutional neural networks, one or more batch normalization functions, at least one pooling layer, and at least one activation function to process the identified probabilities 308, 310 and the original extracted features 306 (optional) accumulated over a period of time. Each temporal convolutional neural network may process its input data using a sliding window 314 and apply a convolution function to the input data.

[0087] At step 512, each portion of the input audio data is directed to an ASR model based on the language identified for each portion of the input audio data. This may include, for example, if a single language is associated with the audio-based input 202, the processor 120 of the electronic device 101 causes the audio-based input 202 to be provided to a single ASR model 218. This may also include, if different portions of the audio-based input 202 are associated with different languages, the processor 120 of the electronic device 101 causes these portions of the audio-based input 202 to be provided to different ASR models 218. At step 514, each portion of the input audio data is converted to text or otherwise understood using the associated ASR model. This may include, for example, the processor 120 of the electronic device 101 uses the ASR engine 216 with the appropriate (multiple) ASR model 218 to convert the audio-based input 202 to a text-based output 204.

[0088] although Figure 5 An example of a method 500 for multi-speech detection is shown, but may be used for Figure 5 Make various changes. For example, although Figure 5 The various steps in are shown as a series of steps, but Figure 5 The steps in a process can overlap, occur in parallel, occur in a different order, or occur any number of times.

[0089] It should be noted here that although it is mostly described above that user input is not required to identify one or more languages ​​associated with input audio data, some embodiments of the present disclosure may use some form of user input when identifying one or more languages ​​associated with input audio data. For example, most users may usually only speak a few languages, such as two or three languages ​​(although more is also possible for some users). In some cases, the user may confirm the language that may be spoken to the electronic device 101 or other device, and this information may be used to control which language models 210, ASR models 218 and possible sub-network models 406 and 408 are used to process the input audio data. The user may confirm the language that may be spoken to the device by any suitable means and at any suitable time (such as during the device setting process or by accessing the setting menu). If multiple users may use the electronic device 101 or other device, the language spoken by a specific user may be stored in a user profile for the user, and the user profile may be used to control which language models 210, ASR models 218 and possible sub-network models 406 and 408 are used to process the input audio data when the user exists. Note that any other suitable technique may be used to limit the number of languages ​​analyzed at any particular time, or all supported languages ​​may be used at any particular time (or at all times).

[0090] Although the present disclosure has been described with reference to various exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method for multi-spoken language detection, include: Using at least one processor, performing feature extraction of the input audio data to identify extracted features associated with the input audio data; detecting, using the at least one processor, a language associated with each of a plurality of portions of the input audio data by processing the extracted features using a plurality of language models, wherein each language model is associated with a different language, each language model being used to determine a probability that each portion of the input audio data is from a particular language; and directing, using the at least one processor, each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on a language associated with each portion of the input audio data, The step of detecting the language associated with each portion of the input audio data comprises: concatenating the probabilities determined using the plurality of language models; and Use a neural classification model to process the probability of connections. The steps of processing the probability of connection using the neural classification model include: Processing the probability of the connection using a plurality of temporal convolutional neural networks, wherein an output from at least one of the plurality of temporal convolutional neural networks is normalized; averaging the output or the normalized output from the last temporal convolutional neural network of the plurality of temporal convolutional neural networks; and Based on the averaged output, a probability is determined that a particular language is associated with a particular portion of the input audio data.

2. The method according to claim 1, in: The multiple language models are distilled from the multiple ASR models; and The plurality of language models use a smaller number of features than the plurality of ASR models.

3. The method according to claim 1, in, The step of detecting a language associated with each portion of the input audio data further comprises: applying the probability of associating the particular language with the particular portion of the input audio data to a plurality of sub-network models adapted for conversion between different languages; and Outputs of the plurality of sub-network models are utilized as feature vectors for language classification of the specific portion of the input audio data.

4. The method according to claim 1, further comprising: include: processing each portion of the input audio data using the ASR model to which each portion of the input audio data is directed; and Text for each portion of the input audio data is generated using the ASR model to which each portion of the input audio data is directed.

5. An electronic device for multi-spoken language detection, include: at least one memory; as well as At least one processor operably coupled to the at least one memory and configured to perform the following operations: performing feature extraction on the input audio data to identify extracted features associated with the input audio data; detecting a language associated with each of a plurality of portions of the input audio data by processing the extracted features using a plurality of language models, wherein each language model is associated with a different language, each language model being used to determine a probability that each portion of the input audio data is from a particular language; and directing each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on a language associated with each portion of the input audio data, The at least one processor is configured to detect a language associated with each portion of the input audio data by: concatenating the probabilities determined using the plurality of language models; and Use a neural classification model to process the probability of connections. The at least one processor is configured to process the probability of the connection using the neural classification model by: Processing the probability of the connection using a plurality of temporal convolutional neural networks, and normalizing the output of at least one temporal convolutional neural network from the plurality of temporal convolutional neural networks; averaging the output or the normalized output from the last temporal convolutional neural network of the plurality of temporal convolutional neural networks; and Based on the averaged output, a probability is determined that a particular language is associated with a particular portion of the input audio data.

6. The electronic device as claimed in claim 5, in: The multiple language models are distilled from the multiple ASR models; and The plurality of language models use a smaller number of features than the plurality of ASR models.

7. The electronic device as claimed in claim 5, in, To detect the language associated with each portion of the input audio data, the at least one processor is further configured to: applying the probability of associating the particular language with the particular portion of the input audio data to a plurality of sub-network models adapted for conversion between different languages; and Outputs of the plurality of sub-network models are utilized as feature vectors for language classification of the specific portion of the input audio data.

8. The electronic device as claimed in claim 5, in, The at least one processor is further configured to: processing each portion of the input audio data using the ASR model to which each portion of the input audio data is directed; and Text for each portion of the input audio data is generated using the ASR model to which each portion of the input audio data is directed.

9. A computer readable medium comprising instructions, in, When executed, the instructions cause at least one processor of the electronic device to perform the method of any one of claims 1-4.

Images