Multi-modal verbal language recognition

Abstract

A method includes obtaining an audio input uttered by a person, wherein the audio input is captured by an electronic device. The method further comprises, for each of a plurality of language types, (i) determining a first probability that the person is speaking in a language type by applying a trained spoken language recognition model to the audio input, (ii) determining at least one second probability that the person is speaking in said language type based on at least one characteristic of the person or the electronic device, and (iii) determining a score for the language type based on the first probability and the second probability. The method also includes identifying a language type associated with the highest score as a spoken language of a person in the audio input.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning systems and more particularly to multimodal spoken language recognition. Background Art

[0002] Spoken language recognition refers to the process of identifying the language a speaker is speaking. Spoken language recognition is an important area in speech signal processing and other fields, and has a wide range of applications in many possible fields. For example, the ability to identify the language spoken by a specific person can enable the execution of multiple automated functions based on what that person said. The execution of these automated functions often depends on the ability to successfully understand what was said. Summary of the invention

[0003] Technical Solutions

[0004] According to an embodiment of the present disclosure, a method includes obtaining an audio input of a person speaking, wherein the audio input is captured by an electronic device. The method also includes, for each of a plurality of language types, (i) determining a first probability that the person is speaking in a language type by applying a trained oral language recognition model to the audio input, (ii) determining at least one second probability that the person is speaking in the language type based on at least one characteristic of the person or the electronic device, and (iii) determining a score for the language type based on the first probability and the second probability. The method also includes identifying the language type associated with the highest score as the spoken language of the person in the audio input.

[0005] According to an embodiment of the present disclosure, an apparatus includes at least one processing device configured to obtain an audio input of a person speaking captured by an electronic device. The at least one processing device is also configured to, for each of a plurality of language types, (i) determine a first probability that a person is speaking in a language type by applying a trained oral language recognition model to the audio input, (ii) determine at least one second probability that a person is speaking in the language type based on at least one characteristic of the person or the electronic device, and (iii) determine a score for the language type based on the first probability and the second probability. The at least one processing device is also configured to identify the language type associated with the highest score as the spoken language of the person in the audio input.

[0006] According to an embodiment of the present disclosure, a non-transitory computer-readable medium includes instructions that, when executed, cause at least one processor to obtain an audio input of a person speaking captured by an electronic device. The medium also includes instructions that, when executed, cause at least one processor to: (i) determine a first probability that a person is speaking in a language type by applying a trained oral language recognition model to the audio input, (ii) determine at least one second probability that a person is speaking in the language type based on at least one characteristic of the person or the electronic device, and (iii) determine a score for the language type based on the first probability and the second probability. The medium also includes instructions that, when executed, cause the at least one processor to identify the language type associated with the highest score as the spoken language of the person in the audio input.

[0007] Other technical features may be apparent to those skilled in the art from the following drawings, descriptions and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0009] Figure 1 An example network configuration including electronic devices according to the present disclosure is shown;

[0010] Figure 2A illustrates an example architecture for multimodal spoken language recognition according to the present disclosure;

[0011] Figure 2B is a block diagram of the structure of a server according to an embodiment of the present disclosure;

[0012] Figure 3 illustrates an example method for multimodal spoken language recognition according to the present disclosure; and

[0013] Figure 4 is a flowchart of a method executed by a server according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] The terms "send," "receive," and "communicate," and their derivatives include direct and indirect communications. The terms "include" and "comprise," and their derivatives, mean including, but not limited to, the same. The term "or" is inclusive, meaning and / or. The phrase "associated with," and its derivatives, means include, included within, interconnected with, contain, contained within, connected to or connected with, coupled to or coupled with, communicable with, cooperating with, intertwined, juxtaposed, proximate to, bound to or bound with, having, having the property of, having a relationship with, and the like.

[0015] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or a portion thereof suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable medium" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disk (CD), a digital video disc (DVD) or any other type of memory. "Non-transitory" computer-readable media does not include wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and rewrite data later, such as rewritable optical disks or erasable memory devices.

[0016] As used herein, terms and phrases such as "having", "may have", "include" or "may include" a feature (such as a number, function, operation or component such as a part) indicate the presence of the feature and do not exclude the presence of other features. In addition, as used herein, the phrases "A or B", "at least one of A and / or B" or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B" and "at least one of A or B" may indicate all of the following: (1) including at least one A, (2) including at least one B or (3) including at least one A and at least one B. In addition, as used herein, the terms "first" and "second" may modify various components regardless of their importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate user devices that are different from each other, regardless of the order or importance of the devices. Without departing from the scope of the present disclosure, a first component may be represented as a second component, and vice versa.

[0017] It should be understood that when an element (such as a first element) is referred to as being "coupled" or "connected" or "connected to" another element (such as a second element) (operably or communicatively), it may be coupled or connected / connected to the other element directly or via a third element. Conversely, it should be understood that when an element (such as a first element) is referred to as being "directly coupled" or "directly coupled to" another element (such as a second element) or "directly connected to" another element (such as a second element), there are no other elements (such as a third element) between the element and the other element.

[0018] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for," "capable of," "designed to," "suitable for," "manufactured to," or "capable of," depending on the circumstances. The phrase "configured (or set) to" does not inherently mean "specially designed in hardware to." On the contrary, the phrase "configured to" may indicate that a device may perform an operation together with another device or component. For example, the phrase "a processor configured (or set) to perform A, B, and C" may represent a general-purpose processor (such as a CPU or application processor) that can perform operations by executing one or more software programs stored in a memory device, or a special-purpose processor (such as an embedded processor) for performing operations.

[0019] The terms and phrases used herein are only used to describe embodiments of the present disclosure, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that the singular forms "one", "an" and "the" include plural references unless the context clearly states otherwise. All terms and phrases used herein, including technical and scientific terms and phrases, have the same meaning as the meanings commonly understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It will be further understood that terms and phrases (such as those defined in a common dictionary) should be interpreted as having a meaning consistent with their meanings in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.

[0020] Examples of "electronic devices" according to embodiments of the present disclosure may include smart phones, tablet personal computers (PCs), mobile phones, video phones, e-book readers, desktop PCs, laptop computers, netbook computers, workstations, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, cameras, or wearable devices (such as smart glasses, head-mounted devices (HMDs), electronic clothing, electronic bracelets, electronic necklaces, electronic accessories, electronic tattoos, smart mirrors, personal computers, personal computers, or smart watches). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include at least one of the following: a TV, a digital video disk (DVD) player, an audio player, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, a dryer, an air purifier, a set-top box, a home automation control panel, a security control panel, a TV box (such as Samsung HOMESYNC, APPLETV, or GOOGLE TV), a smart speaker or a speaker with an integrated digital assistant (such as Samsung GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a game console (such as XBOX, PLAYSTATION, or NINTENDO). Electronic dictionary, electronic key, portable camera or electronic photo frame. Other examples of electronic devices include various medical devices (such as various portable medical measurement devices (such as blood sugar measurement devices, heart rate measurement devices or body temperature measurement devices), magnetic resource angiography (MRA) devices, magnetic resource imaging (MRI) devices, computer tomography (CT) devices, imaging devices or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDR), flight data recorders (FDR), car infotainment devices, navigation electronic devices (such as navigation navigation devices or gyrocompasses), avionics, security devices, vehicle head units, industrial or domestic robots, automatic teller machines (ATMs) point of sale (POS) devices or Internet of Things (IoT) devices (such as light bulbs, various sensors, electric meters or gas meters, sprinklers, fire alarms, thermostats, street lights, toasters, fitness equipment, hot water tanks, heaters or boilers). Other examples of electronic devices include at least a part of a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector or various measuring devices (such as devices for measuring water, electricity, gas or electromagnetic waves). Note that according to an embodiment of the present disclosure, the electronic device may be one or a combination of the devices listed above. According to an embodiment of the present disclosure, the electronic device may be a flexible electronic device. The electronic device disclosed herein is not limited to the devices listed above, and may include new electronic devices according to the development of technology.

[0021] In the following description, an electronic device is described with reference to the accompanying drawings according to an embodiment of the present disclosure. As used herein, the term "user" may refer to a person using an electronic device or another device (such as an artificial intelligence electronic device).

[0022] Definitions for certain other words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most, instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0023] Nothing in the description of this application should be interpreted as implying that any particular element, step, or function is an essential element that must be included in the scope of the claims. The scope of the patented subject matter is limited only by the claims. Applicants understand that any other terms, including but not limited to "mechanism", "module", "device", "unit", "component", "element", "member", "device", "machine", "system", "processor" or "controller" are used in the claims to refer to structures known to technicians in the relevant field.

[0024] The following discussion is described with reference to the accompanying drawings. Figures 1 to 3 As well as embodiments of the present disclosure. However, it should be understood that the present disclosure is not limited to the embodiments, and all changes and / or equivalents or replacements thereof also belong to the scope of the present disclosure. Throughout the specification and the drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0025] As mentioned above, spoken language recognition refers to the process of identifying the language that a speaker is speaking. Spoken language recognition is an important area in speech signal processing and other fields, and has a wide range of applications in many possible fields. For example, the ability to recognize the language spoken by a specific person can enable the execution of multiple automated functions based on what the person said. The execution of these automated functions often depends on the ability to successfully understand what was said.

[0026] Some methods simply require the user to identify the language to be spoken by the user, which is often inconvenient for the user and may limit the use of the system. Other methods attempt to automatically determine the spoken language based on the person's speech. However, these existing methods typically rely solely on the person's speech to determine the spoken language. This can be problematic in various situations, such as when the audio data only captures short duration speech or when the spoken language is associated with limited speech data. In these cases, the language recognition system often incorrectly identifies the spoken language, which can result in inappropriate or incorrect processing of the audio data.

[0027] The present disclosure provides various technologies for multimodal oral language recognition. As described in more detail below, an audio input of a person speaking is obtained, wherein the audio input is captured by an electronic device. The electronic device may, for example, represent a mobile electronic device or other electronic device used by a person or another electronic device. For each language type, a first probability that a person is speaking in the language type is determined by applying a trained oral language recognition model to the audio input. In addition, for each language type, at least one second probability that a person is speaking in the language type is determined based on at least one characteristic of the person or the electronic device. A score for the language type is determined based on a weighted sum of the first probability and the second probability for the language type, and the language type associated with the highest score is identified as the person's oral language.

[0028] At least one characteristic may include one or more characteristics of a person or one or more characteristics of an electronic device. In some cases, for example, at least one characteristic may include a location associated with a person, such as a location determined by an electronic device. As a specific example, a location associated with a person or an electronic device may be used to identify statistics of spoken language types in a city, state / province, country, or other area of ​​the location, and the statistics may be used to help identify which language type or types are more likely to be spoken at the location. Additionally or alternatively, at least one characteristic may include a history of spoken language types associated with a person or an electronic device, such as any spoken language types previously identified for a person. Additionally or alternatively, at least one characteristic may include a keyboard language setting or a browser language setting for an electronic device, such as a language selected by a person when installing or otherwise configuring an electronic device or an app on the electronic device.

[0029] In this way, at least one characteristic of a person or electronic device can be used to help personalize the speech recognition function and improve the accuracy of the speech recognition function. When the individual voice data is insufficient, such as when the analyzed audio data is short, using at least one characteristic of a person or electronic device can help improve the accuracy of the speech recognition system. In addition, these technologies provide for the use of multimodal features during speech recognition, which means that these technologies can consider (i) the audio data of the person being processed and (ii) one or more characteristics, wherein at least one characteristic can be independent of the audio data itself. In some cases, this can be implemented using a public oral language recognition model that has been trained to recognize multiple oral language types, without the need to retrain the oral language recognition model for each person. In addition, at least one characteristic of a person or electronic device is usually easily available on the electronic device, so that the consideration of at least one characteristic can be implemented in a direct manner. If any characteristic of a person or electronic device is not available, the speech recognition function can use one or more characteristics that are available, or the speech recognition function can default to using a standard speech recognition process. In addition, these technologies allow for a shallow fusion of the probabilities determined when recognizing the type of language being spoken, which can lead to an extremely lightweight implementation.

[0030] Multimodal spoken language recognition may refer to a system for recognizing each person's spoken language type from multiple forms of interface information. Multimodality may refer to multiple forms of information for recognizing spoken language types.

[0031] For example, multimodality may include not only audio data, but also information unrelated to audio data, such as the location of a person, the location of an electronic device, a history of spoken language types, a keyboard language setting on an electronic device, and a browser language setting on an electronic device. Note that the phrase "metadata" is used in this document to refer to at least one characteristic of a person or electronic device (such as the location of a person or electronic device, the language history of a person or electronic device, the keyboard language setting of an electronic device, or the browser language setting, or any combination thereof) or information based on at least one characteristic of a person or electronic device. Note also that the phrase "language type" is used in this document to distinguish between different languages ​​and to distinguish between different dialects or other variants within a common language. Thus, for example, the described technology may be used with different languages ​​or with different dialects of the same language, such as different dialects used in different countries or different regions of the same country.

[0032] In addition, note that the language recognition functions described in this document can be used independently or as part of any larger application or system. For example, a multilingual speech recognition model represents a machine learning model that is trained to convert spoken words (which can be of multiple language types) into text. A translation application can be used to convert words spoken in one language into text or audio spoken in a different language. A call routing application can be used to automatically route callers to appropriate personnel based on the type of language the caller speaks. A real-time transcription application can be used to automatically transcribe what one or more people are saying in real time. Dictation and memo applications can be used to automatically convert spoken words into text for email, messaging, notes, or other purposes. All of these example applications can rely on accurate recognition of the spoken language in order to correctly perform the functions described therein. Of course, the language recognition functions described in this document can be used in any other suitable manner.

[0033] Figure 1 An example network configuration 100 including electronic devices according to the present disclosure is shown. Figure 1 The embodiment of network configuration 100 shown is for illustration only.Alternative embodiments of network configuration 100 may be used without departing from the scope of the present disclosure.

[0034] According to an embodiment of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In an embodiment of the present disclosure, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 includes circuits for connecting components 120-180 to each other and for transmitting communications (such as control messages and / or data) between components.

[0035] The processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 is capable of controlling at least one of the other components of the electronic device 101 and / or performing communication-related operations or data processing. In an embodiment of the present disclosure, the processor 120 may be a graphics processor unit (GPU). As described below, the processor 120 may be used to perform or support multimodal oral language recognition.

[0036] The memory 130 may include volatile and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or programs 140. The programs 140 include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0037] The kernel 141 can control or manage system resources (such as bus 110, processor 120 or memory 130) for executing operations or functions implemented in other programs (such as middleware 143, API 145 or application 147). The kernel 141 provides an interface that allows the middleware 143, API 145 or application 147 to access the various components of the electronic device 101 to control or manage system resources. The application 147 includes one or more applications related to executing or supporting multimodal oral language recognition. These functions can be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, the middleware 143 can be used as a repeater to allow the API 145 or application 147 to communicate data with the kernel 141. Multiple applications 147 can be provided. The middleware 143 is capable of controlling work requests received from the application 147, such as by assigning a priority to at least one of the multiple applications 147 for using the system resources (such as bus 110, processor 120 or memory 130) of the electronic device 101. The API 145 is an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (such as a command) for archive control, window control, image processing, or text control.

[0038] The I / O interface 150 serves as an interface that can, for example, transmit commands or data input from a user or other external devices to other components of the electronic device 101. The I / O interface 150 can also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0039] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hover input using an electronic pen or a body part of the user.

[0040] For example, the communication interface 170 can establish communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for sending and receiving signals (such as images).

[0041] The electronic device 101 also includes one or more sensors 180, which can measure physical quantities or detect the activation state of the electronic device 101 and convert the measured or detected information into electrical signals. For example, one or more sensors 180 may include one or more cameras or other imaging sensors, which can be used to capture images of scenes. The sensor 180 may also include one or more buttons for touch input, one or more microphones, gesture sensors, gyroscopes or gyroscope sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (such as RGB sensors), biophysical sensors, temperature sensors, humidity sensors, illumination sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalogram (EEG) sensors, electrocardiogram (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors or fingerprint sensors. The sensor 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes and other components. In addition, the sensor 180 may include a control circuit for controlling at least one sensor included here. Any of these sensors 180 may be located in the electronic device 101.

[0042] The first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) in which the electronic device can be installed. When the electronic device 101 is installed in the electronic device 102 (such as an HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without involving a separate network. The electronic device 101 may also be an augmented reality wearable device including one or more cameras, such as glasses.

[0043] Wireless communication can use, for example, long term evolution (LTE), long term evolution-advanced (LTE-A), fifth generation wireless system (5G), millimeter wave or 60 GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) or global system for mobile communications (GSM) as at least one of the cellular communication protocol. Wired connection can include, for example, at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232) or plain old telephone service (POTS). Network 162 includes at least one communication network, such as a computer network (such as a local area network (LAN) or a wide area network (WAN)), the Internet or a telephone network.

[0044] The first external electronic device 102, the second external electronic device 104 and the server 106 may each be a device of the same or different type as the electronic device 101. According to certain embodiments of the present disclosure, the server 106 includes a group of one or more servers. In addition, according to certain embodiments of the present disclosure, all or some operations performed on the electronic device 101 may be performed on another one or more other electronic devices (such as the electronic devices 102 and 104 or the server 106). In addition, according to certain embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform a certain function or service, the electronic device 101 may request another device (such as the electronic devices 102 and 104 or the server 106) to perform at least some functions associated with it, rather than performing the function or service alone or in addition. Other electronic devices (such as the electronic devices 102 and 104 or the server 106) are capable of performing the requested function or additional function and transmitting the execution result to the electronic device 101. The electronic device 101 may provide the requested function or service by processing the received result as is or additionally. To this end, for example, cloud computing, distributed computing or client-server computing technology may be used. Although Figure 1It is illustrated that the electronic device 101 includes the communication interface 170 to communicate with the external electronic device 104 or the server 106 via the network 162 , but according to an embodiment of the present disclosure, the electronic device 101 may independently operate without a separate communication function.

[0045] The server 106 may include the same or similar components (or a suitable subset thereof) as the electronic device 101. The server 106 may support driving the electronic device 101 by performing at least one of the operations (or functions) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101. As described below, the server 106 may be used to perform or support multimodal spoken language recognition.

[0046] although Figure 1 An example of a network configuration 100 including an electronic device 101 is shown, but the Figure 1 Various changes may be made. For example, network configuration 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0047] Figure 2A An example architecture 200 for multimodal spoken language recognition according to the present disclosure is illustrated. For ease of explanation, Figure 2A The architecture 200 is described as involving Figure 1 The network configuration 100 of FIG. 200 uses a server 106 and an electronic device 101. However, the architecture 200 may involve using any other suitable device(s) in any other suitable system(s).

[0048] like Figure 2AAs shown, the electronic device 101 provides a request 202 to the server 106. The request 202 represents a language identification (LID) request, which represents a request for the server 106 to identify the type of language spoken by at least one person (such as a user of the electronic device 101). The LID request may include any suitable content, such as audio data captured by the electronic device 101 and any metadata available at the electronic device 101. The audio data may, for example, include audio data captured by at least one microphone (at least one sensor 180) of the electronic device 101. As described above, at least some of the metadata represent information unrelated to the audio data captured by the electronic device 101. The metadata includes one or more characteristics of the person or electronic device 101 who is speaking, such as the location of the person or electronic device 101, the language history of the person or electronic device 101, the keyboard language setting or the browser language setting of the electronic device 101, or a combination thereof.

[0049] In this example, the server 106 provides a LID service (language ID service) 204, which can receive a request 202 from the electronic device 101 and provide one or more recognized language types to the electronic device 101 based on the request 202. In this example, the LID service 204 can route different metadata from the request 202 to different services 206, 208, 210, wherein the different services 206, 208, 210 can access different information databases 212, 214, 216 to provide assistance based on language recognition.

[0050] like Figure 2A As shown, the LID service 204 can provide the real-time location associated with the person or electronic device 101 to the location service 206, such as the location of the electronic device 101 determined by the positioning system of the electronic device 101. The location service 206 uses the location of the person or electronic device 101 to estimate which language type or language types the person may speak based on the location. As a specific example, the location service 206 can be used to access the geographic language database 212, which can store statistics or other information about the types of spoken languages ​​in different cities, states / provinces, countries, or other regions. Thus, for example, the geographic language database 212 can indicate that each area in different regions has a certain percentage of people who speak the first language type, another percentage of people who speak the second language type, another percentage of people who speak the third language type, etc. The location service 206 can use the real-time location of the person or electronic device 101 to identify the statistics of the language types used in the area including the real-time location. The location service 206 can return metadata that includes the identification of each of these language types and other information, such as the associated percentage (probability) of people who speak the language type in the area including the real-time location.

[0051] The LID service 204 may also or alternatively provide the identification of the person or electronic device 101 to the history service 208. The history service 208 uses the identification of the person or electronic device 101 to identify the history of the oral language type previously associated with the person or electronic device 101 (such as any oral language type previously identified for the person or electronic device 101). The history of the oral language type may also identify the frequency with which different language types have been previously identified for the person or electronic device 101. The language history here may provide the probability of different language types being spoken based on the previous history of the person or electronic device. In an embodiment of the present disclosure, a model that has been trained based on Mel frequency cepstral coefficients, linear prediction cepstral coefficients (representing vocal cord movement), residual wavelet octave coefficients (representing pitch, harmonic structure and spectral energy distribution) or a combination thereof is used to model the language history associated with a specific person or electronic device 101. The model here may be trained to identify the language type being spoken based on these characteristics in the audio data, and the model may return different probabilities that the person is speaking different language types. The model or other information related to the history may be stored in the history database 214 for use by the history service 208. History service 208 can return metadata including one or more probabilities of one or more language types based on the history of a person or electronic device.

[0052] The LID service 204 may also or alternatively provide identification of a person or electronic device 101 to the device setting service 210. The device setting service 210 uses one or more language settings for a keyboard, browser, or other application or other feature of the electronic device 101 in order to identify one or more language types that have been selected for use. Depending on the implementation and environment, in some cases, the user of the electronic device 101 may select a single language type to be used with the keyboard, browser, or other feature of the electronic device 101. In other cases, the user of the electronic device 101 may select multiple language types to be used with the keyboard, browser, or other feature of the electronic device 101, such as when the user can switch between using different virtual keyboards associated with different language types. Therefore, at least one setting of the electronic device 101 may be used to estimate one or more probabilities that the user of the electronic device 101 is using one or more language types. In an embodiment of the present disclosure, the settings associated with the user's electronic device may be identified and stored in the settings database 216 for use by the device setting service 210. The device setting service 210 may return metadata including one or more probabilities of one or more language types based on the device settings.

[0053] At least some of the information associated with the request 202 is also provided by the LID service 204 to a LID engine (language ID engine) 218. The LID engine 218 includes a spoken language recognition model 220, which represents a machine learning model that has been trained to process audio data and identify the type of language spoken in the audio data. The spoken language recognition model 220 can be trained to recognize multiple language types, thereby allowing the same machine learning model to process multiple requests 202 involving multiple language types. In an embodiment of the present disclosure, the spoken language recognition model 220 can generate multiple probabilities as output for each request 202, wherein each probability is associated with a different language type. Thus, for example, when the spoken language recognition model 220 is more certain about the language recognition, the spoken language recognition model 220 can generate a higher probability for one language type than another language type. When the spoken language recognition model 220 is less certain about the language recognition, the spoken language recognition model 220 can also generate similar probabilities for two or more language types.

[0054] To help improve the accuracy of language identification herein, the LID service 204, LID engine 218, or other component may combine the probability generated by the spoken language identification model 220 for the request 202 with one or more probabilities generated by one or more of the services 206, 208, 210. This may result in a final probability or score identifying the likelihood that different language types are associated with the audio data of the request 202, and the highest score representing one or more language types may be selected to identify the language type most likely associated with the audio data of the request 202.

[0055] In an embodiment of the present disclosure, for each request 202, a score for each of the one or more language types may be determined as a weighted combination of the probabilities determined by the spoken language recognition model 220 and the services 206, 208, 210. As a specific example, the weighted combination for each language type may be determined as follows.

[0056] y * = argmax log P(y|x)+W loc P loc (y)+W speech P speech (y)+W text P text (y).

[0057] Here, y* represents the final probability or score of the associated language type. In addition, P(y|x) represents the probability of the associated language type determined by the spoken language recognition model 220, which means that the probability is based on the acoustic features of the audio data associated with the request 202. The probability of the associated language type determined by the spoken language recognition model 220 can be referred to as a first probability. In addition, P loc (y) represents the probability of the associated language type based on the location associated with the person or electronic device 101 determined by the LID service 204. In some cases, P loc (y) may be based on the percentage of people who speak the associated language type in a given area where the person or electronic device 101 is located. speech (y) represents the probability of the associated language type based on the history of spoken language types associated with the person or electronic device 101 determined by the location service 206, and P text (y) represents the probability of the associated language type based on the keyboard language setting or browser language setting associated with the electronic device 101 as determined by the history service 208. loc ,W speech and W text y represents the weights associated with the probabilities determined by the services 206, 208, 210. These weights may be set in any suitable manner, such as when establishing or adjusting weights during hyperparameter tuning.

[0058] Notice, Figure 2A The three specific characteristics of the person or electronic device 101 shown in are for illustration only and may vary depending on the implementation. For example, some implementations may include one or two of these specific characteristics. In addition, other implementations may include other or additional characteristics of the person or electronic device 101. It should also be noted that even if one or more characteristics of the person or electronic device 101 are supported, the language recognition function here can still operate even when the characteristics of the person or electronic device 101 are not available. For example, the request 202 may lack the location of the person or electronic device 101, or there may be no available information about the language history of the person or electronic device or the settings of the electronic device. When a subset of services 206, 208, 210 can provide metadata about the request 202, the score for each language type can be based on the probability determined by the oral language recognition model 220 and one or more probabilities determined by the subset of services 206, 208, 210. When no service 206, 208, 210 can provide metadata about the request 202, the score for each language type can be based only on the probability determined by the oral language recognition model 220.

[0059] The one or more language types identified as being most likely associated with the request 202 may be used in any suitable manner. For example, for each request 202, the LID service 204 may provide a response 222 that identifies a specific language type identified as being most likely associated with the corresponding request 202. The electronic device 101 may use the identified language type in any suitable manner, such as invoking a specific machine learning model for converting speech of a specific language type to text. However, note that the electronic device 101, the server 106, or another component may use the one or more language types identified as being most likely associated with each request 202 in any other suitable manner. Various example applications (such as multilingual speech recognition models, translation applications, call routing applications, real-time transcription applications, and dictation and memo applications) may be supported by the electronic device 101, the server 106, or other components, and may operate using language recognition provided by the LID service 204.

[0060] In this particular example, the language recognition functionality is provided by the server 106 based on input from the client device (i.e., the electronic device 101). This approach allows the server 106 to provide personalized spoken language recognition services for multiple people who speak different language types using the same trained spoken language recognition model 220. As a result, the server 106 can receive requests 202 from different client devices and provide responses 222 to the client devices, even when the client devices provide audio data in different language types. However, other implementations are also possible, such as when the language recognition functionality is provided "on the device", such as entirely on the client device.

[0061] It should be noted that Figure 2A shown in or about Figure 2A The described functions may be implemented in the electronic device 101, the server 106 or other devices in any suitable manner. For example, in the embodiments of the present disclosure, one or more software applications or other software instructions executed by the processor 120 of the electronic device 101, the server 106 and / or other devices may be used to implement or support Figure 2A shown in or about Figure 2A In the embodiments of the present disclosure, dedicated hardware components may be used to implement or support at least some of the functions described. Figure 2A shown in or about Figure 2A In general, any suitable hardware or any suitable combination of hardware and software / firmware instructions may be used to implement at least some of the functions described in Figure 2A shown in or about Figure 2A In addition, Figure 2A shown in or about Figure 2A The functions described may be performed by a single device or by multiple devices.

[0062] although Figure 2A An example of an architecture 200 for multimodal spoken language recognition is shown, but may be used for Figure 2A For example, the architecture 200 may be added, omitted, combined, further subdivided, duplicated, or placed in any other suitable configuration according to specific needs. Figure 2A In addition, any other or additional service or services may be used to provide metadata associated with request 202 for use in weighted combination. In addition, the above-described language identification functionality may be implemented in any other suitable manner, such as when implemented in a manner other than a service.

[0063] Figure 2B is a block diagram of a structure of a server according to an embodiment of the present disclosure.

[0064] The server 106 according to an embodiment of the present disclosure may include a processor 250 , a communication interface 260 , and a memory 270 .

[0065] The communication interface 260 may include one or more components configured to allow the server 106 to communicate with the electronic device 101. The communication interface 260 may receive the request 202 from the electronic device 101, and may provide the electronic device 101 with one or more recognized language types based on the request 202.

[0066] The memory 270 may store at least one instruction and at least one program for the processor 250 to perform processing and control, and may store data input to or output from the server 106. The memory 270 may include at least one of a memory for temporarily storing data (such as a random access memory (RAM) or a static RAM (SRAM)) and a data storage device for non-temporarily storing data (such as a flash memory type storage medium, a hard disk type storage medium, a multimedia card micro storage medium, a card type memory (e.g., an SD or XD memory), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable RAM (PROM), a magnetic memory, a magnetic disk, or an optical disk).

[0067] The memory 270 includes a memory for the electronic device 101 to execute the above reference Figures 1 to 2A The memory 270 may include the LID service 204, the LID engine 218, and the spoken language recognition model 220. The memory 270 may include the location service 206, the history service 208, and the device settings service 210. The memory 270 may include the geo-language database 212, the history database 214, and the settings database 216.

[0068] The processor 250 may control the overall operation of the server 106. For example, the processor 250 may execute a program stored in the memory 270 of the server 106 to control the overall operation of the communication interface 260. The processor 250 may include at least one general-purpose processor (such as a central processing unit (CPU), an application processor (AP), or a digital signal processor (DSP)), a dedicated graphics processor (such as a graphics processing unit (GPU) or a visual processing unit (VPU)), or a dedicated artificial intelligence processor (such as a neural processing unit (NPU)). The processor 250 may include at least one processor designed to perform a function of identifying a language type associated with a person's spoken language in an audio input.

[0069] Processor 250 can use the location of a person or electronic device 101 to estimate which language type or languages ​​a person may speak based on the location by executing location service 206 stored in memory 270. Processor 250 can access geographic language database 212 by executing location service 206 stored in memory 270.

[0070] The processor 250 may use the identification of the person or electronic device 101 to identify a history of spoken language types previously associated with the person or electronic device 101 by executing the history service 208 stored in the memory 270, such as any spoken language types previously identified for the person or electronic device 101. The processor 250 may access the history database 214 by executing the history service 208 stored in the memory 270.

[0071] The processor 250 can use one or more language settings for a keyboard, browser, or other application or other feature of the electronic device 101 by executing the device settings service 210 stored in the memory 270 to identify the one or more language types that have been selected for use. The processor 250 can access the settings database 216 by executing the history service 208 stored in the memory 270.

[0072] According to an embodiment of the present disclosure, the server 106 includes at least one processor 250 configured to obtain an audio input of a person speaking captured by an electronic device, determine a first probability that the person is speaking in a language type by applying the trained spoken language recognition model 220 to the audio input, determine at least one second probability that the person is speaking in the language type based on at least one characteristic of the person or the electronic device, determine a score for the language type based on the first probability and the second probability, and identify the language type associated with the highest score as the spoken language of the person in the audio input.

[0073] According to the embodiments of the present disclosure, the accuracy of identifying the language type of input audio can be improved.

[0074] At least one second probability may include at least one of: a probability that the person speaks in the language type based on a location associated with the person or electronic device, a probability that the person speaks in the language type based on a history of spoken language types associated with the person or electronic device, or a probability that the person speaks in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

[0075] At least one second probability may include a probability that the person speaks in the language type based on a location associated with the person or electronic device, a probability that the person speaks in the language type based on a history of spoken language types associated with the person or electronic device, and a probability that the person speaks in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

[0076] The scores for the language types are based on the following formula:

[0077] y * = argmax log P(y|x)+W loc P loc (y)+W speech P speech (y)+W text P text (y).

[0078] Where y* represents the score, P(y|x) represents the first probability, and P loc (y) represents the probability of the location associated with the person, P speech (y) represents the probability based on the history of spoken language types associated with the person, P text (y) represents a probability based on a keyboard language setting or a browser language setting associated with the electronic device, and W loc ,W speech and W text Represents weight.

[0079] The history of spoken language types associated with the person includes a model trained using at least one of mel-frequency cepstral coefficients, linear prediction cepstral coefficients representing vocal cord motion, or wavelet octave coefficients representing residuals of pitch, harmonic structure, and spectral energy distribution.

[0080] The at least one processor 250 processing device is configured to obtain audio input from an electronic device.

[0081] At least one processor 250 is configured to provide personalized spoken language recognition services for a plurality of persons speaking different language types using the trained spoken language recognition model.

[0082] Figure 3An example method 300 for multimodal spoken language recognition according to the present disclosure is illustrated. For ease of explanation, Figure 3 The method 300 is described as involving Figure 1 The method 300 uses the server 106 and the electronic device 101 in the network configuration 100. However, the method 300 may involve using any other suitable device(s) in any other suitable system(s).

[0083] like Figure 3 As shown, at step 302, a request to identify at least one language type being spoken is received. This can include, for example, the processor 250 of the server 106 receiving the request 202 containing audio data of a person speaking captured by the electronic device 101. The request 202 can also include metadata associated with the person or the electronic device 101. At step 304, a probability that the person is speaking in different language types is generated based on the audio data. This can include, for example, the processor 250 of the server 106 providing the audio data to the trained spoken language recognition model 220, which can generate the probability that the audio data contains speech of different language types.

[0084] At step 306, determine whether to use location to assist language recognition. This can include, for example, the processor 250 of the server 106 determines whether the real-time location of the person or electronic device 101 who is speaking is available, such as whether the real-time location is included in the request 202. If so, then generate the probability that the person is speaking in different language types based on the real-time location at step 308. This can include, for example, the processor 250 of the server 106 uses the location service 206 to access the geographic language database 212 and identify statistics or other information about the spoken language types in the city, state / province, country or other region, which includes the real-time location of the person or electronic device 101 who is speaking. This can also include the location service 206 returning metadata that identifies one or more language types and the probability of each language type. The probability of each language type can represent or be based on statistics or other information from the geographic language database 212.

[0085] At step 310, determine whether to use language type history to assist language recognition. This can include, for example, the processor 250 of the server 106 determines whether any previous language type is associated with the person or electronic device 101 who is speaking. If so, then generate the probability that the person is speaking in different language types based on the language type history at step 312. This can include, for example, the processor 250 of the server 106 uses the history service 208 to identify any previous language types associated with the person or electronic device 101 who is speaking and the probability that the request 202 is associated with each of these previous language types. As a specific example, this can include the processor 250 of the server 106 using a model that has been trained based on Mel frequency cepstral coefficients, linear prediction cepstral coefficients, residual wavelet octave coefficients, or a combination thereof, wherein the model can generate different probabilities that the person is speaking in different language types. This can also include the history service 208 returning metadata that identifies one or more language types and the probability of each language type.

[0086] At step 314, determine whether to use at least one device setting to assist language identification. This can include, for example, the processor 250 of the server 106 determining whether any language setting associated with the electronic device 101 is available. If so, then at step 316, generate a probability that the person is speaking in a different language type based on the at least one device setting. This can include, for example, the processor 250 of the server 106 using the device setting service 210 to identify whether the electronic device 101 is associated with one or more keyboard language settings, browser language settings, or other language settings associated with one or more language types. This can also include the device setting service 210 returning metadata identifying the one or more language types and the probability of each language type.

[0087] At step 318, a score of the language type is generated based on the probability. This can include, for example, the processor 250 of the server 106 generating a score as a weighted combination of the probability from the spoken language recognition model 220 and the probability from the services 206, 208, 210. At step 320, at least one language type associated with the request is identified. This can include, for example, the processor 250 of the server 106 identifying the language type associated with the highest score and using it as the identification of the spoken language type in the audio data. At step 322, the identified language type can be stored, output or used in some way. This can include, for example, the processor 250 of the server 106 providing a response 222 identifying the language type associated with the highest score to the electronic device 101. This allows the electronic device 101 to display the identified language type or otherwise use the identified language type, such as calling one or more functions based on the spoken language type. This can also or alternatively include the processor 250 of the server 106 or another component that uses the identified language type to call one or more functions based on the spoken language type.

[0088] although Figure 3 One example of a method 300 for multimodal spoken language recognition is shown, but may be used for Figure 3 For example, although shown as a series of steps, Figure 3 The various steps in may overlap, occur in parallel, occur in a different order, or occur any number of times. Figure 4 is a flowchart of a method executed by a server according to an embodiment of the present disclosure.

[0089] Figure 4 An example method 400 for multimodal spoken language recognition according to the present disclosure is illustrated. For ease of explanation, Figure 4 The method 400 is described as involving Figure 1 The method 400 uses the server 106 and the electronic device 101 in the network configuration 100. However, the method 400 may involve using any other suitable device(s) in any other suitable system(s).

[0090] Referring to operation 410, the server 106 may obtain an audio input of a person speaking. The audio input may be captured by the electronic device 101. The server 106 may receive the request 202 from the electronic device 101 and provide the electronic device 101 with one or more recognized language types based on the request 202.

[0091] Referring to operation 420 , the server 106 may determine a first probability that a person is speaking in a language type by applying the trained spoken language recognition model 220 to the audio input.

[0092] Referring to operation 430 , the server 106 may determine at least one second probability that the person is speaking in the language type based on at least one characteristic of the person or the electronic device.

[0093] At least one second probability may include at least one of: a probability that the person speaks in the language type based on a location associated with the person or electronic device, a probability that the person speaks in the language type based on a history of spoken language types associated with the person or electronic device, or a probability that the person speaks in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

[0094] At least one second probability may include a probability that the person speaks in the language type based on a location associated with the person or electronic device, a probability that the person speaks in the language type based on a history of spoken language types associated with the person or electronic device, and a probability that the person speaks in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

[0095] Referring to operation 440, the server 106 may determine the score of the language type based on the first probability and the second probability. For example, the server 106 determines the score of the language type based on a weighted sum of the first probability and the second probability.

[0096] The scores for the language types are based on the following formula:

[0097] y * = argmax log P(y|x)+W loc P loc (y)+W speech P speech (y)+W text P text (y).

[0098] Where y* represents the score, P(y|x) represents the first probability, and P loc (y) represents the probability of the location associated with the person, P speech (y) represents the probability based on the history of spoken language types associated with the person, P text (y) represents a probability based on a keyboard language setting or a browser language setting associated with the electronic device, and W loc ,W speech and W text Represents weight.

[0099] The history of spoken language types associated with the person includes a model trained using at least one of mel-frequency cepstral coefficients, linear prediction cepstral coefficients representing vocal cord motion, or wavelet octave coefficients representing residuals of pitch, harmonic structure, and spectral energy distribution.

[0100] Referring to operation 450 , the server 106 may identify the language type associated with the highest score as the spoken language of the person in the audio input.

[0101] The server 106 may obtain the audio input from the electronic device at the second electronic device, and the second electronic device identifies the language type associated with the highest score. According to an embodiment of the present disclosure, the accuracy of identifying what language type the input audio is can be improved.

[0102] Although the present disclosure has been described with reference to various exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method comprising: Obtaining audio input of a person speaking, wherein the audio input is captured by an electronic device (101); For each of the multiple language types: determining a first probability that the person is speaking in a language type by applying a trained spoken language recognition model (220) to the audio input; determining at least one second probability that the person is speaking in the language type based on at least one characteristic of the person or the electronic device; and determining a score for the language type based on the first probability and the second probability; and The language type associated with the highest score is identified as a spoken language of the person in the audio input.

2. The method according to claim 1, wherein: For each of the language types, the at least one second probability comprises at least one of: a probability that the person is speaking in the language type based on a location associated with the person or the electronic device; a probability that the person is speaking in the language type based on a history of spoken language types associated with the person or the electronic device; or A probability that the person is speaking in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

3. The method according to claim 1 or 2, wherein: For each of the language types, the at least one second probability comprises: a probability that the person is speaking in the language type based on a location associated with the person or the electronic device; a probability that the person is speaking in the language type based on a history of spoken language types associated with the person or the electronic device; and A probability that the person is speaking in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

4. The method according to any one of claims 1 to 3, wherein: For each of the language types, determining a score for the language type includes using the following formula: y * =argmaxlog P(y|x)+ W loc P loc (y) + W speech P speech (y) + W text P text (y), where: y* represents the score; P(y|x) represents the first probability; P loc (y) represents a probability based on the location associated with the person; P speech (y) represents a probability based on a history of spoken language types associated with the person; P text (y) indicating a probability based on the keyboard language setting or the browser language setting associated with the electronic device; and W loc ,W speech and W text Represents weight.

5. The method according to any one of claims 1 to 4, wherein: The history of the spoken language types associated with the person includes a model trained using at least one of: Mel-frequency cepstral coefficients; Linear prediction cepstral coefficients representing vocal fold motion; or Wavelet octave coefficients of the residual representing the pitch, harmonic structure, and spectral energy distribution.

6. The method according to any one of claims 1 to 5, wherein: Obtaining the audio input includes obtaining, at a second electronic device, the audio input from the electronic device, the second electronic device identifying a language type associated with the highest score.

7. The method according to claim 6, wherein: The second electronic device uses the trained spoken language recognition model to provide personalized spoken language recognition services for multiple people who speak different language types.

8. An apparatus comprising: At least one processor (250) is configured to: Obtaining audio input of a person speaking captured by the electronic device 101; For each of the multiple language types: determining a first probability that the person is speaking in a language type by applying a trained spoken language recognition model (220) to the audio input; determining at least one second probability that the person is speaking in the language type based on at least one characteristic of the person or the electronic device; and determining a score for the language type based on the first probability and the second probability; as well as The language type associated with the highest score is identified as the spoken language of the person in the audio input.

9. The device according to claim 8, wherein: For each of the language types, the at least one second probability comprises at least one of: a probability that the person is speaking in the language type based on a location associated with the person or the electronic device; a probability that the person is speaking in the language type based on a history of spoken language types associated with the person or the electronic device; or A probability that the person is speaking in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

10. The device according to claim 8 or 9, wherein: For each of the language types, the at least one second probability comprises: a probability that the person is speaking in the language type based on a location associated with the person or the electronic device; a probability that the person is speaking in the language type based on a history of spoken language types associated with the person or the electronic device; and A probability that the person is speaking in the language type based on a keyboard language setting or a browser language setting associated with the electronic device.

11. The device according to any one of claims 8 to 10, wherein: For each of the language types, the score for the language type is based on the following formula: y * =argmax log P(y|x)+ W loc P loc (y)+W speech P speech (y)+W text P text (y), in: y* represents a fraction; P(y|x) represents the first probability; P loc (y) represents a probability based on the location associated with the person; P speech (y) represents a probability based on a history of spoken language types associated with the person; P text (y) indicating a probability based on the keyboard language setting or the browser language setting associated with the electronic device; and W loc ,W speech and W text Represents weight.

12. The device according to any one of claims 8 to 11, wherein: The history of the spoken language types associated with the person includes a model trained using at least one of: Mel-frequency cepstral coefficients; Linear prediction cepstral coefficients representing vocal fold motion; or Wavelet octave coefficients of the residual representing the pitch, harmonic structure, and spectral energy distribution.

13. The device according to any one of claims 8 to 12, wherein: The at least one processor (250) is configured to obtain the audio input from the electronic device.

14. The device according to any one of claims 8 to 13, wherein: The at least one processor (250) is configured to use the trained spoken language recognition model to provide personalized spoken language recognition services for a plurality of persons speaking different language types.

15. A non-transitory computer readable medium comprising instructions which, when executed, cause at least one processor (250): Obtaining audio input of a person speaking captured by an electronic device (101); For each of the multiple language types: determining a first probability that the person is speaking in a language type by applying a trained spoken language recognition model (220) to the audio input; determining at least one second probability that the person is speaking in the language type based on at least one characteristic of the person or the electronic device; and determining a score for the language type based on the first probability and the second probability; as well as The language type associated with the highest score is identified as a spoken language of a person in the audio input.

Images