Apparatus and method for speech verification of voice assistant

Abstract

A method includes obtaining audio data and identifying an utterance of a wake-up word or a wake-up phrase in the audio data. The method further includes generating an embedded vector based on utterances from the audio data, and accessing a previously generated set of vectors representing previous utterances of a wake-up word or wake-up phrase. The method further includes performing a clustering on the embedded vector and a previously generated set of vectors to identify a cluster including the embedded vector, wherein the identified cluster is associated with the speaker. The method further includes updating a speaker vector associated with the speaker based on the embedded vector, and determining a similarity score between the updated speaker vector and the embedded vector using a speaker verification model. Further, the method includes determining whether a speaker providing the utterance matches a speaker associated with the identified cluster based on the similarity score.

Description

Technical Field

[0001] The present disclosure generally relates to machine learning systems. More specifically, the present disclosure relates to a system and method for speaker verification for a voice assistant. Background Art

[0002] Voice assistants such as BIXBY, SIRI, and ALEXA allow users to perform voice registration as a step to collect user voice profiles. This involves requesting users to record user voice samples, which can be a cumbersome task and a rigid barrier to integrating new users. Therefore, in order to improve user experience and new user registration completion rate, the recent trend is to not require users to perform voice registration. Although this eliminates the cumbersome registration task, it may lead to a degradation of voice wake-up performance and increase the number of invalid wake-ups, which refers to instances where the voice assistant is activated despite the user not having requested or intended the voice assistant to do so. Summary of the invention

[0003] Technical Solutions

[0004] The present disclosure relates to a system and method for speaker verification for a voice assistant.

[0005] An embodiment of the present disclosure may provide a method. The method includes obtaining audio data using at least one processing device of an electronic device. The method also includes identifying the utterance of a wake-up word or a wake-up phrase in the audio data using at least one processing device. The method also includes generating an embedding vector based on the utterance from the audio data using at least one processing device. The method also includes accessing a previously generated vector set representing a previous utterance of a wake-up word or a wake-up phrase using at least one processing device. The method also includes performing clustering on the embedding vector and the previously generated vector set using at least one processing device to identify a cluster including the embedding vector, wherein the identified cluster is associated with a speaker. The method also includes updating a speaker vector associated with the speaker based on the embedding vector using at least one processing device. The method also includes determining a similarity score between the updated speaker vector and the embedding vector using at least one processing device and a speaker verification model. In addition, the method includes determining whether the speaker providing the utterance matches the speaker associated with the identified cluster using at least one processing device and based on the similarity score.

[0006] Embodiments of the present disclosure may provide a device. The device includes at least one processing device configured to obtain audio data. At least one processing device is also configured to identify the speech of a wake-up word or a wake-up phrase in the audio data. At least one processing device is also configured to generate an embedding vector based on the speech from the audio data. The at least one processing device is also configured to access a previously generated vector set representing a previous speech of the wake-up word or the wake-up phrase. At least one processing device is also configured to perform clustering on the embedding vector and the previously generated vector set to identify a cluster including the embedding vector, wherein the identified cluster is associated with a speaker. At least one processing device is also configured to update a speaker vector associated with the speaker based on the embedding vector. At least one processing device is also configured to use a speaker verification model to determine a similarity score between the updated speaker vector and the embedding vector. In addition, at least one processing device is configured to determine whether the speaker providing the speech matches the speaker associated with the identified cluster based on the similarity score.

[0007] Embodiments of the present disclosure may provide a computer-readable medium. The computer-readable medium contains instructions that, when executed, cause at least one processor to obtain audio data. The medium also contains instructions that, when executed, cause at least one processor to identify the speech of a wake-up word or wake-up phrase in the audio data. The medium also contains instructions that, when executed, cause at least one processor to generate an embedding vector based on the speech from the audio data. The medium also contains instructions that, when executed, cause at least one processor to access a previously generated vector set representing a previous speech of a wake-up word or wake-up phrase. The medium also contains instructions that, when executed, cause at least one processor to perform clustering on the embedding vector and the previously generated vector set to identify a cluster including the embedding vector, wherein the identified cluster is associated with a speaker. The medium also contains instructions that, when executed, cause at least one processor to update a speaker vector associated with a speaker based on the embedding vector. The medium also contains instructions that, when executed, cause at least one processor to use a speaker verification model to determine a similarity score between the updated speaker vector and the embedding vector. Additionally, the medium includes instructions that, when executed, cause at least one processor to determine, based on the similarity score, whether a speaker providing the utterance matches a speaker associated with the identified cluster.

[0008] An embodiment of the present disclosure may provide a method. The method includes training a speaker verification model using at least one processing device of an electronic device. The training includes providing a sample embedding vector based on an utterance from audio data, wherein the utterance includes a wake-up word or a wake-up phrase. The training also includes providing a sample speaker vector using at least one processing device. The training also includes receiving a result including a similarity score between the sample speaker vector and the sample embedding vector from the speaker verification model using at least one processing device. In addition, the training includes using a loss function based on the result and modifying the speaker verification model.

[0009] Other technical features may be apparent to those skilled in the art from the following drawings, descriptions and claims.

[0010] Before proceeding to the following detailed description, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms "send," "receive," and "communicate," and their derivatives, include direct and indirect communications. The terms "include" and "comprises," and their derivatives, are meant to include, but are not limited to, this. The term "or" is inclusive, meaning and / or. The phrase "associated with," and its derivatives, means to include, be included within, be interconnected with, include, be contained within, be connected to or connected with, be coupled to or coupled with, be communicable with, collaborate with, be intertwined, be juxtaposed, be close to, be bound to or bound with, have, have the property of, have a relationship with, etc.

[0011] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and embodied in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data or a part thereof suitable for implementation in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code, including source code, object code and executable code. The phrase "computer-readable program code" includes any type of medium that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disc (CD), a digital video disc (DVD) or any other type of memory. "Non-transitory" computer-readable media does not include wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and rewrite data later, such as rewritable optical discs or erasable memory devices.

[0012] As used herein, terms and phrases such as "having", "may have", "include" or "may include" a feature (such as a number, function, operation or component such as a part) indicate the presence of the feature and do not exclude the presence of other features. In addition, as used herein, the phrases "A or B", "at least one of A and / or B" or "one or more of A and / or B" may include all possible combinations of A and B. For example, "A or B", "at least one of A and B" and "at least one of A or B" may indicate all of the following: (1) including at least one A, (2) including at least one B or (3) including at least one A and at least one B. In addition, as used herein, the terms "first" and "second" may modify various components regardless of their importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate user devices that are different from each other, regardless of the order or importance of the devices. Without departing from the scope of the present disclosure, a first component may be represented as a second component, and vice versa.

[0013] It should be understood that when an element (such as a first element) is referred to as being (operably or communicatively) “coupled” or “connected” to another element (such as a second element), it may be coupled or connected to the other element directly or via a third element. Conversely, it should be understood that when an element (such as a first element) is referred to as being “directly coupled” or “directly coupled” or “connected” to another element (such as a second element), no other element (such as a third element) is interposed between the element and the other element.

[0014] As used herein, the phrase "configured (or set) to" may be used interchangeably with the phrases "suitable for," "capable of," "designed to," "suitable for," "manufactured to," or "capable of," depending on the circumstances. The phrase "configured (or set) to" does not inherently mean "specially designed to." On the contrary, the phrase "configured to" may indicate that a device may perform an operation together with another device or component. For example, the phrase "a processor configured (or set) to perform A, B, and C" may represent a general-purpose processor (such as a CPU or application processor) that can perform operations by executing one or more software programs stored in a memory device, or a special-purpose processor (such as an embedded processor) for performing operations.

[0015] The terms and phrases used herein are only used to describe some embodiments of the present disclosure, rather than to limit the scope of other embodiments of the present disclosure. It should be understood that the singular forms "one", "an" and "the" include plural references, unless the context clearly states otherwise. All terms and phrases used herein (including technical and scientific terms and phrases) have the same meaning as the meaning generally understood by those of ordinary skill in the art to which the embodiments of the present disclosure belong. It will be further understood that terms and phrases (such as those defined in a common dictionary) should be interpreted as having a meaning consistent with its meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined herein. In some cases, the terms and phrases defined herein may be interpreted as excluding embodiments of the present disclosure.

[0016] Examples of "electronic devices" according to embodiments of the present disclosure may include at least one of the following: smart phones, tablet personal computers (PCs), mobile phones, video phones, e-book readers, desktop PCs, laptop computers, netbook computers, workstations, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, cameras, or wearable devices (such as smart glasses, head-mounted devices (HMDs), electronic clothing, electronic bracelets, electronic necklaces, electronic accessories, electronic tattoos, or smart watches). Other examples of electronic devices include smart home appliances. Examples of smart home appliances may include at least one of the following: televisions, digital video disk (DVD) players, audio players, refrigerators, air conditioners, vacuum cleaners, ovens, microwave ovens, washing machines, dryers, air purifiers, set-top boxes, home automation control panels, security control panels, TV boxes (such as Samsung HOMESYNC, APPLETV, or GOOGLE TV), smart speakers or speakers with integrated digital assistants (such as Samsung GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), game consoles (such as XBOX, PLAYSTATION, or NINTENDO). Electronic dictionary, electronic key, portable camera or electronic photo frame. Other examples of electronic devices include various medical devices (such as various portable medical measurement devices (such as blood sugar measurement devices, heart rate measurement devices or body temperature measurement devices), magnetic resource angiography (MRA) devices, magnetic resource imaging (MRI) devices, computer tomography (CT) devices, imaging devices or ultrasound devices), navigation devices, global positioning system (GPS) receivers, event data recorders (EDR), flight data recorders (FDR), car infotainment devices, navigation electronic devices (such as navigation navigation devices or gyrocompasses), avionics, security devices, vehicle head units, industrial or domestic robots, automatic teller machines (ATMs) point of sale (POS) devices or Internet of Things (IoT) devices (such as light bulbs, various sensors, electric meters or gas meters, sprinklers, fire alarms, thermostats, street lights, toasters, fitness equipment, hot water tanks, heaters or boilers). Other examples of electronic devices include at least a part of a piece of furniture or a building / structure, an electronic board, an electronic signature receiving device, a projector or various measuring devices (such as devices for measuring water, electricity, gas or electromagnetic waves). Note that according to various embodiments of the present disclosure, the electronic device may be one or a combination of the devices listed above. According to some embodiments of the present disclosure, the electronic device may be a flexible electronic device. The electronic device disclosed herein is not limited to the devices listed above, and may include any other electronic device now known or later developed.

[0017] In the following description, an electronic device is described with reference to the accompanying drawings according to various embodiments of the present disclosure. As used herein, the term "user" may refer to a person using an electronic device or another device (such as an artificial intelligence electronic device).

[0018] Definitions for certain other words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0019] Nothing in the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the claims scope. The scope of the patented subject matter is limited solely by the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0021] Figure 1 An example network configuration including electronic devices according to an embodiment of the present disclosure is shown.

[0022] Figure 2 An example wake-up word detection and speaker verification system according to an embodiment of the present disclosure is shown.

[0023] Figure 3 An example speaker verification process according to an embodiment of the present disclosure is shown.

[0024] Figure 4 An exemplary false-trigger mitigation model architecture according to an embodiment of the present disclosure is shown.

[0025] Figure 5A An example architecture for speaker vector extraction and / or for audio quality classification according to an embodiment of the present disclosure is shown.

[0026] Figure 5B An example architecture for speaker vector extraction and / or for audio quality classification according to an embodiment of the present disclosure is shown.

[0027] Fig. 6A An example method for performing unenrolled speaker verification training according to an embodiment of the present disclosure is shown.

[0028] Figure 6B An example method for performing unenrolled speaker verification training according to an embodiment of the present disclosure is shown.

[0029] Fig. 7AAn example method for performing unenrolled speaker verification according to an embodiment of the present disclosure is shown.

[0030] Figure 7B An example method for performing unenrolled speaker verification according to an embodiment of the present disclosure is shown.

[0031] Figure 7C An example method for performing unenrolled speaker verification according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0032] The following discussion is described with reference to the accompanying drawings. Figures 1 to 7C As well as various embodiments of the present disclosure. However, it should be understood that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or replacements thereof also belong to the scope of the present disclosure. Throughout the specification and the drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0033] As mentioned above, voice assistants such as BIXBY, SIRI, and ALEXA allow users to perform voice registration as a step to collect user voice profiles. This involves requesting users to record user voice samples, which can be a cumbersome task and a rigid barrier to integrating new users. Therefore, in order to improve user experience and new user registration completion rate, the recent trend is not to require users to perform voice registration. Although this eliminates the cumbersome registration task, it may lead to a degradation of voice wake-up performance and increase the number of invalid wake-ups, which refers to instances in which the voice assistant is activated although the user has not requested or intended the voice assistant to do so. For example, some data have shown that invalid wake-ups can account for more than 50% of voice assistant calls.

[0034] The present disclosure provides techniques to help overcome these types of problems by providing systems and methods for performing self-wake-up word speaker verification when learning a speaker's voice without requiring the user to go through a registration process to reduce invalid wakeups. In various embodiments of the present disclosure, these techniques may involve automatically finding a speaker vector for a wake-up word from any incoming audio to combine a no-registration setup and speaker verification. Embodiments of the present disclosure provide for automatically finding a speaker vector for a wake-up word from incoming audio, performing first and second stage embedding vector updates, and performing speaker verification using the updated vectors. Keyword verification may be performed using a weak similarity threshold in the first stage, and speaker verification may be performed using a strong similarity threshold in the second stage.

[0035] In an embodiment of the present disclosure, clustering of an input embedding vector with a previous speaker embedding vector may be performed to determine whether a received utterance associated with the input embedding vector is similar to a previously received utterance. When additional utterances from the same speaker are received, clustering may result in a greater number of clustered embedding vectors, and a more accurate speaker vector may be provided for speaker verification. The clustering results may also be used to determine whether a weak similarity threshold or a strong similarity threshold is used. In various embodiments, the utterance may also be preprocessed to determine whether the utterance is an intentional utterance or a verified utterance, for example by utilizing one or both of a false trigger mitigation model and an audio quality classification model. The false trigger mitigation model may be used to determine whether the utterance may be an intentional attempt to trigger a voice assistant, and the audio quality classification model may be used to determine whether the received audio is reliable. If the speaker is verified using the system and method of the present disclosure, automatic speech recognition (ASR), natural language understanding (NLU), or other models may be used to further execute the commands included in the utterance.

[0036] Figure 1 An example network configuration 100 including electronic devices according to an embodiment of the present disclosure is shown. Figure 1 The embodiment of the network configuration 100 shown is for illustration only. Other embodiments of the network configuration 100 may be used without departing from the scope of the present disclosure.

[0037] According to an embodiment of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In some embodiments, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 includes circuits for connecting components 120-180 to each other and for transmitting communications (such as control messages and / or data) between components.

[0038] The processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 is capable of controlling at least one of the other components of the electronic device 101 and / or performing operations or data processing related to communication. In some embodiments, the processor 120 may be a graphics processor unit (GPU). As described below, the processor 120 may receive and process inputs (such as audio inputs or data received from an audio input device such as a microphone), and use the inputs to perform speaker verification and / or automatic speech recognition tasks. The processor 120 may also instruct other devices to perform certain operations (such as outputting audio using an audio output device such as a speaker) or display content on one or more displays 160. The processor 120 may also receive inputs (such as data samples to be used in training a machine learning model) and manage such training by inputting samples into the machine learning model, receiving outputs from the machine learning model, and performing learning functions (such as loss functions) to improve the machine learning model.

[0039] The memory 130 may include volatile and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or programs 140. The programs 140 include, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0040] The kernel 141 can control or manage system resources (such as bus 110, processor 120 or memory 130) for performing operations or functions implemented in other programs (such as middleware 143, API 145 or application 147). The kernel 141 provides an interface that allows the middleware 143, API 145 or application 147 to access the various components of the electronic device 101 to control or manage system resources. The application 147 includes one or more applications that support the reception of audio data, the recognition of wake-up words in the speech included in the audio data, the use of audio data to perform false trigger mitigation, the use of audio data to perform audio quality classification, the use of audio data to perform speaker verification, the use of speech recognition, and the execution of tasks related to the content of the speech. These functions can be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, the middleware 143 can be used as a repeater to allow the API 145 or application 147 to communicate data with the kernel 141. Multiple applications 147 can be provided. The middleware 143 is capable of controlling work requests received from the applications 147, such as by assigning a priority for using system resources (such as the bus 110, the processor 120, or the memory 130) of the electronic device 101 to at least one of the plurality of applications 147. The API 145 is an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143.

[0041] The I / O interface 150 serves as an interface that can, for example, transmit commands or data input from a user or other external devices to other components of the electronic device 101. The I / O interface 150 can also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0042] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hover input using an electronic pen or a body part of the user.

[0043] For example, the communication interface 170 can establish communication between the electronic device 101 and an external electronic device (such as the first external electronic device 102, the second external electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for sending and receiving signals (such as images).

[0044] The electronic device 101 also includes one or more sensors 180, which can measure physical quantities or detect the activation state of the electronic device 101 and convert the measured or detected information into electrical signals. The sensor 180 may also include one or more buttons for touch input, one or more microphones, gesture sensors, gyroscopes or gyroscope sensors, air pressure sensors, magnetic sensors or magnetometers, acceleration sensors or accelerometers, grip sensors, proximity sensors, color sensors (such as RGB sensors), biophysical sensors, temperature sensors, humidity sensors, illumination sensors, ultraviolet (UV) sensors, electromyography (EMG) sensors, electroencephalogram (EEG) sensors, electrocardiogram (ECG) sensors, infrared (IR) sensors, ultrasonic sensors, iris sensors or fingerprint sensors. The sensor 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes and other components. In addition, the sensor 180 may include a control circuit for controlling at least one sensor included here. Any of these sensors 180 may be located in the electronic device 101.

[0045] The first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) in which the electronic device can be installed. When the electronic device 101 is installed in the first external electronic device 102 (such as an HMD), the electronic device 101 may communicate with the first external electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the first external electronic device 102 to communicate with the first external electronic device 102 without involving a separate network. The electronic device 101 may also be an augmented reality wearable device including one or more cameras, such as glasses.

[0046] Wireless communication can use, for example, long term evolution (LTE), long term evolution-advanced (LTE-A), fifth generation wireless system (5G), millimeter wave or 60 GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) or global system for mobile communications (GSM) as at least one of the cellular communication protocol. Wired connection can include, for example, at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232) or plain old telephone service (POTS). Network 162 includes at least one communication network, such as a computer network (such as a local area network (LAN) or a wide area network (WAN)), the Internet or a telephone network.

[0047] The first external electronic device 102 and the second external electronic device 104 and the server 106 can each be a device of the same or different type as the electronic device 101. According to some embodiments of the present disclosure, the server 106 includes a group of one or more servers. In addition, according to some embodiments of the present disclosure, all or some operations performed on the electronic device 101 can be performed on another one or more other electronic devices (such as the first external electronic device 102 and the second external electronic device 104 or the server 106). In addition, according to some embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform a certain function or service, the electronic device 101 can request another device (such as the first external electronic device 102 and the second external electronic device 104 or the server 106) to perform at least some functions associated with it, rather than performing the function or service alone or in addition. Other electronic devices (such as the first external electronic device 102 and the second external electronic device 104 or the server 106) can perform the requested function or additional function and transmit the execution result to the electronic device 101. The electronic device 101 can provide the requested function or service by processing the received result as is or additionally. To this end, for example, cloud computing, distributed computing or client-server computing technology can be used. Although Figure 1 The electronic device 101 is shown to include a communication interface 170 to communicate with the first and second external electronic devices 102 and 104 or the server 106 via the network 162 , but according to some embodiments of the present disclosure, the electronic device 101 may operate independently without a separate communication function.

[0048] The server 106 may include components that are the same or similar to the electronic device 101 (or a suitable subset thereof). The server 106 may support driving the electronic device 101 by performing at least one of the operations (or functions) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101. As described below, the server 106 may receive and process inputs (such as audio inputs or data received from an audio input device such as a microphone), and use the inputs to perform speaker verification and / or automatic speech recognition tasks. The server 106 may also instruct other devices to perform certain operations (such as outputting audio using an audio output device such as a speaker) or displaying content on one or more displays 160. The server 106 may also receive inputs (such as data samples to be used in training a machine learning model) and manage such training by inputting samples into the machine learning model, receiving outputs from the machine learning model, and performing learning functions (such as loss functions) to improve the machine learning model.

[0049] although Figure 1 An example of a network configuration 100 including an electronic device 101 is shown, but the Figure 1 Various changes may be made. For example, network configuration 100 may include any suitable number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0050] Figure 2 An example wake-up word detection and speaker verification system 200 according to an embodiment of the present disclosure is shown. For ease of explanation, the system 200 is described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the system 200 may be used with any other suitable electronic device (such as the server 106), and in any other suitable system.

[0051] like Figure 2As shown, the system 200 includes an electronic device 101, which includes a processor 120. The processor 120 is operably coupled to or otherwise configured to use one or more machine learning models, such as a wake-up word detector model 202, a false trigger mitigation model 203, and a speaker verification model 204. The wake-up word detector model 202 can be trained to recognize one or more wake-up words or phrases. In some embodiments, the wake-up word detector model 202 may include (i) a frame-level detector model for detecting and assigning frame-level probabilities regarding whether a frame of audio input includes a wake-up word, and (ii) a word-level verifier model for assigning word-level probabilities to audio frames. Moreover, in some embodiments, the frame-level detector model and the word-level verifier model may be separate models. The speaker verification model 204 is used in various embodiments of the present disclosure to determine whether the verified speaker utters the audio input using a registration-free process as described herein.

[0052] The processor 120 may also be operably coupled to or otherwise configured to use one or more other models 205, such as an audio quality classification model, one or more ASR models, and / or one or more NLU models. It should be understood that the machine learning models 202-205 may be stored in a memory (such as memory 130) of the electronic device 101 and accessed by the processor 120 to perform automatic speech recognition tasks or other tasks. However, the machine learning models 202-205 may be stored in any other suitable manner.

[0053] The system 200 also includes an audio input device 206 (such as a microphone), an audio output device 208 (such as a speaker or headphones), and a display 210 (such as a screen or monitor, such as the display 160). The processor 120 receives audio input from the audio input device 206 and provides the audio input to the trained wake-up word detector model 202. The processor detects whether a wake-up word or a wake-up phrase is included in the speech within the audio input by executing the trained wake-up word detector model 202, and outputs a result, such as one or more predictions that the speech includes the wake-up word or the wake-up phrase. If a wake-up word or a wake-up phrase is detected, the processor 120 analyzes the acoustic features of the audio input to predict whether the user intends to trigger the wake-up of the voice assistant or whether the speech is a random speech by executing the false trigger mitigation model 203. In some embodiments, the processor 120 determines whether the received audio input is of sufficiently high quality for further processing by executing the audio quality classification model of the other model 205. By executing the speaker verification model 204, the processor 120 can also generate an embedding vector from the audio data, perform clustering on the embedding vector and any embedding vectors from previously received utterances, update a speaker vector associated with the speaker, and perform speaker verification to determine whether the embedding vector associated with the received utterance is similar to the updated speaker vector.

[0054] If the processor 120 determines that the embedding vector is similar to the speaker vector, such as if one or more probabilities are above a threshold, the processor 120 may instruct at least one action of the electronic device 101 or another device or system. For example, in response to positive detection of a wake word or wake phrase and speaker verification, the processor 120 may instruct one or more further actions corresponding to one or more instructions or requests provided in the utterance.

[0055] As a specific example, assume that an utterance including a wake-up word or a wake-up phrase (such as "Hey, BIXBY, call Mom") is received from the user via the audio input device 206. In an embodiment of the present disclosure, a wake-up word or a wake-up phrase may mean a word or a phrase including information that the electronic device 101 will be controlled to perform a specific action. Here, the processor detects the presence of the wake-up word "BIXBY" or the phrase "Hey, BIXBY" included in the utterance by executing the trained wake-up word detector model 202. The processor verifies the speaker by executing the trained speaker verification model 204, and the processor 120 instructs the audio output device 208 to output "Call Mom". The processor 120 also causes a phone application or other communication application to start a communication session with a "Mom" contact stored on the electronic device 101 or otherwise associated with the user of the electronic device 101. As another specific example, assume that the utterance "Hey, BIXBY, start the timer" is received. The processor detects the presence of the wake-up word "BIXBY" or the phrase "Hey, BIXBY" included in the speech by executing the trained wake-up word detector model 202. The processor verifies the speaker by executing the trained speaker verification model 204, and the processor 120 can instruct the execution of the timer application and the display of the timer on the display 210 of the electronic device 101.

[0056] In various embodiments, it will be appreciated that trained machine learning models, such as the wake-up word detector model 202 and the speaker verification model 204, can operate to detect whether a wake-up word or wake-up phrase is in an utterance and whether a known speaker provides the utterance. Based on this determination, the utterance may or may not be provided to another machine learning model, such as an automatic speech recognition model, for further processing of the utterance in order to recognize a command given by a user. In addition, in various embodiments, the wake-up word detector model 202 and the speaker verification model 204, as well as other models, such as the audio quality classification model of the false trigger mitigation model 203 and other models 205, act as gatekeepers to provide a lightweight solution for detecting whether a wake-up word or wake-up phrase is even present in an utterance and whether a known speaker provides the utterance before committing additional resources to be processed by the electronic device 101.

[0057] although Figure 2 One example of a wake-up word detection and speaker verification system 200 is shown, but may be used for Figure 2Various changes are made. For example, the audio input device 206, the audio output device 208, and the display 210 can be connected to the processor 120 within the electronic device 101, such as via a wired connection or circuit. In other embodiments, the audio input device 206, the audio output device 208, and the display 210 can be outside the electronic device 101 and connected via a wired or wireless connection. In addition, in some cases, the wake-up word detector model 202, the false trigger mitigation model 203, the speaker verification model 204, and one or more other machine learning models 205 can be stored as separate models called by the processor 120 to perform certain tasks, or can be included in one or more larger machine learning models and form part of one or more larger machine learning models. In addition, in some embodiments, one or more machine learning models (including the wake-up word detector model 202, the false trigger mitigation model 203, the speaker verification model 204, and one or more other machine learning models 205) can be stored away from the electronic device 101, for example, stored on the server 106. Here, the electronic device 101 may send a request including input (such as captured audio data) to the server 106 to process the input using a machine learning model, and the result may be sent back to the electronic device 101. In addition, in some embodiments, the electronic device 101 may be replaced by a server 106 that receives audio input from a client device and sends instructions back to the client device to perform a function associated with the instructions included in the utterance.

[0058] Figure 3 An example speaker verification process 300 according to an embodiment of the present disclosure is shown. For ease of explanation, the process 300 is described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the process 300 can be used with any other suitable electronic device (such as server 106) or combination of devices (such as electronic device 101 and server 106) and in any other suitable system.

[0059] like Figure 3As shown, process 300 includes a wake-up word detector model 202. Wake-up word detector model 202 receives audio input 302, such as audio data received via an audio input device coupled to an electronic device using wake-up word detector model 202 and other models used in process 300. For example, audio input 302 may include a wake-up command (such as "HEY, BIXBY") that triggers process 300. In various embodiments, wake-up word detector model 202 continuously listens for speech and attempts to detect whether the speech includes a wake-up word or a wake-up phrase. Once wake-up word detector model 202 classifies the speech as likely to include a wake-up word or a wake-up phrase, wake-up word detector model 202 passes at least a portion of audio input 302 to false trigger mitigation model 203.

[0060] The false trigger mitigation model 203 analyzes the acoustic characteristics of the audio input 302 to determine whether the utterance is classified as an intentional wake-up command or an unintentional wake-up command, for example, based on the probability that the utterance is intentional being greater than an intention threshold. For example, the audio input 302 to the false trigger mitigation model 203 can be a command word or a wake-up phrase following a wake-up word in the utterance. For example, if the utterance includes "Hey, BIXBY", and after that, includes the user saying "Call Mom", then in this example, the utterance according to "Call Mom" ​​is the input to the false trigger mitigation model 203. The false trigger mitigation model 203 then analyzes the command utterance to see if the user intends to wake up the assistant. If the detected wake-up command is unintentional, the utterance is rejected, and the process 300 ends to wait for receiving new audio input 302. If the detected wake-up command is determined to be intentional, in some embodiments, only the wake-up word utterance is used to extract a speaker embedding vector. In addition, if the wake-up command is determined to be intentional, at least a portion of the audio input 302 can be provided to the audio quality classification model 304. In some embodiments, process 300 may not use audio quality classification model 304, and use of audio quality classification model 304 may be an optional step to filter out audio input 320 below a quality threshold (such as audio input that is too noisy) to ensure that good quality embedding vectors can be extracted from good quality audio.

[0061] In an embodiment using an audio quality classification model 304, the audio quality classification model 304 is used to analyze the audio data and determine whether the audio data is reliable or unreliable, for example, based on determining whether an audio quality score of the audio data is greater than an audio quality threshold. If the audio data is determined to be unreliable using the audio quality classification model 304, the utterance is rejected and the process 300 ends to wait for receiving new audio input 302. If the audio is determined to be reliable, the audio data is provided to a user clustering step 306, which can be performed as part of a task associated with the speaker verification model 204. In this way, the false trigger mitigation model 203 and the audio quality classification model 304 act as gatekeepers to ensure that the audio input 302 is sufficient for further speaker verification processing.

[0062] At the user clustering step 306, a speaker embedding vector is generated or extracted from the wake word utterance included in the audio input 302. At step 306, a set of previously generated speaker vectors representing previous utterances of the wake word or wake phrase are accessed. These previous speaker vectors and the newly extracted speaker embedding vectors are clustered to identify one or more vector clusters, where the clustering reveals cluster vectors with similar characteristics. Therefore, the vectors in the cluster have a high probability of representing utterances made by the same speaker. In various embodiments, the vectors are clustered using spectral clustering, which utilizes the spectrum (eigenvalues) of the similarity matrix of the audio data to perform dimensionality reduction before clustering in fewer dimensions. The similarity matrix can be provided as an input and can represent a quantitative assessment of the relative similarity of each pair of points in the data set. However, other clustering methods, such as K-means clustering, can be used.

[0063] At step 308 of process 300, it is determined whether the number of speaker embedding vectors associated with a cluster is above a threshold, such as a threshold of five, ten, or fifteen speaker embedding vectors (although any suitable number may be used for the threshold). In this example, process 300 includes a two-stage update and validation step, depending on whether the number of cluster vectors is above a threshold. Figure 3As shown, if the number of clustered utterance vectors is below the threshold, then at step 310, the speaker vector associated with the speaker in the clustered utterance vector is updated using the new speaker embedding vector, and the verification (or similarity) threshold is set to a weaker threshold (such as a lower threshold, such as a value between 0.01-0.1). If the number of clustered utterance vectors is equal to or higher than the threshold, then at step 312, the speaker vector is updated using the new speaker embedding vector, and the verification threshold is set to a stronger threshold (such as a higher threshold, such as a value between 0.3-0.4). The speaker verification model 204 uses the updated speaker vector and the new embedding vector to determine a similarity score between the updated speaker vector and the new embedding vector. The speaker verification model 204 compares the similarity score with the similarity threshold set at step 310 or 312 to determine whether the speaker providing the audio input 302 matches the speaker associated with the identified cluster.

[0064] In various embodiments, when a lower first value is used for the similarity threshold, the determination of the similarity score may be performed substantially using keyword verification to detect the similarity between the speaker vector and the wake-up word or wake-up phrase in the embedding vector. In addition, in various embodiments, when a higher second value is used for the similarity threshold, the determination of the similarity score may be performed substantially using speaker verification to detect the similarity between the speaker providing the utterance and the speaker voice characteristics of the speaker associated with the identified cluster.

[0065] like Figure 3 As shown, the speaker verification model 204 can provide feedback in the form of additional embedding vectors to be used in subsequent clustering at step 306. In this way, over time, the process 300 can become more accurate in determining whether an utterance belongs to a particular user. This is because the utterances from the previous wake-up word detection and speaker verification process can be used as feedback for the subsequent wake-up word detection and speaker verification process. That is, by collecting more speaker embedding vectors for each individual speaker, the speaker verification model becomes more robust and accurate because (by switching from a weak similarity threshold to a strong similarity threshold) the speaker embedding vectors closest to the centroid point of the cluster can be collected. Once all speaker embedding vectors are close enough to the centroid of the cluster (such as five vectors, although this number can be adjustable), other speaker embedding vectors that are farther away from the centroid point can be pruned, leaving the top five speaker embedding vectors for each cluster to retain for future inference.

[0066] Based on the comparison performed by the speaker verification model 204, at step 314, it is determined whether the speaker providing the audio input 302 is a verified speaker. If the speaker is not verified, the process 300 ends. If the speaker is verified, one or more further actions can be indicated to be performed by the electronic device. For example, the speech can be transmitted to one or more devices or to the server-side ASR / NLU model 316, and then at least one command included in the speech can be satisfied. For example, the electronic device can use the ASR / NLU model 316 to further process the audio input 302 or subsequent audio inputs and determine at least one command issued in the speech, such as starting a timer, creating a reminder, initiating a phone call, etc. The electronic device can execute the determined command. Therefore, if a wake-up word or wake-up phrase is detected in the process 300 and the speaker is verified, the electronic device can use the ASR / NLU model 316 to further process the speech and instruct the electronic device 101 or another device or system at least one action according to the command issued by the user.

[0067] although Figure 3 One example of a speaker verification process 300 is shown, but may be used for Figure 3 Make various changes. For example, the data in Figure 3 306-316. In some embodiments, the speaker verification model 204 may be used to perform all steps 306-316. In other embodiments, the processor 120 may perform steps 306-312, and the speaker verification model 204 may be used to perform steps 314-316 using a similarity threshold set by the processor 120 at step 310 or 312. In other embodiments, the speaker verification model 204 may not be used to perform the comparison at step 314, but may only output a similarity score, and the processor 120 may use the similarity score output by the speaker verification model 204 and perform the comparison based on a similarity threshold determined using the number of utterances of the speaker (such as based on the clustering results as described in the embodiments of the present disclosure).

[0068] Process 300 may also be performed using a distributed architecture. For example, speaker verification model 204, false trigger mitigation model 203, and audio quality classification model 304 may be executed on a client electronic device (such as electronic device 101) or at a server (such as server 106). In various embodiments, (one or more) ASR / NLU model 316 may also be executed by an electronic device or by a server. When executed by a server, the server may provide the client electronic device with the determined action to be performed by the client electronic device. In some embodiments, self-wake-up word verification may be performed by a server (such as server 106), the client electronic device may provide audio data received via an audio input device of the client electronic device to the server, and speaker verification model 204, false trigger mitigation model 203, audio quality classification model 304, and ASR / NLU model 316 may be performed by the server based on the audio data provided from the client electronic device.

[0069] Furthermore, in some embodiments, additional steps may be performed to verify whether a valid speaker-based wake-up has been performed. Figure 3 After the process 300 shown, further steps may be performed, such as issuing a prompt to the user of the electronic device to ask the current user whether the latest utterance is from the user. In some cases, the user may also be determined or weighted based on the most active user of the electronic device. In addition, in some cases, the prediction may take into account the voice assistant usage pattern, such as whether a particular user is allowed to unlock the electronic device. It has been shown that speaker verification of various embodiments of the present disclosure significantly reduces the number of invalid wake-ups, with a success rate of up to 99.9% in some instances.

[0070] Figure 4 FIG. 4 shows a false trigger mitigation model architecture 400 according to an embodiment of the present disclosure. For ease of explanation, the false trigger mitigation model architecture 400 is described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the false trigger mitigation model architecture 400 can be used with any other suitable electronic device (such as server 106) or a combination of devices (such as electronic device 101 and server 106), and can be used in any other suitable system. In addition, in various embodiments, the false trigger mitigation model architecture 400 can be used as at least a part of the false trigger mitigation model 203 described in the embodiments of the present disclosure.

[0071] like Figure 4As shown, in this example embodiment, the false trigger mitigation model architecture 400 can be arranged with a four-layer long short-term memory (LSTM) based acoustic embedding. The false trigger mitigation model architecture 400 receives acoustic features 402 as initial inputs to multiple LSTM units 404. The LSTM units 404 use feedback connections to process variable length input sequences in a cyclic manner, and the acoustic embedding focuses on low-level information, such as the presence of speech, background noise, and acoustic environment.

[0072] The last layer of the false trigger mitigation model architecture 400 generates a class output 406 and an acoustic embedding output 408. The class output 406 indicates whether the acoustic feature 402 should be accepted or rejected. For example, in an embodiment of the present disclosure, the class output 406 may indicate whether the utterance is intentional or unintentional. In addition, in various embodiments of the present disclosure, the acoustic embedding output 408 may be used during training of the false trigger mitigation model architecture 400, while the acoustic embedding output 408 may be discarded during inference. In various embodiments, during training, the output acoustic embedding may be used to estimate the acoustic embedding from an external pre-trained model.

[0073] although Figure 4 An example of a false trigger mitigation model architecture 400 is shown, but may be used for Figure 4 Various changes may be made. For example, the false trigger mitigation model architecture 400 may include any suitable number of LSTM units in any suitable arrangement. Figure 4 The false trigger mitigation model architecture 400 is shown as an LSTM machine learning model, but other architectures can be used for the false trigger mitigation model architecture 400. In general, machine learning models and architectures can have a variety of configurations, and Figure 4 The scope of the present disclosure is not limited to any particular configuration.

[0074] Figure 5A and Figure 5B FIG. 5 shows an architecture 500 for speaker vector extraction and / or for audio quality classification according to an embodiment of the present disclosure. For ease of explanation, the architecture 500 is described as involving Figure 1 The electronic device 101 is used in the network configuration 100 of FIG. However, the architecture 500 can be used with any other suitable electronic device (such as a server 106) or a combination of devices (such as an electronic device 101 and a server 106) and in any other suitable system.

[0075] like Figure 5AAs shown, in this example embodiment, the architecture 500 may include an emphasized channel attention, propagation and aggregation time delay neural network (ECAPA-TDNN) configuration that applies statistical pooling to project variable length utterances into fixed length speaker representation embeddings. In various embodiments, the architecture 500 may be used as at least a portion of the speaker verification model 204 and / or at least a portion of the audio quality classification model 304 described in the embodiments of the present disclosure. The difference between using the architecture 500 for an audio quality classification model and for speaker verification and / or speaker embedding extraction models involves using different training data sets and labels for the corresponding models. For audio quality classification model training, both clean data and noisy data can be used to train the audio quality classification model how to classify audio data as clean or noisy (reliable or unreliable). For speaker verification and / or speaker embedding extraction model training, speaker samples containing or not containing a wake-up word or phrase can be used to train the model to a speaker verification and / or speaker embedding extraction model to identify and extract speaker embeddings for audio data. Furthermore, audio samples with labeled speaker IDs can be used during training for speaker verification and speaker embedding extraction model training.

[0076] The architecture 500 includes a convolutional layer 502 that uses input features of a specific dimension, such as a 40-dimensional Mel-Frequency Cepstral Coefficient (MFCC) feature vector. The convolutional layer 502 performs a one-dimensional convolution (Conv1D), a rectified linear unit (ReLU), and a batch normalization (BN) operation using the input features. In this example, k=5 and d=1, where k represents the kernel size and d represents the dilation interval of the Conv1D layer of the squeeze and excitation (SE) Res2Blocks (SE-Res2Blocks). Figure 5A As shown, the output from the convolutional layer 502 is received at box 503a, where C corresponds to the channel and T corresponds to the time dimension of the intermediate feature map. Box 503a is SE-Res2Block, which is used to scale the frame-level features of the audio input given the global properties of the recording. Figure 5A In the example of FIG. 5 , at blocks 503 a - 503 c , three SE-Res2Blocks are included in the architecture 500 . Figure 5A As shown, for box 503a, k=3 and d=2, for box 503b, k=3 and d=3, and for box 503c, k=3 and d=4.

[0077] Example configuration of blocks 503a-503c 503 in Figure 5BHere, configuration 503 includes a box 505 where SE-Res2Block performs Conv1D, ReLU, and BN operations on the input from the previous layer. At box 507, expanded Conv1D, ReLU, and BN operations are performed, followed by additional Conv1D, ReLU, and BN operations at box 509. The task of the expanded convolutional layer is to gradually build up the temporal context. SE block 511 is used to expand the temporal context of the frame layer by rescaling the channels according to the global properties of the record. Configuration 503 enhances the central convolutional layer so that it can handle multi-scale features by constructing a hierarchical residual connection therein, which can improve performance while significantly reducing the number of model parameters.

[0078] Reference again Figure 5A , the output from each of the SE-Res2Block blocks 503a-503c is provided to the next convolutional layer 504, which performs additional Conv1D and ReLU operations (such as k=1 and d=1). At box 506, attentive stat pooling and BN (such as using 512 channels x T) are performed. At box 508, a fully connected layer using BN is used (such as the number of channels is set to 1536 channels). At box 510, an additive angular margin (AAM) softmax function (such as the number of channels is set to 256) is performed as an activation function to provide a final output, where S is the number of training speakers. For example, speaker embeddings can be extracted from the final fully connected layer and used to perform additional speaker verification tasks.

[0079] although Figure 5A and Figure 5B One example of an architecture 500 for speaker vector extraction and / or for audio quality classification is shown, but may be used for Figure 5A and Figure 5B Various changes may be made. For example, when architecture 500 is used for speaker vector embedding extraction, architecture 500 may be at least a portion of speaker verification model 204, or may be a portion of a separate speaker embedding extraction model that extracts speaker embeddings for use by speaker verification model 204. Figure 5A and Figure 5B The architecture 500 is shown as having an ECAPA-TDNN configuration, but other suitable configurations may be used for the architecture 500. In general, machine learning models and architectures may have a variety of configurations, and Figure 5A and Figure 5B The scope of the present disclosure is not limited to any particular configuration.

[0080] Fig. 6A and Figure 6BAn example method 600 for performing unenrolled speaker verification training according to an embodiment of the present disclosure is shown. For ease of explanation, Fig. 6A and Figure 6B The method 600 shown in FIG. 1 is described as using Figure 1 As a specific example, the method 600 may be performed by the server 106 in the network configuration 100. Figure 1 The method 600 is executed on the server 106 in the network configuration 100, and the trained machine learning models (such as the trained wake-up word detector model 202, the trained speaker verification model 204, the trained false trigger mitigation model 203, and the trained audio quality classification model 304) can be deployed to the client electronic device 101 for use. However, the method 600 can be used with any other suitable device (such as the electronic device 101) and can be used in any other suitable system.

[0081] In order to train various machine learning models, the processor 120 of the server 106 provides various training data to the machine learning model. The training data typically includes training input data and known ground truth values, wherein the training input data is provided to the machine learning model for processing, and the ground truth value represents the expected output of the machine learning model. At box 602, the processor provides a sample embedding vector to a speaker verification model, such as speaker verification model 204, based on the speech included in the sample audio data. In some embodiments, the speech includes a wake-up word or a wake-up phrase. At box 604, the processor provides a sample speaker vector to the speaker verification model, wherein the sample speaker vector represents one or more speeches associated with a speaker or user. During the training of the speaker verification model, the sample speaker vector can be a positive sample or a negative sample. For example, if a positive training sample is provided for a particular round of training, the speaker vector can represent the speaker's speech, which includes a wake-up word or a wake-up phrase. If a negative training sample is provided for a particular round of training, the speaker vector represents the speaker's speech that does not include a wake-up word or a wake-up phrase. As another example, the sample speaker vector may be positive or negative based on whether the speaker vector is from the same speaker as the sample embedding vector or from a different speaker.

[0082] At box 606, the processor receives a result or output including a similarity score between a sample speaker vector and a sample embedding vector from a speaker verification model. At box 608, the processor determines an error or loss based on the output from the speaker verification model using a loss function, and modifies the speaker verification model based on the error or loss. The output of the speaker verification model generated using the training input data can represent a confidence, such as a similarity score, and can be provided to the loss function. The loss function calculates an error or loss associated with the prediction of the speaker verification model. For example, when the output of the speaker verification model is different from the ground truth, the difference can be used to calculate a loss as defined by the loss function. The loss function can use any suitable measure of the loss associated with the output generated by the speaker verification model, such as a cross entropy loss or a mean squared error.

[0083] At decision block 610, the processor determines whether initial training of the speaker verification model is complete, meaning that the speaker verification model is predicting speaker similarity with an acceptable level of accuracy using the input training data. When the loss calculated by the loss function is greater than expected, the parameters of the speaker verification model can be adjusted. Once adjusted, the method 600 moves back to block 602 to provide the same or additional training data to the adjusted speaker verification model, and the additional output from the speaker verification model can be compared to the ground truth so that the additional loss can be determined using the loss function. Over time, the speaker verification model produces more accurate outputs that more closely match the ground truth, and the measured loss becomes less. The amount of training data used can vary depending on the number of training cycles, and can include a large amount of training data (such as 9,375 hours of data containing 10,001 speaker samples). At some point, at decision block 610, the measured loss can drop below a specified threshold, the initial training of the speaker verification model can be completed, and the method 600 moves to block 612.

[0084] At block 612, the processor further trains the speaker verification model to compare the similarity score to a similarity threshold. In various embodiments of the present disclosure, different similarity thresholds may be used based on the number of speaker utterances associated with less than, equal to, or greater than a threshold number of utterances. During the training method 600, the speaker verification model may be trained to use a threshold based on sample data to use a first value for the similarity threshold (based on the sample speaker vector being associated with less than the threshold number of utterances) or a second value greater than the first value of the similarity threshold (based on the sample speaker vector being associated with equal to or greater than the threshold number of utterances).

[0085] At box 614, the processor trains a false trigger mitigation model, such as false trigger mitigation model 203, to determine the probability that the speech is an intentional speech. In order to train the false trigger mitigation model, the processor provides positive and / or negative training data to the false trigger mitigation model. In some embodiments, the training data may include one or more of: (i) a set of full-length audio samples, (ii) a set of chunk-wise speech, and (iii) audio samples including one or more wake-up words or phrases. Any number of training samples can be used in the training process, such as a large amount of training data (e.g., using 80,000 full-length audio samples and 10,000 wake-up word / phrase audio samples). The processor receives outputs such as class outputs and acoustic embedding outputs from the false trigger mitigation model, about Figure 4 An example thereof is described. The processor determines an error or loss using a loss function based on an output from a false trigger mitigation model, and modifies the false trigger mitigation model based on the error or loss. The output of the false trigger mitigation model generated using the training input data can represent a confidence level as to whether an utterance is an intentional utterance that invokes a wake word or a wake phrase. The loss function calculates an error or loss associated with a prediction of the false trigger mitigation model. For example, when the output of the false trigger mitigation model differs from the ground truth, the difference can be used to calculate the loss defined by the loss function. The loss function can use any suitable measure of the loss associated with the output generated by the false trigger mitigation model, such as a cross entropy loss or a mean squared error.

[0086] Once the predictions provided by the false trigger mitigation model are at an acceptable level of accuracy, the processor determines whether the training of the false trigger mitigation model is complete. When the loss calculated by the loss function is greater than the expectation, the parameters of the false trigger mitigation model can be adjusted. Once adjusted, the processor can provide the same or additional training data to the adjusted false trigger mitigation model, and the additional output from the false trigger mitigation model can be compared with the ground truth so that the additional loss can be determined using the loss function. Ideally, over time, the false trigger mitigation model produces more accurate outputs that more closely match the ground truth, and the measured loss becomes less. At some point, the measured loss can drop below a specified threshold, at which point the training of the false trigger mitigation model can be completed, and the method 600 moves to decision box 616.

[0087] At decision block 616, the processor determines whether to use an audio quality classification model, such as audio quality classification model 304. In some embodiments, the use of an audio quality classification model may be an optional step to filter out audio inputs below a quality threshold in order to ensure that good quality embedding vectors can be extracted from good quality audio. If it is determined at decision block 616 that the audio quality classification model will be used, then the method 600 moves to block 618. If it is determined at decision block 616 that the audio quality classification model will not be used, then the method 600 moves to block 620. At block 618, the processor trains the audio quality classification model to determine an audio quality score associated with the speech. Training includes providing positive and / or negative training samples to the audio quality classification model, such as clean audio sample data of a first predetermined duration and noisy audio sample data of a second predetermined duration. For example, during the training process, the amount of training data used may vary depending on the number of training cycles, and may include a larger amount of training data, such as 14.5 hours of clean data and 14.5 hours of noise data, which contains 95,000 noise profiles, in order to classify clean and noisy (i.e., reliable and unreliable) data.

[0088] The processor executes a loss function based on the output from the audio quality classification model and modifies the audio quality classification model based on the loss function. The output of the audio quality classification model generated using the training input data can represent a confidence level as to whether the audio is reliable or unreliable. The output is provided to the loss function, which calculates a loss associated with the prediction of the audio quality classification model. For example, when the output of the audio quality classification model differs from the ground truth, the difference can be used to calculate the loss defined by the loss function.

[0089] The loss function may use any suitable measure of the loss associated with the output generated by the audio quality classification model, such as cross entropy loss or mean square error. Once the prediction provided by the audio quality classification model is at an acceptable level of accuracy, the processor determines whether the training of the audio quality classification model is complete. When the loss calculated by the loss function is greater than the expectation, the parameters of the audio quality classification model may be adjusted. Once adjusted, the processor may provide additional training data to the adjusted audio quality classification model, and the additional output from the audio quality classification model may be compared with the ground truth, so that the additional loss may be determined using the loss function. Over time, the audio quality classification model produces more accurate outputs that more closely match the ground truth, and the measured loss becomes less. At a certain point, the measured loss may drop below a specified threshold, at which point the training of the audio quality classification model may be completed, and the method 600 moves to box 620. At box 620, after the training of the machine learning model is completed, the trained machine learning model is deployed to a client electronic device (such as electronic device 101). In some cases, the model may be deployed as a lightweight model for performing speaker verification during wake-up word detection.

[0090] although Fig. 6A and Figure 6B One example of a method 600 for performing unenrolled speaker verification training is shown, but may be used for Fig. 6A and Figure 6B For example, although shown as a series of steps, Fig. 6A and Figure 6B The various steps in can overlap, occur in parallel, occur in different orders, or occur any number of times. As a specific example, blocks 602-610 can occur any number of times, depending on the number of batches or epochs used to train the speaker verification model. Similarly, the training steps covered in blocks 614 and 618 can be performed any number of times, depending on the amount of training to be performed to train the false trigger mitigation model and the audio quality classification model. Moreover, in some embodiments, the speaker verification model may not perform comparisons at block 612, and may be trained to output a similarity score (such as at block 606), and the processor may use the similarity score output by the speaker verification model at block 612 and perform comparisons based on a similarity threshold determined according to the number of utterances of the speaker (such as based on the clustering results described in the embodiments of the present disclosure).

[0091] FIG. 7A to FIG. 7C An example method 700 for performing non-registered speaker verification according to an embodiment of the present disclosure is shown. For ease of explanation, FIG. 7A to FIG. 7C The method 700 shown is described as using Figure 1The method 700 is performed by the electronic device 101 in the network configuration 100. However, the method 700 may be performed using any other suitable device (such as the server 106) and in any other suitable system.

[0092] At box 702, the processor obtains audio data from an audio input device. At box 704, the processor identifies an utterance of a wake word or wake phrase in the audio data, such as by using a wake word detector model similar to the wake word detector model 202. At box 706, the processor provides the audio data to a false trigger mitigation model, such as the false trigger mitigation model 203. At decision box 708, the processor uses the false trigger mitigation model to determine whether the utterance is classified as an intentional utterance or an unintentional utterance, for example, based on a probability that the utterance is intentional being greater than an intention threshold. If the processor determines that the utterance is unintentional, the utterance is rejected at box 709, and the method 700 ends at box 711.

[0093] If the processor determines that the utterance is intentional, the method 700 moves to decision block 710. At decision block 710, it is determined whether to use an audio quality classification model, such as the audio quality classification model 304. In some embodiments, the use of an audio quality classification model can be an optional step to filter out audio inputs below a quality threshold (such as audio inputs that are too noisy) to ensure that good quality embedding vectors can be extracted from good quality audio. If it is determined not to use an audio quality classification model, the method 700 proceeds to block 716 to begin the process of clustering embedding vectors in response to the probability that the utterance is intentional being greater than the intent threshold.

[0094] If it is determined that the audio quality classification model will be used, then the method 700 moves to box 712. At box 712, the processor provides the audio data associated with the audio input to the audio quality classification model. At decision box 714, the processor determines whether the audio data is reliable or unreliable based on determining whether the audio quality score of the audio data determined by the audio quality classification model is greater than the audio quality threshold. If the processor determines that the audio data is unreliable, the utterance is rejected at box 709 and the method 700 ends at box 711. If the processor determines that the audio is reliable, the method 700 moves to box 716 to begin the process of clustering the embedding vectors in response to the probability that the utterance is intentional being greater than the intent threshold and the audio quality score being greater than the audio quality threshold.

[0095] At box 716, the processor generates or extracts an embedding vector based on the speech from the audio input data. At box 718, the processor accesses a previously generated vector set representing a previous speech of a wake word or wake phrase. At box 720, the processor performs clustering on the embedding vector and the previously generated vector set to identify a cluster including the embedding vector, wherein the identified cluster is associated with a specific speaker. At box 722, the processor updates the speaker vector associated with the speaker based on the embedding vector. At decision box 724, the processor determines whether the number of clustered utterances of the speaker is below a threshold, such as a threshold of five, ten, or fifteen utterances (although any suitable number of utterances can be used for the threshold). If the number of clustered utterances of the speaker is less than the threshold, then at box 726, the processor sets the similarity threshold to a first value (such as a lower threshold, such as a value between 0.01-0.1). Otherwise, if the number of clustered utterances of the speaker is equal to or greater than the threshold, the processor sets the similarity threshold to a second value (such as a higher threshold, such as a value between 0.3-0.4). In this way, as described above, method 700 can more accurately determine whether an utterance belongs to a particular user over time because utterances from previous wake word detection and speaker verification processes can be used as feedback for subsequent wake word detection and speaker verification processes.

[0096] At box 730, the processor uses a speaker verification model (such as speaker verification model 204) to determine a similarity score between the updated speaker vector and the embedding vector. At box 732, the processor compares the similarity score to the similarity threshold set at box 726 or 728 to determine whether the speaker providing the utterance matches the speaker associated with the identified cluster. In various embodiments, when a lower first value is used for the similarity threshold, the determination of the similarity score can be performed substantially using keyword verification to detect the similarity between the speaker vector and the wake-up word or wake-up phrase in the embedding vector. When a higher second value is used for the similarity threshold, the determination of the similarity score can be performed substantially using speaker verification to detect the similarity between the speaker providing the utterance and the speaker associated with the identified cluster.

[0097] At decision block 734, the processor determines whether the speaker providing the audio data received at block 702 is a verified speaker based on the comparison performed in block 732, so as to trigger one or more further actions based on the detected wake-up word / phrase and speaker verification. If the processor determines that the speaker is not verified, the method 700 ends at block 738. If the processor determines that the speaker is verified, the method 700 moves to block 736. At block 736, the processor instructs to perform at least one action. For example, the utterance may be transmitted to, for example, one or more on-device or server-side ASR / NLU models, such as ASR / NLU model 316, and then at least one command included in the utterance may be satisfied. For example, the processor may use the ASR / NLU model to further process the audio data and determine at least one command issued in the utterance, such as starting a timer, creating a reminder, initiating a phone call, etc. The processor may execute the determined command to satisfy the user's intent.

[0098] although FIG. 7A to FIG. 7C One example of a method 700 for performing non-enrolled speaker verification is shown, but may be used for FIG. 7A to FIG. 7C For example, although shown as a series of steps, FIG. 7A to FIG. 7C The various steps in may overlap, occur in parallel, occur in a different order, or occur any number of times. As a specific example, in some embodiments, the speaker verification model 204 may be used to perform all of blocks 716 to 734. In other embodiments, the processor may perform blocks 716 to 728, and the speaker verification model 204 may be used to perform blocks 730 to 732 using a similarity threshold set by the processor at blocks 726 or 728. In other embodiments, the speaker verification model 204 may not be used to perform a comparison at block 732 and may output a similarity score (such as at block 730), and the processor may perform a comparison at block 732 using a similarity score output by the speaker verification model 204 and based on a similarity threshold determined using the number of utterances of the speaker (such as based on a clustering result as described in an embodiment of the present disclosure).

[0099] Method 700 may also be performed using a distributed architecture. For example, speaker verification model 204, false trigger mitigation model 203, and audio quality classification model 304 may be executed on a client electronic device (such as electronic device 101) or at a server (such as server 106). In various embodiments, (one or more) ASR / NLU model 316 may also be executed by an electronic device or by a server. When executed by a server, the server may provide the client electronic device with the determined action to be performed by the client electronic device. In some embodiments, self-wake-up word verification may be performed by a server (such as server 106), the client electronic device may provide audio data received via an audio input device of the client electronic device to the server, and speaker verification model 204, false trigger mitigation model 203, audio quality classification model 304, and ASR / NLU model 316 may be performed by the server based on the audio data provided from the client electronic device.

[0100] Furthermore, in some embodiments, additional steps may be performed to verify whether a valid speaker-based wake-up has been performed. FIG. 7A to FIG. 7C After the method 700 shown, a prompt may be issued to the user of the electronic device to ask the user whether the latest utterance is from the user. In some embodiments, the user may also be determined or weighted based on the most active user of the electronic device. In addition, in some cases, the prediction may take into account the voice assistant usage pattern, such as whether a particular user is allowed to unlock the electronic device.

[0101] Although the present disclosure has been described with exemplary embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method, include: obtaining audio data using at least one processing device of the electronic device; identifying, using the at least one processing device, utterance of a wake word or a wake phrase in the audio data; generating, using the at least one processing device, an embedding vector based on the utterance from the audio data; accessing, using the at least one processing device, a set of previously generated vectors representing previous utterances of the wake word or wake phrase; performing clustering on the embedding vector and the set of previously generated vectors using the at least one processing device to identify a cluster including the embedding vector, the identified cluster being associated with a speaker; updating, using the at least one processing device, a speaker vector associated with the speaker based on the embedding vector; determining a similarity score between an updated speaker vector and the embedding vector using the at least one processing device and a speaker verification model; as well as A determination is made using the at least one processing device and based on the similarity score whether a speaker providing the utterance matches a speaker associated with the identified cluster.

2. The method according to claim 1, in, Determining whether a speaker providing the utterance matches a speaker associated with the identified cluster includes comparing the similarity score to a similarity threshold using the at least one processing device.

3. The method according to claim 2, in, If the number of clustered utterances for the speaker is less than a threshold number of utterances, a first value is used for the similarity threshold.

4. The method according to claim 3, in, If the number of clustered utterances of the speaker is equal to or greater than a threshold number of utterances, a second value greater than the first value is used for the similarity threshold.

5. The method according to claim 4, in: When the first value is used for the similarity threshold, determining the similarity score is performed using keyword verification to detect similarity between the wake-up word or wake-up phrase in the speaker vector and in the embedding vector; and When the second value is used for the similarity threshold, determining the similarity score is performed using speaker verification to detect a similarity between a speaker providing the utterance and a speaker associated with the identified cluster.

6. The method according to claim 1, further comprising: include: determining, using the at least one processing device, a probability that the utterance is an intentional utterance; as well as In response to the probability being greater than an intent threshold, the clustering is performed using the at least one processing device.

7. The method according to claim 6, further comprising: include: Training a false trigger mitigation model using the at least one processing device to determine a probability that the utterance is an intentional utterance, wherein training the false trigger mitigation model comprises: One or more of a set of full-length audio samples, a set of chunked utterances, and audio samples including one or more wake words or phrases are provided to the false trigger mitigation model using the at least one processing device.

8. The method according to claim 6, further comprising: include: In response to the probability being greater than the intent threshold, determining, using the at least one processing device, an audio quality score associated with the utterance; as well as In response to the audio quality score being greater than an audio quality threshold, the clustering is performed using the at least one processing device.

9. The method according to claim 8, further comprising: include: Training an audio quality classification model using the at least one processing device to determine the audio quality score associated with the utterance, wherein training the audio quality classification model comprises: Using the at least one processing device, clean audio sample data of a first predetermined duration and noisy audio sample data of a second predetermined duration are provided to the audio quality classification model.

10. The method according to claim 1, further comprising: include: Training the speaker verification model using the at least one processing device, wherein training the speaker verification model comprises: providing a sample embedding vector based on an utterance from sample audio data, wherein the utterance includes the wake word or wake phrase; providing a sample speaker vector using the at least one processing device; receiving, using the at least one processing device, a result from the speaker verification model including a similarity score between the sample speaker vector and the sample embedding vector; and A loss function is used and the speaker verification model is modified based on the results.

11. A device, include: At least one processing device configured to: Get audio data; identifying utterances of a wake-up word or a wake-up phrase in the audio data; generating an embedding vector based on the utterance from the audio data; accessing a set of previously generated vectors representing previous utterances of the wake word or wake phrase; performing clustering on the embedding vector and the set of previously generated vectors to identify a cluster including the embedding vector, the identified cluster being associated with a speaker; updating a speaker vector associated with the speaker based on the embedding vector; determining a similarity score between the updated speaker vector and the embedding vector using a speaker verification model; as well as A determination is made based on the similarity score whether a speaker providing the utterance matches a speaker associated with the identified cluster.

12. The device according to claim 11, in, To determine whether a speaker providing the utterance matches a speaker associated with the identified cluster, the at least one processing device is further configured to compare the similarity score to a similarity threshold.

13. The device according to claim 12, in, If the number of clustered utterances for the speaker is less than a threshold number of utterances, a first value is used for the similarity threshold.

14. The device according to claim 13, in, If the number of clustered utterances of the speaker is equal to or greater than a threshold number of utterances, a second value greater than the first value is used for the similarity threshold.

15. A computer readable medium comprising instructions which, when executed, cause at least one processor to perform the method according to any one of claims 1-10.

Images