System and method for detecting wake-up commands for voice assistant

Abstract

A method for detecting a wake-up command of a voice assistant is provided. The method includes receiving an audio signal from one or more sources and determining at least one of an acoustic parameter or an ambient context of a user. Further, the method includes generating an embedding vector representation associated with the received audio signal, and comparing the generated embedding vector representation with one or more pre-stored embedding vector representations. Further, the method includes detecting a wake-up command in the received audio signal.

Description

Technical Field

[0001] The present disclosure relates to audio processing and more particularly to systems and methods for detecting a wake-up command of a voice assistant implemented at an electronic device. Background Art

[0002] As technology advances, voice assistants have revolutionized the way users interact with technology. Voice assistants have become an integral part of our daily lives, providing convenience, efficiency, and a hands-free means to access information and perform tasks. A voice assistant is a digital assistant that uses speech recognition and natural language processing (NLP) technologies to provide information and perform tasks to users through voice commands. In addition, voice assistants are often implemented in electronic devices such as smartphones, smart speakers, televisions, and the like. Voice assistants can perform a wide range of tasks, such as providing weather updates, answering questions, playing music, setting alarms and reminders, sending messages, making calls, controlling smart home devices, and the like. For example, a voice assistant may be Bixby®.

[0003] Typically, a wake-up command, also called a wake-up word or activation phrase, is used to trigger the voice assistant to start listening and responding to user commands. When the user speaks the wake-up command, the voice assistant becomes active and ready to receive instructions and perform tasks accordingly. The wake-up command is generally a unimodal input through voice or a long key press.

[0004] Figure 1A A block diagram 100 is shown for executing a wake-up command of a voice assistant in an environment according to the related art.

[0005] refer to Figure 1A , the wake-up command is mainly performed by two components, namely, a keyword detector 102 and a keyword classifier 104. The keyword detector 102 includes a set of modules, such as a Mel-frequency cepstral coefficient (MFCC) calculation 106, a frame buffer 108, a small deep neural network (DNN) 110, and a hidden Markov model (HMM) scorer 112. Typically, the keyword detector 102 is always active and listens to a speech signal 114 in the environment. The keyword detector 102 corresponds to a low-computation digital signal processing (DSP) model. In addition, the keyword classifier 104 includes a set of modules, such as a large DNN 116 and an HMM scorer 118. The keyword classifier 104 is a large module that is more precise and accurate in wake-up detection. In addition, Table 1 shows various examples of inputs and outputs of the keyword detector 102 and the keyword classifier 104.

[0006]

[0007] Typically, voice assistants are susceptible to noise, interference, and conversation in the surrounding environment, which affects the accuracy or reliability of wake-up command recognition. Excessive background noise or environmental factors (such as background conversations, loud music, or electrical appliances) may interfere with the ability of voice assistants to accurately identify wake-up commands. Noise may make it difficult for electronic devices that implement voice assistants to pick up wake-up phrases. In some cases, voice assistants may mistakenly interpret sounds or words similar to wake-up commands as actual commands, resulting in false positives. This activation of the voice assistant may occur due to misunderstanding of voice patterns, unintentional triggering of irrelevant sounds, or environmental noise. In addition, acoustic signals similar to wake-up words cause the activation of voice assistants, which is generally referred to as false wake-ups. Every time the same or similar noise appears around an electronic device, the electronic device that implements the voice assistant cannot discard or reject the false activation. Local wake-up fingerprint recognition using acoustic word embeddings (AWE) or equivalent technology is completed in the electronic device. In AWE, an embedding vector is generated for an acoustic word signal. The false wake-up of local learning is completed using a voice assistant wake-up service, and a robust global wake-up model is sent to the electronic device. The wake-up word (acoustic speech word) contains environmental noise. The generated embedding vector indicates the user's acoustic speech characteristics and environmental noise. In addition, the generated embedding vector together with the global embedding from the wake-up service is used to build a device personalization model, such as Figure 1B Further explained in .

[0008] Figure 1B A block diagram 120 is shown depicting on-device wake-up fingerprinting according to related techniques.

[0009] refer to Figure 1B, 122 represents a wake-up audio signal, and 124 represents a wake-up trigger signal received from a user. In an embodiment of the present disclosure, a wake-up acoustic signal that can be classified as a true positive, true negative, false positive, and false negative wake-up is referred to as a wake-up audio sample (wake-up audio signal). In an embodiment of the present disclosure, a wake-up trigger signal is an acoustic signal similar to a wake-up audio sample but used in a real-time environment. The wake-up trigger signal corresponds to a wake-up attempt from a real user. In addition, the wake-up audio signal 122 received from the false positive and false negative analysis passes through an on-device wake-up fingerprint recognition unit 126. In addition, the on-device wake-up fingerprint recognition unit 126 extracts features from the wake-up audio signal 122. More specifically, the on-device wake-up fingerprint recognition unit 126 extracts features including spectral features 126A, energy-based features 126B, and acoustic word embeddings 126C from the wake-up audio signal 122. In an embodiment of the present disclosure, the spectral feature 126A corresponds to frequency information present in an acoustic signal extracted using several types of Fourier transforms. In addition, the energy-based feature corresponds to the derivation of energy in a given acoustic signal. In an embodiment of the present disclosure, acoustic word embedding corresponds to an embedding vector representation in an acoustic signal. The extracted features are stored in a user local fingerprint database (DB) 128. In addition, a matcher module 130 matches the wakeup trigger 124 with the user local fingerprint database 128 and generates a result marking a real wakeup or a false wakeup 131.

[0010] In addition, the user local fingerprint DB sends the extracted features to the server centralized wake-up service 132 for global learning. The server centralized wake-up service 132 includes a custom wake-up model generator 132A, a model synchronization 132B, and a wake-up fingerprint 132C. In an embodiment of the present disclosure, the centralized wake-up service 132 corresponds to a service responsible for generating an AWE model by using a user voice record from a log server. In addition, the centralized wake-up service 132 synchronizes the custom AWE model from the server to the device on demand. The centralized wake-up service 132 generates an embedding vector and features corresponding to the user voice record and stores them in a global fingerprint sound database. In an embodiment of the present disclosure, the custom wake-up model generator 132A corresponds to a module that divides the user voice record into voice segments aligned with the spoken words. In addition, the custom wake-up model generator 132A classifies the voice segments into words from a vocabulary. The custom wake-up model generator 132A performs unsupervised training of an AWE or equivalent embedding model based on a subset of words from a vocabulary. In an embodiment of the present disclosure, the model synchronization 132B communicates between the client device and the wake-up service. In addition, model synchronization 132B builds a custom wake-up model for a specific user based on the specifications received from the client device. In an embodiment of the present disclosure, for each voice segment extracted from the user's voice recording, the wake-up fingerprint 132C generates features such as spectral features, energy information, zero crossing counts, etc. All generated features are stored in a global fingerprint vocab database. In addition, the global fingerprint vocab database also contains associations between voice segments and words.

[0011] In addition, the on-device personalized model generator 134 downloads the server-generated global model learned from multiple users to the electronic device. The on-device personalized model generator 134 uses the global model to incrementally learn the personalized input of the user. In addition, the global fingerprint vocabulary DB 136 includes features from high-resolution words that can be used to model wakeup. In an embodiment of the present disclosure, the global fingerprint vocabulary DB 136 stores features and embedding vectors corresponding to acoustic words extracted from the user's voice recording. High-resolution words are clean acoustic signal segments from which the corresponding words can be easily perceived. In addition, user-specific audio segments of custom wake-up words are extracted from the user log server 138 via one or more tools 140 for model generation. The tool 140 includes a personalized word extraction 142 and a high-resolution (hi-res) word audio signal 144.

[0012] Figure 2 A graphical representation 200 is shown depicting a scenario of a false wakeup of a voice assistant in an environment according to the related art.

[0013] refer to Figure 2, as shown, 202 represents a media device in a user environment. In the current environment, the voice assistant may experience noise caused by media devices 202 (such as radio, television, and / or human conversation 204). In human conversation 204, humans can use words / phrases similar to the wake-up words / commands of the voice assistant. For example, human conversation 204 can be "Bigs Bee is very talented". In this article, the term "Bigs Bee" is similar to the term "Bixby®", which causes the voice assistant (in this case Bixby®) to wake up based on a false wake-up word (i.e., "Bigs Bee"). 206 describes a Bixby® enabled device, in which Bixby® wakes up without an actual voice wake-up trigger from the user.

[0014] Figure 3 A graphical representation of a scenario 300 depicting true wakeup rejection according to the related art is shown.

[0015] refer to Figure 3 , in noisy environments such as traffic, restaurants, shopping malls, etc., voice assistants often fail to recognize the wake-up word from the user of the electronic device. 302 depicts a noisy environment. The user uses the wake-up word "Hey Bixby®". The voice assistant fails to recognize the actual wake-up word spoken in a noisy environment. 304 represents a Bixby® enabled device where Bixby® fails to recognize the wake-up word.

[0016] Figure 4 A graphical representation is shown depicting a scenario 400 for learning wake word recognition according to the related art.

[0017] refer to Figure 4 , the AI-enabled refrigerator learns to recognize real wake-up situations in a noisy environment. 402 represents the interaction between the AI-enabled refrigerator and the user. The user uses the wake-up word (i.e., Hey Bixby®) to trigger Bixby®. The user is in a noisy environment where the blender is running. Bixby® responds to the user based on the learned knowledge. When the blender is running, the wake-up word recognition improves over time in the kitchen. Therefore, the AI-enabled refrigerator can recognize the wake-up word even when the blender is running. 404 represents the interaction between the smart phone and the user. The user uses the wake-up word (i.e., Hey Bixby®) to trigger Bixby®. The user is in a noisy environment where the blender is running. However, in the same noisy environment, Bixby® in the smart phone cannot recognize the wake-up word because the learned knowledge from the AI-enabled refrigerator is not transmitted to the smart phone.

[0018] Therefore, the related art solutions for implementing voice assistants encounter multiple problems in detecting the wake-up command of the voice assistant. Specifically, the related art solutions fail to identify real wake-ups and false wake-ups. In addition, the on-device wake-up model for the voice assistant is not tuned specifically for each user, which leads to false wake-ups due to various noise signals (such as ambient noise, conversations, or electrical equipment noise present in the environment). Custom wake-up models are weak in wake-up word recognition.

[0019] Therefore, a technology to overcome the above problems is needed.

[0020] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure. Summary of the invention

[0021] Solution to the problem

[0022] Aspects of the present disclosure are intended to address at least the above-mentioned problems and / or disadvantages and to provide at least the advantages described below. Therefore, an aspect of the present disclosure is to provide a system and method for detecting a wake-up command of a voice assistant.

[0023] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0024] According to aspects of the present disclosure, a method for detecting a wake-up command of a voice assistant implemented in a user equipment (UE) is provided. The method includes receiving an audio signal from one or more sources. The one or more sources include at least one of a user of the UE or one or more environmental elements. In addition, the method includes determining at least one of an acoustic parameter of the user or an environmental context based on the received audio signal. In addition, the method includes generating an embedding vector representation associated with the received audio signal by using an embedding generator model based on machine learning (ML), based on at least one of the determined acoustic parameters and the determined environmental context. The method also includes comparing the generated embedding vector representation with one or more pre-stored embedding vector representations. In addition, the method includes detecting a wake-up command in the received audio signal based on the determined environmental context and the comparison of the generated embedding vector representation with one or more pre-stored embedding vector representations.

[0025] According to another aspect of the present disclosure, a system for detecting a wake-up command of a voice assistant implemented in a UE is provided. The system includes one or more processors configured to receive an audio signal from one or more sources. The one or more sources include at least one of a user of the UE or one or more environmental elements. The one or more processors are also configured to determine at least one of an acoustic parameter of the user or an environmental background based on the received audio signal. In addition, the one or more processors are configured to generate an embedded vector representation associated with the received audio signal by using an ML-based embedding generator model, based on at least one of a determined acoustic parameter or a determined environmental background. The one or more processors are configured to compare the generated embedded vector representation with one or more pre-stored embedded vector representations. In addition, the one or more processors are configured to detect a wake-up command in the received audio signal based on a determined environmental background and a comparison of the generated embedded vector representation with one or more pre-stored embedded vector representations.

[0026] Other aspects, advantages, and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description, which, in conjunction with the accompanying drawings, discloses various embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0028] Figure 1A A block diagram for executing a wake-up command of a voice assistant in an environment according to the related art is shown;

[0029] Figure 1B A block diagram depicting on-device wake-up fingerprinting according to related art is shown;

[0030] Figure 2 A graphical representation depicting a scenario of a false wakeup of a voice assistant in an environment according to the related art is shown;

[0031] Figure 3 A graphical representation depicting a scenario of actual wakeup rejection according to the related art is shown;

[0032] Figure 4 A graphical representation depicting a scenario for learning wake word recognition according to the related art is shown;

[0033] Figure 5 A block diagram of a system for detecting a wake-up command of a voice assistant according to an embodiment of the present disclosure is shown;

[0034] Figure 6A block diagram showing multiple modules of a system for detecting a wake-up command of a voice assistant at a user equipment (UE) according to an embodiment of the present disclosure is shown;

[0035] Figure 7 A block diagram depicting the operation of a system for detecting a wake-up command for a voice assistant according to an embodiment of the present disclosure is shown;

[0036] Figure 8 A block diagram depicting the operation of a system for detecting a wake-up command for a voice assistant according to an embodiment of the present disclosure is shown;

[0037] Fig. 9 shows a graphical depiction representing a scenario for detecting a wake-up command in a noisy environment according to an embodiment of the present disclosure;

[0038] Fig.10 A graphical depiction representing a scenario for transmitting one or more pre-stored embedding vector representations from a UE to one or more other UEs according to an embodiment of the present disclosure is shown;

[0039] Fig.11 shows a graphical depiction representing a scene represented at different locations using one or more pre-stored embedding vectors in accordance with an embodiment of the present disclosure;

[0040] Fig.12 shows a graphical depiction representing a scenario for detecting false wakeups according to an embodiment of the present disclosure;

[0041] Fig.13 shows a graphical depiction representing a scenario for recommending an alternative wake-up word according to an embodiment of the present disclosure;

[0042] Fig.14 A process flow chart for recommending alternative wake-up words according to an embodiment of the present disclosure is shown;

[0043] Fig.15 shows a graph depicting a trend of a prediction score associated with each wake-up command in a wake-up command set according to an embodiment of the present disclosure; and

[0044] Fig.16 A process flow depicting a method for detecting a wake-up command for a voice assistant according to an embodiment of the present disclosure is shown.

[0045] Throughout the drawings, like reference numerals will be understood to refer to like parts, components and structures. DETAILED DESCRIPTION

[0046] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. It includes various specific details to assist in understanding, but these details are considered to be exemplary only. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0047] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0048] It should be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0049] References throughout this specification to "aspect," "another aspect," or similar language mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases "in an embodiment," "in another embodiment," and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0050] The terms "comprises", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process or method that includes a list of steps includes not only those steps but may also include other steps not expressly listed or inherent to such process or method. Similarly, one or more devices or subsystems or elements or structures or components preceded by "comprises ..." do not exclude the presence of other devices or other subsystems or other elements or other structures or other components or additional devices or additional subsystems or additional elements or additional structures or additional components, without more constraints.

[0051] Figure 5 A block diagram 500 of a system 502 for detecting a wake-up command for a voice assistant is shown according to an embodiment of the present disclosure.

[0052] refer to Figure 5In an embodiment of the present disclosure, a voice assistant is a digital assistant that uses speech recognition and natural language processing (NLP) technology to provide information and perform tasks for a user through voice commands. For example, tasks may include, but are not limited to, providing weather updates, answering questions, playing music, setting alarms and reminders, sending messages, making calls, controlling smart home devices, etc. In an embodiment of the present disclosure, the system 502 is implemented in a user equipment (UE) 504.

[0053] In an embodiment of the present disclosure, UE 504 may correspond to an electronic device in which a voice assistant is implemented. Examples of UE 504 may include, but are not limited to, a refrigerator, a television, a smart phone, a laptop computer, a speaker, a smart watch, and the like. In an embodiment of the present disclosure, a user may operate UE 504 in an environment in which one or more objects 506 are located near UE 504. One or more objects 506 generate an audio signal 508, which may be considered as the noise of a voice assistant implemented at UE 504. For example, one or more objects 506 may include a grinder, a television, a radio, a speaker, and the like. In addition, one or more activities near UE 504 may also generate an audio signal 508. For example, one or more activities may include people talking, thunder, traffic noise, and the like.

[0054] System 502 may include one or more processors / controllers 510 , an input / output (I / O) interface 512 , a number of modules 514 , and a memory 516 .

[0055] In an embodiment of the present disclosure, one or more processors / controllers 510 may be operably coupled to each of the corresponding I / O interfaces 512, the plurality of modules 514, and the memory 516. In one embodiment of the present disclosure, the one or more processors / controllers 510 may include at least one data processor for executing a process in a virtual storage area network. The one or more processors / controllers 510 may include a dedicated processing unit, such as an integrated system (bus) controller, a memory management control unit, a floating point unit, a graphics processing unit, a digital signal processing unit, etc. In one embodiment of the present disclosure, the one or more processors / controllers 510 may include a central processing unit (CPU), a graphics processing unit (GPU), or both. The one or more processors / controllers 510 may be one or more general-purpose processors, digital signal processors, application-specific integrated circuits, field programmable gate arrays, servers, networks, digital circuits, analog circuits, combinations thereof, or other devices now known or later developed for analyzing and processing data. The one or more processors / controllers 510 may execute software programs, such as manually generated (i.e., programmed) code, to perform desired operations. In an embodiment of the present disclosure, the processor / controller may be a general-purpose processor (such as a CPU, an application processor (AP), etc.), a graphics processing unit (such as a GPU), a vision processing unit (VPU), and / or an artificial intelligence (AI) dedicated processor (such as a neural processing unit (NPU)).

[0056] In addition, the one or more processors / controllers 510 control the processing of input data according to predefined operating rules or machine learning (ML) models stored in non-volatile memory and volatile memory. The predefined operating rules or ML models are provided by training or learning.

[0057] Here, providing by learning means making a ML model of predefined operation rules or desired characteristics by applying learning techniques to a plurality of learning data. The learning may be performed in the device itself in which the ML according to the embodiment is performed, and / or may be implemented by a separate server / system.

[0058] In addition, the ML model can be composed of multiple neural network layers. Each layer has multiple weight values, and the layer operation is performed by the calculation of the previous layer and the operation of multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0059] A learning technique is a method for training a predetermined target device (e.g., a robot) using a plurality of learning data to enable, allow, or control the target device to make a determination or prediction. Examples of learning techniques include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0060] The one or more processors / controllers 510 may be arranged to communicate with one or more input / output (I / O) devices via corresponding I / O interfaces 512. The I / O interfaces 512 may employ communications code division multiple access (CDMA), high speed packet access (HSPA+), global system for mobile communications (GSM), long term evolution (LTE), WiMax, etc.

[0061] One or more processors / controllers 510 may be arranged to communicate with a communication network via a network interface. In an embodiment of the present disclosure, the network interface may be an I / O interface. The network interface may be connected to a communication network to enable connection of UE 504 with one or more other UEs. The network interface may employ connection protocols including, but not limited to, direct connection, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 802.11 a / b / g / n / x, etc. The communication network may include, but is not limited to, direct interconnection, a local area network (LAN), a wide area network (WAN), a wireless network (e.g., using a wireless application protocol), the Internet, etc. For details on one or more other UEs, refer at least to Figure 6 Set out in further paragraphs of this disclosure.

[0062] In an embodiment of the present disclosure, one or more processors 510 are configured to receive an audio signal from one or more sources. One or more processors 510 are also configured to determine the acoustic parameters, environmental background, or a combination thereof of the user based on the received audio signal. In addition, one or more processors 510 are configured to generate an embedded vector representation associated with the received audio signal by using an ML-based embedding generator model, based on determined acoustic parameters, determined environmental background, or a combination thereof. One or more processors 510 are configured to compare the generated embedded vector representation with one or more pre-stored embedded vector representations. In addition, one or more processors 510 are configured to detect a wake-up command in the received audio signal based on a determined environmental background and a comparison of the generated embedded vector representation with one or more pre-stored embedded vector representations.

[0063] In some embodiments of the present disclosure, the memory 516 may be communicatively coupled to one or more processors / controllers 510. The memory 516 may be configured to store data and instructions executable by one or more processors / controllers 510. The memory 516 may include, but is not limited to, non-transitory computer-readable storage media, such as various types of volatile and non-volatile storage media, including but not limited to random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, tape or disk, optical media, etc. In one example, the memory 516 may include a cache or random access memory for one or more processors / controllers 510. In an alternative example, the memory 516 is part of one or more processors / controllers 510, such as a cache memory, system memory, or other memory of a processor. In some embodiments of the present disclosure, the memory 516 may be an external storage device or database for storing data. The memory 516 may be operable to store instructions executable by one or more processors / controllers 510. The functions, actions, or tasks shown or described in the figures may be performed by a processor / controller programmed to execute instructions stored in the memory 516. The functions, actions or tasks are independent of a particular type of instruction set, storage medium, processor or processing strategy and may be performed by software, hardware, integrated circuits, firmware, microcode, etc. operating alone or in combination. Likewise, processing strategies may include multi-processing, multi-tasking, parallel processing, etc.

[0064] In some embodiments of the present disclosure, multiple modules 514 may be included in the memory 516. The memory 516 may also include a database 518 for storing data. The multiple modules 514 may include an instruction set that can be executed to cause the system to perform any one or more of the methods / processes disclosed herein. The multiple modules 514 may be configured to use the data stored in the database 518 for detecting the wake-up command of the voice assistant to perform the steps of the present disclosure, as discussed herein. In an embodiment of the present disclosure, each of the multiple modules 514 may be a hardware unit that may be external to the memory 516. In addition, the memory 516 may include an operating system 520 for executing one or more tasks of the system 502, such as executed by a general-purpose operating system in the communication domain. In one embodiment of the present disclosure, the database 518 may be configured to store the information required by the multiple modules 514 and one or more processors / controllers 510 to detect the wake-up command of the voice assistant.

[0065] In an embodiment of the present disclosure, at least one of the plurality of modules 514 may be implemented by an ML model. Functions associated with ML may be performed by a non-volatile memory, a volatile memory, and one or more processors 510.

[0066] In an embodiment of the present disclosure, the I / O interface 512 may implement input to and output from the system 502 using appropriate devices such as, but not limited to, a display, keyboard, mouse, touch screen, microphone, speaker, and the like.

[0067] In addition, the present disclosure also contemplates a computer-readable medium, which includes instructions or receives and executes instructions in response to a propagated signal. In addition, instructions can be sent or received over a network via a communication port or interface or using a bus (not shown). The communication port or interface can be a part of one or more processors / controllers 510, or can be a separate component. The communication port can be created in software, or can be a physical connection in hardware. The communication port can be configured to be connected to a network, an external medium, a display, or any other component in UE 504, or a combination thereof. The connection to the network can be a physical connection, such as a wired Ethernet connection, or can be established wirelessly. Similarly, additional connections to other components of UE 504 can be physical or can be established wirelessly. The network can be directly connected to the bus instead. For the sake of brevity, the architecture and standard operation of the operating system 520, the memory 516, the database 518, the one or more processors / controllers 510 and the I / O interface 512 are not discussed in detail.

[0068] Figure 6 A block diagram 600 showing multiple modules of a system for detecting a wake-up command of a voice assistant at a UE according to an embodiment of the present disclosure is shown.

[0069] refer to Figure 6 In an embodiment of the present disclosure, the plurality of modules 514 may include but are not limited to a receiving module 602, a determining module 604, a generating module 606, a comparing module 608, a detecting module 610, a sending module 612, a training module 614, a storing module 616, and a recommending module 618. The plurality of modules may be implemented by suitable hardware and / or software applications.

[0070] In an embodiment of the present disclosure, the receiving module 602 of the system 502 may be configured to receive an audio signal / acoustic signal from one or more sources. In an embodiment of the present disclosure, the one or more sources may include, but are not limited to, a user of the UE 504, one or more environmental elements, or a combination thereof. In an embodiment of the present disclosure, the audio signal includes a wake-up signal, a noise signal, or a combination thereof. The wake-up signal facilitates the activation of the voice assistant to perform the user's task. In addition, the noise signal may refer to an unwanted signal that interferes with the wake-up signal so that the UE 504 is difficult to identify the wake-up signal to activate the voice assistant. In an embodiment of the present disclosure, the noise signal is generated by one or more environmental elements. For example, the environmental element corresponds to one or more objects / electrical appliances 506 that generate the noise signal. For example, one or more objects 506 may include a grinder, a television, a radio, a speaker, etc. In addition, one or more environmental elements also include one or more activities occurring near the UE 504. Other examples of the one or more activities may include, but are not limited to, people talking, thunder, traffic noise, etc. In another embodiment of the present customer, one or more objects 506 may also include an artificial voice device capable of generating human voices. For example, a television and a radio may generate one or more human conversations.

[0071] In addition, the determination module 604 may be configured to determine the acoustic parameters, environmental background, or a combination thereof of the user based on the received audio signal. The acoustic parameters associated with the user and / or the user's environment may include, but are not limited to, the pitch, intensity, amplitude, or any combination thereof of the received audio signal. In addition, the determined environmental background includes information associated with the occurrence of one or more objects 506 located near the UE 504 and one or more activities near the UE 504 that generates the audio signal. The information includes identification information, location information, and state information. In an embodiment of the present disclosure, the identification information includes the type of one or more objects 506. For example, the type of one or more objects 506 may be a refrigerator, a television, etc. In addition, the state information corresponds to information associated with the state of one or more objects 506. In an embodiment of the present disclosure, the state of one or more objects 506 corresponds to an open state and / or a closed state. The location information includes information associated with the location of one or more objects 506 in the user environment. For example, the location information associated with the grinder is that the grinder is located inside the kitchen. In another embodiment of the present disclosure, the location of one or more objects 506 may be defined as a geographic location, a point of interest (POI), an indoor map location, etc.

[0072] Thereafter, the generation module 606 may be configured to generate an embedding vector representation associated with the received audio signal by using an embedding generator model based on machine learning (ML), based on the determined acoustic parameters, the determined environmental background, or a combination thereof. In an embodiment of the present disclosure, the embedding generator model based on ML is trained offline and deployed to the UE 504. Thereafter, the ML-based embedding generator model may be fine-tuned by dynamic learning. During training, the ML-based embedding generator model maximizes the gap / margin between each cluster based on the source category of the noise.

[0073] In addition, the comparison module 608 can be configured to compare the generated embedding vector representation with one or more pre-stored embedding vector representations. In an embodiment of the present disclosure, one or more pre-stored embedding vector representations correspond to vector representations associated with a combination of a true wake-up set and one or more environmental contexts of the user, a combination of a false wake-up set and one or more environmental contexts of the user, or a combination thereof. The true wake-up set corresponds to instances in which the voice assistant is activated and responds to voice commands without any manual initiation from the user. In addition, the false wake-up set corresponds to instances in which the voice assistant mistakenly activates and responds to commands or queries when the user does not intend to interact with the voice assistant. False wake-up occurs when the voice assistant misinterprets a sound or word from the environment as a wake-up command, prompting the voice assistant to activate and listen for further instructions or questions.

[0074] The detection module 610 may be configured to detect a wake-up command in the received audio signal based on the determined environmental context and a comparison of the generated embedding vector representation with one or more pre-stored embedding vector representations. When detecting a wake-up command in the received audio signal, the detection module 610 may be configured to classify the received audio signal into a true wake-up instance or a false wake-up instance by using a trained ML-based classification model. In an embodiment of the present disclosure, the trained ML-based classification model is a light-weight ML or deep learning model, which may refer to an embedded vector representation of a filter based on noise source clustering. In an embodiment of the present disclosure, an ML model (such as a random forest) may be used to perform on-device training. In addition, as the margin between the clustered embedding vector representations increases, the on-device classification of the received audio signal into a true wake-up instance or a false wake-up instance may be improved. Therefore, given a noisy environment, the accuracy of wake-up command detection increases.

[0075] In an embodiment of the present disclosure, the sending module 612 is configured to send one or more pre-stored embedded vector representations associated with UE504 from UE 504 to one or more other UEs associated with the user via one or more known technologies. In an embodiment of the present disclosure, the one or more other UEs are electronic devices that do not include UE504 in which a voice assistant is implemented. For example, the one or more other UEs may be a refrigerator, a television, a smart phone, a laptop computer, etc. Therefore, the one or more other UEs can now detect the wake-up command in the audio signal based on the learning of the UE. In an embodiment of the present disclosure, the one or more technologies may correspond to a wire-based communication technology, a wireless-based communication technology, etc.

[0076] In addition, the training module / linker module 614 may be configured to determine the occurrence of a real wake-up instance or a false wake-up instance for the generated embedding vector representation via a voice assistant. When determining the occurrence of one of the real wake-up instance or the false wake-up instance, the training module 614 may be configured to receive a wake-up command set from the user. The wake-up command set can be used to activate the voice assistant on the UE 504 to perform the user's task. For example, the wake-up command set may be "wake up", "hello", etc. In addition, the training module 614 may be configured to predict the score of each wake-up command in the wake-up command set. The training module 614 may be configured to determine whether the predicted score is less than a predefined threshold score. In an embodiment of the present disclosure, the score is typically a probability value in the range of 0.0 to 1.0. The predefined threshold within this range can determine whether the wake-up word is likely to be rejected or accepted. For example, for a threshold of 0.9, any generated prediction score greater than 0.9 will cause the wake-up to be accepted, otherwise it will be rejected. In addition, the training module 614 may be configured to reject the wake-up command set when it is determined that the predicted score is less than the predefined threshold score. The training module 614 can be configured to identify patterns associated with the prediction of the score and the rejection of the wake-up command set based on the generated score and the predefined threshold score. In addition, the training module 614 can be configured to determine the number of attempts by the user to wake up the voice assistant based on the identified pattern.

[0077] Thereafter, the training module 614 may be configured to perform an acoustic operation set on the audio signal associated with the received wake-up command based on the predicted number of attempts for a predefined time period. In an embodiment of the present disclosure, the acoustic operation set includes noise suppression, speech enhancement, speech filtering, etc. In addition, the training module 614 may be configured to monitor the identified pattern associated with the prediction of the score to evaluate the audio signal when performing the acoustic operation set. In an embodiment of the present disclosure, the audio signal is evaluated at the time point of the predicted number of attempts. The training module 614 may also be configured to determine whether the wake-up signal exists in the received wake-up command set based on the evaluation result of the audio signal by using an ML model based on signal detection. The training module 614 may be configured to generate an embedded vector representation associated with the audio signal when determining that the wake-up signal exists in the received wake-up command set. In addition, the training module 614 may be configured to associate a real wake-up instance with the generated embedded vector representation. The training module 614 may be configured to store the generated embedded vector representation in the embedded vector repository when the real wake-up instance is associated with the generated embedded vector representation.

[0078] In addition, the training module 614 can be configured to associate the determined real wake-up instance or false wake-up instance with the generated embedding vector representation, the determined acoustic parameters, and the determined environmental background. The training module 614 can also be configured to store the generated embedding vector representation in the embedding vector repository when associating the determined real wake-up instance or one of the false wake-up instances with the generated embedding vector representation, the determined acoustic parameters, and the determined environmental background. The training module 614 can be configured to train the ML-based classification model by using the embedding vector repository when storing the generated embedding vector representation in the embedding vector repository. In addition, Tables 2 and 3 are tables generated by the training module 614 based on associating the determined real wake-up instance or one of the false wake-up instances with the generated embedding vector representation, the determined acoustic parameters, and the determined environmental background. In addition, Table 4 shows sample inputs stored in the embedding vector repository.

[0079]

[0080]

[0081]

[0082] In an embodiment of the present disclosure, the storage module 616 may be configured to determine that the voice assistant cannot obtain a natural label of the generated embedded vector representation of the audio signal. In addition, the storage module 616 may be configured to generate a score for the wake-up command in the audio signal based on the number of wake-up attempts, the embedded vector representation, the user's acoustic parameters, and the environmental background. The storage module 616 may also be configured to associate the generated score with the embedded vector representation, the wake-up command, the acoustic parameters, and the environmental background. In addition, the storage module 616 may be configured to store the generated embedded vector representation in the embedded vector repository when the generated score is associated with the embedded vector representation, the wake-up command, the acoustic parameters, and the environmental background.

[0083] In addition, the recommendation module 618 can be configured to determine that the voice assistant cannot determine the occurrence of a real wake-up instance or a false wake-up instance for the embedded vector representation of the audio signal. The recommendation module 618 can also be configured to obtain an alternative wake-up command from the embedded vector repository based on the embedded vector representation, acoustic parameters associated with the audio signal, environmental background, and the generated score. The recommendation module 618 can be configured to recommend the obtained alternative wake-up command to the user via one or more modes. For example, when the user uses a "wake-up" phrase as a wake-up command and the voice assistant fails to determine the occurrence of a real wake-up instance or a false wake-up instance, the user can use a "hello" phrase instead of the "wake-up" phrase as an alternative wake-up command. In an embodiment of the present disclosure, one or more modes may correspond to displaying an alternative wake-up command on the user interface screen of the UE, outputting an alternative wake-up command in the form of a voice prompt, and the like. In addition, at least with reference to Fig.13 and Fig.14 , details about recommending alternative wake-up commands to the user have been explained in subsequent paragraphs of this specification.

[0084] In addition, at least reference has been made Figure 7 and Figure 8 Details regarding the operation of system 502 are set forth in subsequent paragraphs of this specification. Figures 9 to 13 Details regarding use case scenarios for system 502 are set forth in subsequent paragraphs of this specification.

[0085] Figure 7 A block diagram depicting the operation of a system for detecting a wake-up command for a voice assistant according to an embodiment of the present disclosure is shown.

[0086] Figure 8 A block diagram depicting the operation of a system for detecting a wake-up command of a voice assistant according to an embodiment of the present disclosure is shown. Figure 7 and Figure 8 . refer to Figure 5 and Figure 6A system 502 for detecting a wake-up command for a voice assistant is described.

[0087] refer to Figure 7 and Figure 8 , as depicted, 702 represents a voice signal (i.e., a real wake-up signal generated by a user or a false wake-up signal generated by an artificial voice device). In addition, 704 represents a noise signal generated from one or more environmental elements. In an embodiment of the present disclosure, an audio / acoustic signal 706 representing a superimposed voice signal and environmental noise is input into the system 502. As depicted, 708 represents an environmental context, which includes information associated with one or more objects 506 located near the UE 504 and the occurrence of one or more activities near the UE 504 that generates the audio signal. In an embodiment of the present disclosure, the environmental context 708 may be: the electrical device that generates the noise may be a blender, the location of the electrical device may be a kitchen, and the state of the electrical device is turned on. In an embodiment of the present disclosure, data 709 is associated with the noise signal 704, and the acoustic signal 706 corresponds to audio data and a sampling rate. For example, the audio data may be a byte buffer, and the sampling rate may be a 16KHz Mel Frequency Cepstral Coefficient (MFCC) spectrogram.

[0088] In addition, 710 represents the learning pipeline of the system 502, and 712 represents the inference pipeline of the system 502. The generation module 606 generates an embedding vector representation 713 for the received audio signal associated with speech and environmental noise. In an embodiment of the present disclosure, the speech may be related to a user or an artificial speech device. In addition, the voice assistant 714 is used to determine a real wake-up instance or a false wake-up instance for the generated embedding vector representation. The voice assistant 714 uses out-of-vocabulary (OOV) and false alarm (FA). In an embodiment of the present disclosure, OOV corresponds to all kinds of acoustics that are different from real wake-ups. In an embodiment of the present disclosure, FA corresponds to acoustic words that sound similar to real alarm situations. In addition, the linker module / training module 614 removes noise from the audio signal and associates the determined real wake-up instance or the determined false wake-up instance with the generated embedding vector representation, acoustic parameters, and environmental background 708. In addition, table 716 represents a table with positions and noise targets, and table 718 represents a table with embedding vectors and types of wake-up instances (natural labels, as shown). These tables are generated by the linker module 614. In addition, when associating the determined real wake-up instance or the determined false wake-up instance with the generated embedding vector representation, acoustic parameters, and environmental background 708, the linker module 614 stores the generated embedding vector representation in the embedding vector repository 720. In an embodiment of the present disclosure, one or more pre-stored embedding vector representations stored in the embedding vector repository 720 are represented by 722. The determined real wake-up instance or the determined false wake-up instance associated with the generated embedding vector representation, acoustic parameters, and environmental background is represented by 724.

[0089] In addition, in the inference pipeline 712, the generation module 606 generates an embedding vector representation associated with the audio signal based on acoustic parameters, environmental context, or a combination thereof. The detection module 610 classifies the received audio signal into a wakeup decision 725, i.e., a real wakeup instance or a false wakeup instance, by using the trained ML-based classification model.

[0090] In an embodiment of the present disclosure, the wake-up recommender 726 is used to determine an alternative wake-up word 728 that has a greater chance of activating the voice assistant than the existing wake-up word. Fig.14 and Fig.15 Provides details about the operation of alternative wake words.

[0091] Fig. 9 A graphical depiction representing a scenario for detecting a wake-up command in a noisy environment according to an embodiment of the present disclosure is shown.

[0092] refer to Figure 5and Figure 6 A system 502 for detecting a wake-up command for a voice assistant is described.

[0093] refer to Fig. 9 As depicted, 902 corresponds to the coffee machine being running, thereby creating noise in the user's environment. 904 corresponds to the user providing a wake-up command to the voice assistant to listen to the latest news. Further, at 906, the voice assistant fails to recognize the wake-up command due to the noise in the environment caused by the coffee machine. In addition, the system 502 performs multiple processes in the learning pipeline 710, as shown in reference Figure 7 As explained. For example, multiple processes in the learning pipeline may include generating an embedding vector representation, associating a real wake-up instance or a false wake-up instance with the embedding vector representation and an environmental context, and storing the embedding vector representation, the type of the wake-up instance (i.e., a real wake-up instance or a false wake-up instance), the environmental context, etc. in the form of a table in an embedding vector repository, as shown in 908. In the current scenario, the environmental context includes that the electrical device is a coffee machine, the location is the kitchen, and the state is on. The embedding vector repository includes one or more pre-stored embedding vector representations 910, such as a real wake-up embedding for a speaker, a real wake-up embedding for a coffee machine, a real wake-up embedding for an air conditioner, a real wake-up embedding for a blender, etc.

[0094] In addition, 912 corresponds to the coffee machine generating noise in the user's environment. 914 corresponds to the user providing a wake-up command to the voice assistant for reading the latest news. In addition, 916 corresponds to the voice assistant. The voice assistant successfully identifies the wake-up command as a wake-up decision 918 in the noisy environment by performing multiple processes in the inference pipeline 712, as shown in reference Figure 7 Explained.

[0095] Fig.10 A graphical depiction showing a scenario for transmitting one or more pre-stored embedding vector representations from a UE to one or more other UEs according to an embodiment of the present disclosure is shown.

[0096] refer to Figure 5 and Figure 6 A system 502 is described for transmitting one or more pre-stored embedding vector representations from a UE 504 to one or more other UEs.

[0097] In the current scenario, the user drives a car and has a smartphone. The smartphone has learned robust wake-up detection in traffic. For a picnic plan, the user carries a speaker with him in the car. In rush hour traffic, the user interacts with the speaker, trying to wake up the voice assistant in the speaker. However, the speaker is unable to recognize the true intended wake-up signal.

[0098] refer to Fig.10 , 1002 corresponds to a user. The user provides a wake-up command to the smartphone to view alternative routes to the home. 1004 corresponds to a voice assistant on the smartphone. The voice assistant searches for alternative routes based on learning performed by the smartphone and displays them to the user for wake-up recognition in traffic over a period of time.

[0099] In addition, 1006 corresponds to a user. The user provides a wake-up command to the speaker to view alternative routes to the home. 1008 corresponds to a voice assistant on the speaker. Even though the smartphone has learned over a period of time, the voice assistant is unable to recognize the expected wake-up signal in the traffic noise. In addition, the system 502 performs a transmission of the learned embedding vector representation from the smartphone to the speaker based on the background demand, as shown in 1010. In an embodiment of the present disclosure, the background demand corresponds to background adaptation. The UE in traffic needs to adapt the wake-up detection in the traffic environment. To this end, the embedding needs to be transmitted to other UE devices.

[0100] In addition, 1012 corresponds to a user. The user again provides a wake-up command to the speaker to view an alternative route to the home. In addition, 1014 corresponds to the speaker. When the learned embedding vector representation is transmitted from the smartphone to the speaker, the speaker recognizes the wake-up signal in the traffic noise.

[0101] In another scenario, multiple UEs are configured for the same user account. In the next step, acoustic parameters and environmental background are generated for each of the multiple UEs. In the next step, the embedded vector representation of each of the multiple UEs is stored in the embedded vector repository, classified by environmental background. In addition, the embedded vector representation is uploaded to a common server for aggregation. In the next step, the aggregated embedded vector representation is downloaded to the device repository of each of the multiple UEs. Therefore, multiple UEs can use each other's learned embedded vector representation.

[0102] Fig.11 A graphical depiction is shown representing a scene represented at different locations using one or more pre-stored embedding vectors in accordance with an embodiment of the present disclosure.

[0103] refer to Figure 5 and Figure 6 A system 502 for detecting a wake-up command for a voice assistant is described. Fig.11 Depicted is the use of one or more pre-stored embedding vector representations at different locations to detect a wake-up command for a voice assistant.

[0104] refer to Fig.11, in the current scenario, the coffee machine is located in the user's home kitchen. At operation 1102, the system 502 receives a real wake-up signal at home when the coffee machine is turned on. In addition, the system 502 performs multiple processes of the learning pipeline 710 for learning wake-up detection with coffee machine noise at home based on environmental context. In the current scenario, the environmental context may include that the electrical device is a coffee machine, the location of the electrical device is a kitchen, and the state is turned on.

[0105] At operation 1104, system 502 receives a real wake-up signal at the user's office when the coffee machine is turned on. In addition, system 502 performs multiple processes of inference pipeline 712 based on the results of learning the wake-up detection with coffee machine noise at home. Multiple processes of the inference pipeline are performed based on the user's environmental context. In the current scenario, the environmental context may include that the electrical device is a coffee machine, the location of the electrical device is an office coffee break, and the state is turned on. In addition, 1106 represents one or more pre-stored embedding vector representations and the inferred embedding vector at the office.

[0106] Fig.12 A graphical depiction representing a scenario for detecting false wakeups according to an embodiment of the present disclosure is shown.

[0107] refer to Figure 6 A system 502 for detecting spurious wakeups is described.

[0108] refer to Fig.12 , in the current scenario, the user is watching a TV show in which the host talks about the voice assistant. The host mentions the voice assistant several times during his speech. This causes the voice assistant in the user's smartphone to be awakened several times. 1202 represents a TV set. The host in the TV is talking about the voice assistant, that is, "the voice assistant is a very powerful tool." At operation 1204, it represents the voice assistant. The voice assistant is activated and responds "I don't understand." In addition, the system 502 performs multiple processes in the learning pipeline 710 based on the user's environmental context, as shown in reference Figure 7 In the current scenario, the environmental context may include that the electrical device is a television, the location of the electrical device is a living room, and the state is turned on.

[0109] Additionally, 1206 depicts the host on TV talking about the voice assistant, namely, “the voice assistant is a very powerful tool.” 1208 depicts the voice assistant no longer waking up to a false wakeup triggered by the TV speaker by executing multiple processes of the inference pipeline 712 based on the environmental context. Thus, the system 502 uses the environmental context to learn from the false wakeups and thus reject similar wakeup acoustic signals.

[0110] Fig.13A graphical depiction representing a scenario for recommending an alternative wake-up word according to an embodiment of the present disclosure is shown.

[0111] refer to Figure 6 A system 502 for recommending alternative wake words is described.

[0112] In the current scenario, the user is in a home kitchen where one or more appliances (such as a coffee maker, blender, etc.) are turned on. In this noisy situation, the user attempts to initiate a voice interaction with the voice assistant. Due to the ambient noise, the voice assistant cannot recognize the wake-up signal because it rejects the request for voice interaction. In an embodiment of the present disclosure, the system rejects the user's request for one or more reasons, such as the voice assistant's automatic speech recognition module failing to generate speech-to-text, the voice assistant's natural language understanding module rejecting invalid utterances, and partial information in the voice assistant's response. System 502 detects consecutive attempts and recommends alternative wake-up words to activate the voice assistant.

[0113] refer to Fig.13 As depicted, 1302 depicts that a coffee machine is running and creating noise in the user's environment. 1304 represents a user. The user provides a wake-up command to the voice assistant for listening to the latest news. For example, the wake-up command is "Hey voice assistant". In addition, 1306 represents the voice assistant. Due to the noise in the environment caused by the coffee machine, the voice assistant fails to recognize the wake-up command for consecutive attempts. In addition, the system 502 performs multiple processes in the learning pipeline 710, as shown in reference Figure 7 As explained. In the current scenario, the system 502 detects continuous attempts by the user to activate the voice assistant, and recommends alternative wake-up words that can work based on the environmental context by using the recommendation module 618. In the current scenario, the environmental context includes that the electrical device is a coffee machine, the location is the kitchen, and the state is turned on. In an embodiment of the present disclosure, the recommendation module 618 of the learning pipeline generates a table 1308, which includes an embedded vector representation, a wake-up word, a score associated with each wake-up word, the type of wake-up instance, a noise source, a location, etc. In addition, 1310 represents a cluster matching the environmental context. In the cluster matching environmental context, the system 502 obtains an alternative wake-up command from the embedding vector repository. The embedding vector repository includes multiple vector representations corresponding to the wake-up word and the environmental context. For example, multiple vector representations include "Hey Voice Assistant Embedded Coffee Machine" 1312, "Hello Voice Assistant Embedded Coffee Machine" 1314, "Hey Voice Assistant Embedded Mixer" 1316, and "Hey Fold Embedded AC" 1318.

[0114] When obtaining an alternative wake-up command from the embedding vector repository, the system 502 determines consecutive failed real wake-up attempts. In addition, the system 502 determines whether the number of consecutive failed real wake-up attempts exceeds a predefined threshold. When it is determined that the number of consecutive failed real wake-up attempts exceeds a predefined threshold, the system 502 identifies one or more possible vector representations from multiple vector representations. Finally, the system 502 identifies a vector representation from one or more vector representations and recommends an alternative wake-up word associated with the identified vector representation. In an embodiment of the present disclosure, the alternative wake-up word comes from a cluster that is fully or partially related to the environmental context. For example, the alternative wake-up command can be "Hello voice assistant". 1320 represents a "Hey voice assistant" reference vector; the failed attempt is located away from the "Hey voice assistant" keyword. 1322 represents an inferred embedding vector.

[0115] In addition, 1324 depicts that the coffee machine is running, thereby generating noise in the user's environment. 1326 represents that the user provides the voice assistant with an alternative wake-up command for reading the latest news. In addition, 1328 represents that the voice assistant successfully recognizes the wake-up command in a noisy environment by executing multiple processes in the inference pipeline 712, as shown in reference Figure 7 Explained.

[0116] In another embodiment of the present disclosure, the user can also manually set one or more alternative wake-up words for a specific wake-up word, so that the user can use one or more alternative wake-up words to activate the voice assistant when the previously used wake-up word does not work. In addition, the system eliminates information associated with the previously used wake-up word from the embedding vector repository. In addition, one or more alternative wake-up words are stored in the embedding vector repository to model a new custom wake-up model for identifying wake-up words. The custom wake-up model is generated using acoustic signals from each user's wake-up command recorded during the interaction with the virtual assistant. Such acoustic signals accumulate over the period of use and interaction with the virtual assistant and represent a wide range of vocabulary. Acoustic signals corresponding to custom wake-up words that match the vocabulary list can be used to build a custom wake-up model.

[0117] Fig.14 A process flow chart for recommending alternative wake-up words according to an embodiment of the present disclosure is shown.

[0118] refer to Figure 6 and Fig.13 A system 502 for recommending alternative wake words is described.

[0119] refer to Fig.14At operation 1402, the system 502 receives a wake-up command set in a noisy environment from a user and predicts a score for each wake-up command in the wake-up command set. The system 502 determines whether the predicted score is less than a predefined threshold score. In addition, at operation 1404, the system 502 rejects the wake-up command base when it is determined that the predicted score is less than the predefined threshold score. In addition, at operation 1406, the system 502 identifies a pattern associated with the prediction of the score and the rejection of the wake-up command set based on the generated score and the predefined threshold score. At operation 1408, the system 502 determines the number of attempts by the user to wake up the voice assistant based on the identified pattern.

[0120] At operation 1410, the system 502 performs an acoustic operation set on the audio signal associated with the received wake-up command based on the predicted number of attempts for a predefined time period. At operation 1412, the system 502 monitors the identified pattern associated with the prediction of the score while performing the acoustic operation set to evaluate the audio signal. The audio signal is evaluated at the time point when the number of attempts is predicted. At operation 1414, the system 502 determines whether the wake-up signal exists in the received wake-up command set based on the evaluation result of the audio signal by using an ML model based on signal detection. In an embodiment of the present disclosure, the system 502 uses a time-sliced ​​audio signal to determine the presence of the wake-up signal. In an embodiment of the present disclosure, the ML model based on signal detection is an accurate and resource-consuming deep neural network (DNN) model. At operation 1416, the system 502 generates an embedded vector representation by using the original audio signal of the time slice without acoustic synthesis when determining that the wake-up signal exists in the received wake-up command set. The embedded vector representation is stored in an embedded vector repository for matching with an upcoming attempt. In addition, at operation 1416, the system 502 recommends an alternative wake-up word that has a greater chance of activating the voice assistant than the existing wake-up word. At operation 1418, the system 502 reverts to its default operation to monitor the wake-up word if no valid wake-up is detected.

[0121] Fig.15 A graph depicting a trend of a prediction score associated with each wake-up command in a wake-up command set according to an embodiment of the present disclosure is shown.

[0122] refer to Figure 6 and Fig.14 Details are described regarding predicting a score associated with each wake-up command in a set of wake-up commands. Fig.15 The trends of the prediction scores are depicted.

[0123] refer to Fig.15 As shown in the figure, 1502 represents the user's wake-up attempt. In the embodiment of the present disclosure, the real wake-up threshold is 0.9.

[0124] Fig.16 A process flow depicting a method for detecting a wake-up command for a voice assistant according to an embodiment of the present disclosure is shown.

[0125] refer to Fig.16 , the method 1600 may be performed by a system 502 implemented in a user equipment (UE) 504, such as Figure 5 as shown in .

[0126] At operation 1602, method 1600 includes receiving an audio signal from one or more sources. In an embodiment of the present disclosure, the one or more sources include a user of UE 504, one or more environmental elements, or a combination thereof. The audio signal includes a wake-up signal, a noise signal, or a combination thereof. In an embodiment of the present disclosure, the noise signal is generated by one or more environmental elements.

[0127] At operation 1604, method 1600 includes determining an acoustic parameter of the user, an environmental context, or a combination thereof based on the received audio signal. In an embodiment of the present disclosure, the acoustic parameter associated with the user and the user's environment includes the pitch, intensity, amplitude, or any combination thereof of the received audio signal. In an embodiment of the present disclosure, the determined environmental context includes information associated with one or more objects located near the UE and the occurrence of one or more activities that generate audio signals near the UE. For example, the information includes identification information, location information, and state information.

[0128] At operation 1606 , the method 1600 includes generating an embedding vector representation associated with the received audio signal based on the determined acoustic parameters, the determined environmental context, or a combination thereof by using a machine learning (ML) based embedding generator model.

[0129] At operation 1608, method 1600 includes comparing the generated embedding vector representation with one or more pre-stored embedding vector representations. In an embodiment of the present disclosure, the one or more pre-stored embedding vector representations correspond to vector representations associated with a combination of a real wake-up set and one or more environmental contexts of the user, a combination of a false wake-up set and one or more environmental contexts of the user, or a combination thereof.

[0130] At operation 1610, method 1600 includes detecting a wake-up command in a received audio signal based on the determined environmental context and a comparison of the generated embedding vector representation with one or more pre-stored embedding vector representations. When detecting the wake-up command in the received audio signal, method 1600 includes classifying the received audio signal as a real wake-up instance or a false wake-up instance by using a trained machine learning (ML)-based classification model.

[0131] In an embodiment of the present disclosure, method 1600 includes sending one or more pre-stored embedding vector representations associated with a UE from the UE to one or more other UEs associated with a user via one or more techniques.

[0132] In addition, method 1600 includes determining, via a voice assistant, the occurrence of a real wake-up instance or a false wake-up instance for the generated embedding vector representation. When determining the occurrence of a real wake-up instance or a false wake-up instance, method 1600 includes receiving a wake-up command set from a user. Method 1600 also includes predicting a score for each wake-up command in the wake-up command set. In addition, method 1600 includes determining whether the predicted score is less than a predefined threshold score. Method 1600 includes rejecting the wake-up command set when it is determined that the predicted score is less than a predefined threshold score. Method 1600 also includes identifying a pattern associated with the prediction of the score and the rejection of the wake-up command set based on the generated score and the predefined threshold score. In addition, method 1600 includes determining the number of attempts by the user to wake up the voice assistant based on the identified pattern.

[0133] In addition, method 1600 includes performing an acoustic operation set on an audio signal associated with the received wake-up command for a predefined time period based on the predicted number of attempts. In an embodiment of the present disclosure, the acoustic operation set includes noise suppression, speech enhancement, and speech filtering. Method 1600 includes monitoring the identified pattern associated with the prediction of the score to evaluate the audio signal when performing the acoustic operation set. The audio signal is evaluated at the time point when the number of attempts is predicted. Method 1600 includes determining whether the wake-up signal exists in the received wake-up command set based on the evaluation result of the audio signal by using an ML model based on signal detection. In addition, method 1600 includes generating an embedded vector representation associated with the audio signal when determining that the wake-up signal exists in the received wake-up command set. Method 1600 includes associating a real wake-up instance with the generated embedded vector representation. In addition, method 1600 includes storing the generated embedded vector representation in an embedded vector repository when associating the real wake-up instance with the generated embedded vector representation.

[0134] In addition, method 1600 includes associating the determined real wake-up instance or the determined false wake-up instance with the generated embedding vector representation, the determined acoustic parameters, and the determined environmental context. Method 1600 includes storing the generated embedding vector representation in an embedding vector repository when associating the determined real wake-up instance or one of the false wake-up instances with the generated embedding vector representation, the determined acoustic parameters, and the determined environmental context. Method 1600 also includes training an ML-based classification model by using the embedding vector repository when storing the generated embedding vector representation in the embedding vector repository.

[0135] Method 1600 includes determining that the voice assistant cannot obtain a natural label for the generated embedded vector representation of the audio signal. In addition, method 1600 includes generating a score for the wake-up command in the audio signal based on the number of wake-up attempts, the embedded vector representation, the acoustic parameters of the user, and the environmental context. Method 1600 includes associating the generated score with the embedded vector representation, the wake-up command, the acoustic parameters, and the environmental context. In addition, method 1600 includes storing the generated embedded vector representation in an embedded vector repository when associating the generated score with the embedded vector representation, the wake-up command, the acoustic parameters, and the environmental context.

[0136] In addition, method 1600 includes determining that the voice assistant cannot determine an occurrence of a real wake-up instance or a false wake-up instance for the embedded vector representation of the audio signal. Method 1600 includes obtaining an alternative wake-up command from the embedded vector repository based on the embedded vector representation, acoustic parameters associated with the audio signal, environmental context, and the generated score. Method 1600 also includes recommending the obtained alternative wake-up command to the user via one or more modes.

[0137] Although described in a specific order Fig.16 The above steps are shown in , but according to various embodiments of the present disclosure, these steps can occur in a variation of the order. In addition, for the sake of brevity, the following will not be discussed here. Fig.16 The details of each step are covered in Figures 6 to 15 Related description.

[0138] The present disclosure provides various technical advances based on the key features discussed above. In addition, the present disclosure detects wake-up commands in noisy environments by using environmental contexts. The present disclosure improves wake-up recognition in different environmental contexts over a period of time. In addition, learning from one device in one context can be applied to another device in a similar or identical context. In addition, the present disclosure also improves the rejection of false wake-up detection and reduces the number of false wake-ups on the device side. Thereby reducing the load on the server automatic speech recognition (ASR) model to determine the wake-up event. The system learns from the user's voice without explicitly routing the user through the registration process. In addition, the embedding vector repository indicates true / false wake-up events in different environmental contexts. The embedding vector library is used to improve the accuracy of wake-up recognition. The system reduces the number of false negative situations (i.e., situations where the user says a wake-up command but the system fails to recognize the wake-up word). In addition, the present disclosure also transmits wake-up improvement learning from one device to another. Over time, the size of the embedding vector library increases. During this period of time, the embedding of the wake-up trigger learning from the user can be used to personalize the model. In this way, the user can use the model more for invoking wake-up in a user-specific environment. For example, if the user primarily uses wake-up triggers in the kitchen, the system can adapt the wake-up triggers from the kitchen. The system facilitates customized wake-up recognition by using environmental context.

[0139] Multiple modules 514 can be implemented by any suitable hardware and / or instruction set. In addition, embodiments can include addition / omission of steps as required. In some embodiments of the present disclosure, one or more operations performed by multiple modules 514 can be performed by a processor / controller based on requirements.

[0140] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. A method implemented in a user equipment (UE) for detecting a wake-up command of a voice assistant, the method comprising: receiving an audio signal from one or more sources, wherein the one or more sources include at least one of a user of the UE or one or more environmental elements; determining at least one of an acoustic parameter of a user or an environmental context based on the received audio signal; generating an embedding vector representation associated with the received audio signal based on at least one of the determined acoustic parameter and the determined environmental context by using a machine learning (ML) based embedding generator model; comparing the generated embedding vector representation to one or more pre-stored embedding vector representations; and A wake-up command in the received audio signal is detected based on the determined environmental context and a comparison of the generated embedding vector representation with the one or more pre-stored embedding vector representations.

2. The method according to claim 1, in, The audio signal includes at least one of a wake-up signal or a noise signal, and The noise signal is generated by the one or more environmental elements.

3. The method according to claim 1, wherein: The acoustic parameters associated with the user and the user's environment include at least one of pitch, intensity, or amplitude of the received audio signal.

4. The method according to claim 1, in, The determined environmental context includes information associated with one or more objects located in the vicinity of the UE and occurrences of one or more activities generating audio signals in the vicinity of the UE, and The information includes identification information, location information and status information.

5. The method according to claim 1, wherein: The one or more pre-stored embedded vector representations correspond to vector representations associated with at least one of a combination of a real arousal set and one or more environmental contexts of a user, or a combination of a false arousal set and the one or more environmental contexts of a user.

6. The method according to claim 5, further comprising: The one or more pre-stored embedding vector representations associated with the UE are transmitted from the UE to one or more other UEs associated with the user via one or more techniques.

7. The method according to claim 1, further comprising: determining, via the voice assistant, an occurrence of one of a real wakeup instance or a false wakeup instance for the generated embedding vector representation; associating one of the determined real wake-up instance or the false wake-up instance with the generated embedding vector representation, the determined acoustic parameter, and the determined environmental context; When associating one of the determined real wake-up instance or the false wake-up instance with the generated embedding vector representation, the determined acoustic parameter, and the determined environmental context, storing the generated embedding vector representation in an embedding vector repository; as well as When the generated embedding vector representation is stored in the embedding vector repository, an ML-based classification model is trained by using the embedding vector repository.

8. The method according to claim 7, wherein: Determining the occurrence of one of a real wakeup instance or a false wakeup instance includes: receiving a wake-up command set from a user; Predict the score of each wake-up command in the wake-up command set; determining whether the predicted score is less than a predefined threshold score; rejecting the wake-up command set upon determining that the predicted score is less than a predefined threshold score; identifying patterns associated with prediction of scores and rejection of wake-up command sets based on the generated scores and predefined threshold scores; and A number of attempts by the user to wake up the voice assistant is determined based on the identified pattern.

9. The method according to claim 8, wherein: Determining the occurrence of one of a real wakeup instance or a false wakeup instance includes: performing a set of acoustic operations on an audio signal associated with the received wake-up command for a predefined period of time based on the predicted number of attempts, wherein the set of acoustic operations includes noise suppression, speech enhancement, and speech filtering; monitoring the identified pattern associated with the prediction of the score while performing the set of acoustic operations to evaluate the audio signal, wherein the audio signal is evaluated at a point in time at which the number of attempts is predicted; determining whether a wake-up signal is present in the received wake-up command set based on an evaluation result of the audio signal using a signal detection-based ML model; upon determining that a wake-up signal is present in the received wake-up command set, generating an embedding vector representation associated with the audio signal; Associating the real wake-up instance with the generated embedding vector representation; and When the real wake-up instance is associated with the generated embedding vector representation, the generated embedding vector representation is stored in an embedding vector repository.

10. The method according to claim 7, further comprising: Determining that the voice assistant is unable to obtain a natural label for the embedded vector representation of the generated audio signal; generating a score for a wake-up command in the audio signal based on the number of wake-up attempts, the embedding vector representation, acoustic parameters of the user, and environmental context; Associating the generated score with the embedding vector representation, the wake-up command, the acoustic parameters, and the environmental context; as well as When the generated score is associated with the embedding vector representation, the wake-up command, the acoustic parameters, and the environmental context, the generated embedding vector representation is stored in an embedding vector repository.

11. The method according to claim 10, further comprising: determining that the voice assistant cannot determine an occurrence of one of a real wakeup instance or a false wakeup instance for the embedding vector representation of the audio signal; obtaining a substitute wake-up command from an embedding vector repository based on the embedding vector representation, acoustic parameters associated with the audio signal, environmental context, and the generated score; as well as The obtained alternative wake-up commands are recommended to the user via one or more patterns.

12. The method according to claim 1, wherein: Detecting a wake-up command in a received audio signal comprises: The received audio signal is classified as one of a real wakeup instance or a false wakeup instance by using the trained ML-based classification model.

13. A system implemented in a user equipment (UE) for detecting a wake-up command of a voice assistant, the system comprising one or more processors, the one or more processors being configured to: An audio signal is received from one or more sources, wherein The one or more sources include at least one of a user of the UE or one or more environmental elements, determining at least one of an acoustic parameter of a user or an environmental context based on the received audio signal, generating an embedding vector representation associated with the received audio signal based on at least one of the determined acoustic parameter and the determined environmental context using a machine learning (ML) based embedding generator model, comparing the generated embedding vector representation to one or more pre-stored embedding vector representations, and A wake-up command in the received audio signal is detected based on the determined environmental context and a comparison of the generated embedding vector representation with the one or more pre-stored embedding vector representations.

14. The system according to claim 13, in, The audio signal includes at least one of a wake-up signal or a noise signal, and The noise signal is generated by the one or more environmental elements.

15. The system of claim 13, wherein: The acoustic parameters associated with the user and the user's environment include at least one of pitch, intensity, or amplitude of the received audio signal.

Images