Systems and methods for dynamically adjusting a listening time of a voice assistant device
Patent Information
- Application Number
- IN202211016620
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-03-24
- Publication Date
- 2026-07-17
- Estimated Expiration
- 2042-03-24
AI Technical Summary
Current voice assistant devices are unable to predict when a user intends to continue speaking, leading to incomplete utterances due to fixed listening times, which increases latency and requires unnecessary reactivation.
A system that dynamically adjusts the listening time of a voice assistant device by analyzing a user's breathing pattern and non-speech artifacts to determine their intention to continue speaking, eliminating the need for repeated wake-up commands.
This solution enhances user experience by allowing continuous conversation without interruptions, improving accuracy and reducing latency by dynamically adjusting the listening time based on the user's breathing patterns and environmental attributes.
Abstract
Description
FIELD OF THE INVENTION
[0001] The present invention generally relates to voice assistant devices, andmore particularly relates to systems and methods for dynamically adjusting apredefined listening time of a voice assistant device.BACKGROUND
[0002] With the recent advancements in the voice recognition field, voiceassistants have found applications in many areas like smart phone, smart TV,smart speaker, smart home, connected cars, robotics etc. They support voicecommands, voice searching, and voice-activated device control. They are able tocapture, detect end point, and execute the user commands. Improvements arecontinuously being done in this field for enhancing user experience.
[0003] Currently available voice assistants are trying to improve latency,accuracy and user experience. In existing voice assistants, users need to againand again activate the mic by saying wake-up words like "Hi <<Voice AssistantName>>", pressing wakeup button, or by clicking on mic, which is veryinconvenient and unnecessary increases the latency in case of continuation andfollow-up commands. Specifically, the existing voice assistants are incapable ofpredicting whether a user intends to speak more during pause phases, and so thevoice assistants stop listening after a static time leading to incomplete utterancefrom user's perspective.
[0004] Accordingly, there is a need for a system which can recognize user'sintention and adjust mic awake time. Further, there is a need for a system whichcan recognize user's intention to speak more and adjust mic listening timeaccordingly.SUMMARY
[0005] This summary is provided to introduce a selection of concepts, in asimplified format, that are further described in the detailed description of theinvention. This summary is neither intended to identify key or essential inventiveconcepts of the invention and nor is it intended for determining the scope of theinvention.
[0006] According to one embodiment of the present disclosure, a method ofdynamically adjusting a predefined listening time of a voice assistant device isdisclosed. The method includes receiving, by the voice assistant device, an audioinput. Further, the method includes extracting one or more of a speechcomponent and a non-speech artifact from the audio input. Furthermore, themethod includes determining a user breathing pattern based on the one or moreof the speech component and the non-speech artifact. Additionally, the methodincludes identifying one or more attributes that impact the user breathing patternbased on at least one non-speech component, captured from the surroundingenvironment and the voice assistant device. Still further, the method includesdetermining, after detecting a pause in the audio input, whether a user's intentionis to continue the conversation based on an analysis of the user breathing patternand the one or more attributes. Finally, the method includes dynamicallyadjusting the predefined listening time of the voice assistant device to continuelistening for voice commands in the conversation based on a determination thatthe user's intention is to continue the conversation.
[0007] According to another embodiment of the present disclosure, a systemfor dynamically adjusting a predefined listening time of a voice assistant deviceis disclosed. The system comprises a receiving module configured to receive anaudio input. Further, the system comprises a breath analysis module configuredto extract one or more of a speech component and a non-speech artifact from theaudio input. The breath analysis module is further configured to determine a userbreathing pattern based on the one or more of the speech component and the non-speech artifact. Additionally, the breath analysis module is configured to identifyone or more attributes that impact the user breathing pattern based on at least onenon-speech component, captured from the surrounding environment and thevoice assistant device. Furthermore, the system comprises a listening timeadjustment module configured to determine, after detecting a pause in the audioinput, whether a user's intention is to continue the conversation based on ananalysis of the user breathing pattern and the one or more attributes. Also, thelistening time adjustment module is configured to dynamically adjust thepredefined listening time of the voice assistant device to continue listening forvoice commands in the conversation based on a determination that the user'sintention is to continue the conversation.
[0008] To further clarify the advantages and features of the present invention,a more particular description of the invention will be rendered by reference tospecific embodiments thereof, which is illustrated in the appended drawings. Itis appreciated that these drawings depict only typical embodiments of theinvention and are therefore not to be considered limiting of its scope. Theinvention will be described and explained with additional specificity and detailwith the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS
[0009] These and other features, aspects, and advantages of the presentinvention will become better understood when the following detailed descriptionis read with reference to the accompanying drawings in which like charactersrepresent like parts throughout the drawings, wherein:
[0010] Figures 1A and 1B illustrate continuation case of conversationaccording to various embodiments of the present invention with respect to theprior art;
[0011] Figures 1C and 1D illustrate follow-up case of conversation accordingto various embodiments of the present invention with respect to the prior art.
[0012] Figure 2A illustrates a schematic block diagram of a system fordynamically adjusting a predefined listening time of the voice assistant device,according to an embodiment of the present invention;
[0013] Figure 2B illustrates another schematic block diagram of a system fordynamically adjusting a predefined listening time of the voice assistant device,according to an embodiment of the present invention;
[0014] Figure 3 illustrates a schematic block diagram of modules of the systemfor dynamically adjusting a predefined listening time of the voice assistantdevice, according to an embodiment of the present invention;
[0015] Figure 4 illustrates an exemplary process of determining user'sintention to speak more based on user's inhalation and exhalation breathingpattern while the user is speaking to a voice assistant, according to oneembodiment of the present invention;
[0016] Figure 5 illustrates an exemplary process of determining user'sintention to speak more based on user's quick exhalation or inhalation while theuser is speaking to a voice assistant, according to one embodiment of the presentinvention;
[0017] Figure 6 illustrates an exemplary process of determining user'sintention to speak more based on user's breath hold while the user is speaking toa voice assistant, according to one embodiment of the present invention;
[0018] Figure 7 illustrates an exemplary process of determining user'sintention to speak more during a continuation utterance based on a combinationof different breathing patterns of the user while the user is speaking to a voiceassistant, according to one embodiment of the present invention;
[0019] Figure 8 illustrates an exemplary process of determining user'sintention to speak more during a follow-up utterance based on a combination ofdifferent breathing patterns of the user while the user is speaking to a voiceassistant, according to one embodiment of the present invention;
[0020] Figure 9 illustrates various exemplary scenarios of user's breathingpatterns in different surroundings while the user is speaking to a voice assistant,according to one embodiment of the present invention; and
[0021] Figure 10 illustrates an exemplary process flow for dynamicallyadjusting a predefined listening time of the voice assistant device, according toan embodiment of the present invention.
[0022] Further, skilled artisans will appreciate that elements in the drawingsare illustrated for simplicity and may not have necessarily been drawn to scale.For example, the flow charts illustrate the method in terms of the most prominentsteps involved to help to improve understanding of aspects of the presentinvention. Furthermore, in terms of the construction of the device, one or morecomponents of the device may have been represented in the drawings byconventional symbols, and the drawings may show only those specific detailsthat are pertinent to understanding the embodiments of the present invention soas not to obscure the drawings with details that will be readily apparent to thoseof ordinary skill in the art having the benefit of the description herein.DETAILED DESCRIPTION
[0023] For the purpose of promoting an understanding of the principles of theinvention, reference will now be made to the various embodiments and specificlanguage will be used to describe the same. It will nevertheless be understoodthat no limitation of the scope of the invention is thereby intended, suchalterations and further modifications in the illustrated system, and such furtherapplications of the principles of the invention as illustrated therein beingcontemplated as would normally occur to one skilled in the art to which theinvention relates.
[0024] It will be understood by those skilled in the art that the foregoinggeneral description and the following detailed description are explanatory of theinvention and are not intended to be restrictive thereof.
[0025] Reference throughout this specification to "an aspect", "another aspect"or similar language means that a particular feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment of the present invention. Thus, appearances of the phrase "in anembodiment", "in another embodiment" and similar language throughout thisspecification may, but do not necessarily, all refer to the same embodiment.
[0026] The terms "comprises", "comprising", or any other variations thereof,are intended to cover a non-exclusive inclusion, such that a process or methodthat comprises a list of steps does not include only those steps but may includeother steps not expressly listed or inherent to such process or method. Similarly,one or more devices or sub-systems or elements or structures or componentsproceeded by "comprises... a" does not, without more constraints, preclude theexistence of other devices or other sub-systems or other elements or otherstructures or other components or additional devices or additional sub-systemsor additional elements or additional structures or additional components.
[0027] The present invention is directed towards a method and system fordynamically adjusting a predefined listening time of a voice assistant devicebased on an analysis of the breathing pattern of the user.
[0028] Breathing pattern during voice commands is an important aspect whichpredicts the intention of user to speak more. While providing commands, usersfirst inhale and then produce the voice command while exhaling. The presentinvention is directed towards using the breathing pattern to identify user'sintention to speak more and accordingly adjust the mic awake time.
[0029] Once the user provides commands including a hot word (e.g., "Hi<<Name of Voice Assistant>>) to a voice assistant device, the system disclosedin the present invention continuously analyzes user's breathing pattern and non-speech artifacts from audio spectrogram or waveform to determine whether theuser's intention is to speak more. Based on the user's intention to speak more,the listening time is dynamically adjusted for continuation and follow-uputterances.
[0030] Figures 1A and 1B illustrate continuation case of conversationaccording to various embodiments of the present invention with respect to theprior art.
[0031] In conventional voice assistants (VAs), the user may initially provide avoice command to a voice assistant such as "Send mom a message". To providesuch a command, the user may initiate the command with a voice inputcomprising a hot word such as "Hi <<VA Name>>". In response to receipt ofsuch hot word-based voice input, the VA is triggered and processes a voicecommand which is received within a specific time duration after the hot word.Various VAs may be triggered based on different pre-configured hot words orwake-up commands, such as "Hi <<VA name>>", "Hello <<VA name>>", "OK<<VA name>>", etc. The trigger / wake-up commands including the hot wordsmay also be manually modified by the users. The specific duration after the hotword during which the to-be-executed voice command needs to be provided bythe user, is pre-configured by the manufacturer during manufacturing of the VAdevice or system.
[0032] In the current conventional example illustrated in Figure 1A, when useris taking long pause (may be due to thinking about message content), the VAwaits for a pre-configured or predefined listening time (e.g., 600 Milliseconds)for receiving speech. If the user does not provide any command within such apre-configured time, the VA marks end of utterance, and stops listening whichresults in missing the continuation of utterance.
[0033] As illustrated in Figure 1B, the present invention analyzes breathingpattern of the user and further identify user's intention to speak more (ISM).Accordingly, the pre-configured or predefined listening time of the user may bemodified dynamically to continue listening for continuation of utterance by theuser.
[0034] Figures 1C and 1D illustrate follow-up case of conversation accordingto various embodiments of the present invention with respect to the prior art.
[0035] As illustrated in Fig. 1C associated with conventional solutions (i.e.,prior art), once the user provides a wake-up command comprising a hot word,such as "Hi <<VA name>>" and subsequently provides a command such as"How is weather today?". When user completes an utterance, the VA waits for600 Milliseconds (static time) for speech, marks the end of utterance, and stopslistening. If the user wants to provide a follow-up utterance, then he / she isrequired to manually re-trigger the VA by providing a wake-up commandincluding the hot word.
[0036] As depicted in Fig. 1D, in accordance with various embodiments of thepresent invention, once the user provides a command such as "how is weathertoday", and the system of the present invention continuously analyzes breathingpattern of the user during VA's response and identify user's intention to speakmore (ISM). Accordingly, the VA will be auto-awaken to listen for follow-uputterance without any need for receiving wake-up command comprising hotword.
[0037] Figure 2A illustrates a schematic block diagram 200 of a system 203for dynamically adjusting a predefined listening time of the voice assistantdevice 201, according to an embodiment of the present invention.
[0038] In one embodiment, the system 203 may be included within the voiceassistant device 201. In other embodiments, the system 203 may be configuredto operate as a standalone device or a system. In yet other embodiments, thesystem 203 may be included within a mobile device which includes a voiceassistant system (such as 201). The voice assistant system may be configured asa mobile software application or as an operating system function of the mobiledevice. Examples of mobile device may include, but not limited to, a mobilephone or any other electronic device providing voice input capability requiredby a voice assistant. Examples of voice assistant device 201 may include, but notlimited to, a voice assistant, a smart watch, a laptop computer, a desktopcomputer, a Personal Computer (PC), a notebook, a tablet, a mobile phone, ande-book readers or any other smart device having a voice input capability requiredby a voice assistant.
[0039] In various embodiments of the present invention, the system 203 withinthe voice assistant device 201 may be configured to dynamically adjust apredefined listening time of a voice assistant device. The system 203 may furtherinclude a processor / controller 202, an I / O interface 204, modules 206,transceiver 208, and a memory 210.
[0040] In some embodiments, the memory 210 may be communicativelycoupled to the at least one processor / controller 202. The memory 210 may beconfigured to store data, instructions executable by the at least oneprocessor / controller 202. In some embodiments, the module 206 may beincluded within the memory 210. The memory may further include a database212 to store data. The one or more modules 206 may include a set of instructionsthat may be executed to cause the system 203 to perform any one or more of themethods disclosed herein. The one or more modules 206 may be configured toperform the steps of the present disclosure using the data stored in the database212, to dynamically adjust a predefined listening time of a voice assistant device,as discussed herein. In an embodiment, each of the one or more modules 206may be a hardware unit which may be outside the memory 210. Further, thememory 210 may include an operating system 214 for performing one or moretasks of the system 203 and / or voice assistant device 201, as performed by ageneric operating system in the communications domain. The transceiver 208may be capable of receiving and transmitting signals to and from the voiceassistant device 201. The I / O interface 204 may include a speaker and / or amicrophone to receive voice commands and provide a voice output to the user216. Further, the I / O interface 204 may provide a display function and one ormore physical buttons on the device 201 to input / output various functions of thevoice assistant device 201. For the sake of brevity, the architecture and standardoperations of operating system 214, memory 210, database 212,processor / controller 202, transceiver 208, and I / O interface 204 are not discussedin detail. In one embodiment, the database 212 may be configured to store theinformation as required by the one or more modules 206 and processor / controller202 to perform one or more functions to dynamically adjust a predefinedlistening time of the voice assistant device 201.
[0041] The I / O interface 204 may be configured to receive the voice inputs andnon-speech sounds from the users within a surrounding environment and alsoconfigured to handle an audio or a video call which is either initiated on or istransferred from a connected mobile device. The voice commands may behandled by the I / O interface 204 in conjunction with a speaker, which may beconfigured to provide the voice output to the users (e.g., 216).
[0042] In one embodiment, the memory 210 may communicate via a buswithin the system 203. The memory 210 may include, but not limited to, a non-transitory computer-readable storage media, such as various types of volatile andnon-volatile storage media including, but not limited to, random access memory,read-only memory, programmable read-only memory, electricallyprogrammable read-only memory, electrically erasable read-only memory, flashmemory, magnetic tape or disk, optical media and the like. In one example, thememory 210 may include a cache or random-access memory for theprocessor / controller 202. In alternative examples, the memory 210 is separatefrom the processor / controller 202, such as a cache memory of a processor, thesystem memory, or other memory. The memory 210 may be an external storagedevice or database for storing data. The memory 210 may be operable to storeinstructions executable by the processor / controller 202. The functions, acts ortasks illustrated in the figures or described may be performed by the programmedprocessor / controller 202 for executing the instructions stored in the memory 210.The functions, acts or tasks are independent of the particular type of instructionsset, storage media, processor or processing strategy and may be performed bysoftware, hardware, integrated circuits, firmware, micro-code and the like,operating alone or in combination. Likewise, processing strategies may includemultiprocessing, multitasking, parallel processing, and the like.
[0043] Further, the present invention contemplates a computer-readablemedium that includes instructions or receives and executes instructionsresponsive to a propagated signal, so that a device (e.g., voice assistant device201) connected to a network may communicate voice, video, audio, images, orany other data over a network. Further, the instructions may be transmitted orreceived over the network via a communication port or interface or using a bus(not shown). The communication port or interface may be a part of theprocessor / controller 202 or maybe a separate component. The communicationport may be created in software or maybe a physical connection in hardware. Thecommunication port may be configured to connect with a network, externalmedia, the display, or any other components in system, or combinations thereof.The connection with the network may be a physical connection, such as a wiredEthernet connection or may be established wirelessly. Likewise, the additionalconnections with other components of the system 203 may be physical or maybe established wirelessly. The network may alternatively be directly connectedto the bus.
[0044] In one embodiment, the processor / controller 202 may include at leastone data processor for executing processes in Virtual Storage Area Network. Theprocessor / controller 202 may include specialized processing units such as,integrated system (bus) controllers, memory management control units, floatingpoint units, graphics processing units, digital signal processing units, etc. In oneembodiment, the processor / controller 202 may include a central processing unit(CPU), a graphics processing unit (GPU), or both. The processor / controller 202may be one or more general processors, digital signal processors, application-specific integrated circuits, field-programmable gate arrays, servers, networks,digital circuits, analog circuits, combinations thereof, or other now known orlater developed devices for analyzing and processing data. Theprocessor / controller 202 may implement a software program, such as codegenerated manually (i.e., programmed).
[0045] The processor / controller 202 may be disposed in communication withone or more input / output (I / O) devices via the I / O interface 204. The I / Ointerface 204 may employ communication code-division multiple access(CDMA), high-speed packet access (HSPA+), global system for mobilecommunications (GSM), long-term evolution (LTE), WiMax, or the like, etc.
[0046] Using the I / O interface 204, the voice assistant device 201 maycommunicate with one or more I / O devices. For example, the input device maybe an antenna, microphone, touch screen, touchpad, storage device, transceiver,video device / source, etc. The output devices may be a printer, fax machine, videodisplay (e.g., cathode ray tube (CRT), liquid crystal display (LCD), light-emitting diode (LED), plasma, Plasma Display Panel (PDP), Organic light-emitting diode display (OLED) or the like), audio speaker, etc.
[0047] The processor / controller 202 may be disposed in communication witha communication network via a network interface. The network interface may bethe I / O interface 204. The network interface may connect to a communicationnetwork. The network interface may employ connection protocols including,without limitation, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 BaseT), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE802.11a / b / g / n / x, etc. The communication network may include, withoutlimitation, a direct interconnection, local area network (LAN), wide area network(WAN), wireless network (e.g., using Wireless Application Protocol), theInternet, etc. Using the network interface and the communication network, thevoice assistant device 201 may communicate with other devices. The networkinterface may employ connection protocols including, but not limited to, directconnect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission controlprotocol / internet protocol (TCP / IP), token ring, IEEE 802.11a / b / g / n / x, etc.
[0048] The processor / controller 202 may identify voice or non-voicecommands from the received speech sounds and non-speech sounds. A trainedclassifier model or various speech extraction techniques may be implemented toextract or identify the voice commands and the senders of the commands, suchas user 216.
[0049] Figure 2B illustrates another schematic block diagram 200b of a system203 for dynamically adjusting a predefined listening time of the voice assistantdevice, according to an embodiment of the present invention. As depicted, Fig.2B depicts the system 203 along with sequential flow of various inputs to thesystem 203. As a first step, the hot word or wake-up input is detected andanalyzed by a hot word detector of receiving module 218. Further, an audioanalyzer of the receiving module 218 may be configured to process and executean audio input or command received from the user to perform the desiredfunction.
[0050] Once the audio input or command is executed to perform the desiredfunction, the pause analysis module may be configured to detect a pause in thespeech. Upon detection of a pause, an intention to speak more (ISM) systemcomprising a breath analysis module and a listening time adjustment module maybe configured to analyze user's breath to dynamically adjust the pre-configuredlistening time in order to continue listening for user's utterance or follow-upcommands without requiring a wake-up / trigger command comprising any hotword. The system 203 may further include an artificial intelligence (AI) platformincluding an automatic speech recognition (ASR), a natural languageunderstanding (NLU), and a natural language generation (NLG) / text-to-speechTTS. In one embodiment, the AI platform may be configured to process the audioinputs or commands, and further configured to output audio. The ASR may beconfigured to use the Machine Learning or Artificial Intelligence (AI)technology to process human speech into readable text. As widely known, theNLU is a branch of natural language processing (NLP), which helps computersunderstand and interpret human language by breaking down the elemental piecesof speech. While speech recognition captures spoken language in real-time,transcribes it, and returns text, NLU goes beyond recognition to determine auser's intent. The NLG is a branch of artificial intelligence (AI) that generateslanguage as an output on the basis of data as input. Further, the TTS is a type ofassistive technology that reads digital text aloud. It's sometimes called "readaloud" technology. For the sake of brevity, the detailed functioning of the ASR,NLU, and NLG / TTS are not discussed in detail here, as the generic functions arewidely known in the art. The functions of each of the other modules are discussedin conjunction with Fig. 3 hereinafter.
[0051] Figure 3 illustrates a schematic block diagram 300 of modules 206 ofthe system 203 for dynamically adjusting a predefined listening time of the voiceassistant device, according to an embodiment of the present invention. In oneembodiment, the modules 206 may include a receiving module 302, a pauseanalysis module 304, a breath analysis module 306, a listening time adjustmentmodule 308, and a training module 310.
[0052] In one embodiment, the receiving module 302 may be configured toreceive an audio input. The receiving module 302 may be similar to the receivingmodule 218 of Fig. 2B. In one embodiment, the audio input may include a hotword or wake-up command indicating initiation of a conversation. Examples ofhot words may include, but not limited to, "Hi <<VA name>>", "Hello <<VAname>>", and "OK <<VA name>>". The wake-up commands including the hotwords may also be manually modified / configured by the users in the voiceassistant devices. In some embodiments, the receiving module may beconfigured to filter the noise from the audio input and provide a noise-free outputto the pause analysis module 304.
[0053] Further, the receiving module 302 may be configured to receive thespeech or voice input received after the hot-word detection from the user. Thevoice input may be processed and the command associated with the voice inputmay be executed. A person skilled in the art would appreciate that the execution(i.e., performing a function associated with voice command) of voice commandsmay be performed in a conventional manner, and hence, for the sake of brevity,the execution of voice commands at the system 203 or the voice assistance device201 is not discussed in detail here. In some embodiments, a processing module(not shown) may be provided to execute the voice commands.
[0054] In one embodiment, the pause analysis module 304 may be configuredto detect a pause of a specific time duration after receiving the audio input. Thepause analysis module may be configured to receive the audio input withoutnoise component from the receiving module 302. In some embodiments, thepause analysis module 304 may not be required to perform further actions asdiscussed hereinbelow. For example, the speech or voice input may be analyzedby the breath analysis module 306 without or before detection of any pausesignal. In one embodiment, the pause duration may be specific to each user. Inparticular, the pause duration detection may be based on identification of a user.Further, in an embodiment, the pause may correspond to a pause between twowords of the language spoken by the user. In another embodiment, the pause maycorrespond to a pause between two sentences of the spoken language. Theidentification of words and sentences of the spoken language is well known inthe art, and is thus not discussed in detail here for the sake of brevity.
[0055] In one embodiment, the breath analysis module 306 may be configuredto extract one or more of a speech component and a non-speech artifact from theaudio input which may be used to determine breathing patterns and continuationof utterances for the conversation (follow-up utterances). In an embodiment, thecaptured speech signal may be evaluated for estimating the breath signal in anasynchronous manner. Further, the user may be identified based on the detectedbreathing pattern or one or more other user specific characteristics / parameters.In an embodiment, the user's breathing pattern may be matched to one or morepre-stored user profiles. The pre-stored user's profile may include, but notlimited to, voice input, breathing pattern, device profile and user's age. In otherembodiments, the user's profile may be determined based on voice input detectedby the receiving module 302. In one embodiment, the speech componentcomprises one or more of phonemes and graphemes. The non-speech artifactcomprises one or more of breathing sounds, non-lexical speaker noise, andbackground noise.
[0056] Further, the breath analysis module 306 may be configured to determinea user breathing pattern based on the one or more of the speech component andthe non-speech artifact. In one embodiment, the user breathing pattern isdetermined using one of an audio spectrogram and a waveform analysis. Further,the user breathing pattern may be determined based on the at least one speechcomponent and the at least one non-speech artifact using a neural network model.In one embodiment, the estimation of breath signals or patterns may be based ona regression model. The breathing pattern may be determined based on an inputspeech feature vector corresponding to the voice input, and the approach usedmay be speech feature vector-to-signal regression. Further, the breathing patternof the user may be output as a breath vector for further processing, as discussedherein. In one embodiment, the user breathing pattern identification may includeidentifying a duration and an amplitude of at least one inhalation and exhalationof the user while providing the audio input to the voice assistant device.
[0057] In one exemplary embodiment, the breath signal or pattern may bedetermined based on labeled training data stored in a neural network model.Further, speech Log Mel Filters may be used to determine an output featurevector corresponding to breath signal or pattern. An exemplary set of speechsamples with associated breath signal vector is illustrated in Table 1 below:Table 1A corresponding feature vector associated with the user's breathing pattern orsignal is derived as shown in Table 2 below. A feed forward neural network withback propagation may be used to determine breath signal feature vectors.Table 2
[0058] Additionally, the breath analysis module 306 may be configured toidentify one or more attributes that impact the user breathing pattern based on atleast one non-speech component, captured from the surrounding environmentand the voice assistant device. Such non-speech components are captured toidentify parameters / components that may impact the breathing pattern of usersfor the continuation of conversation (follow-up utterances). In one embodiment,the at least one non-speech component comprises one or more of environmentnoise, microphone channel noise, and non-speech artifacts (e.g., non-lexicalspeaker noise such as hum, hmm, aaa, bb, etc. along with breathing sounds).Further, the one or more attributes comprise one or more of environmentalparameters, user data (e.g., gender), and voice assistant device data (e.g., deviceGPS coordinates). In one embodiment, the breath analysis module 306 may beconfigured to use a neural net regression model for identifying environmentalparameters.
[0059] Further, after detecting a pause in the audio input, the breath analysismodule 308 may determine whether a user's intention is to continue theconversation based on an analysis of the user breathing pattern and the one ormore attributes. In one embodiment, the determining comprises determining anintention-to-speak more (ISM) coefficient based on the analysis of the userbreathing pattern and the one or more attributes. The user's profile may also beconsidered while determining the ISM coefficient, and hence, the ISMcoefficient may be specific to each user. In one embodiment, the ISM coefficientmay have values from -1 to 0 indicating that the user does not have an intentionto speak more, the values more than 0 to 1 may indicate that the user has anintention to speak more which may require an adjustment to the current listeningtime of the voice assistant device. In another embodiment, it may be determinedwhether the ISM coefficient is monotonically increasing or decreasing. In caseof monotonic decrease in the ISM coefficient over time, it may bedetermined / concluded that the user's tendency to speak is declining. On the otherhand, in case of monotonic increase in the ISM coefficient over time, it may bedetermined / concluded that the user's tendency to speak is advancing. In oneembodiment, the determination of whether the user's intention is to continue theconversation is based on the identified duration and the amplitude of the at leastone inhalation and exhalation of the user while providing the input to the voiceassistant device. More specifically, the determination of whether user's intentionis to speak more is based on the user's breathing pattern, environmentalparameters, and user device data to dynamically adjust the voice assistant's miclistening (wake-up) time.
[0060] In one exemplary embodiment, the breath signal ISM coefficient valuesmay be determined based on labeled training data stored in a neural networkmodel. An exemplary set of breath signal vectors and corresponding ISMcoefficient values is illustrated in Table 3 below:Table 3
[0061] An exemplary feature vector output based on the labeled data stored inneural network model may be derived as indicated in Table 4 below. A feedforward neural network with back propagation may be used to determine featurevectors associated with ISM coefficients.Table 4
[0062] In one exemplary embodiment, the inhalation and exhalation durationsmay be pre-stored in the database to derive the ISM coefficient. An exemplaryset of values is shown here in Table 5 below:Table 5
[0063] In one embodiment, the listening time adjustment module 308 may beconfigured to dynamically adjust the predefined listening time of the voiceassistant device to continue listening for voice commands in the conversationbased on a determination that the user's intention is to continue the conversation.The predefined listening time of the voice assistant device is one of a defaultlistening time set during manufacturing of the voice assistant device, and adynamically set listening time different from the default listening time. Todynamically adjust the predefined listening time, the listening time adjustmentmodule 308 may be configured to determine a listening duration based on theestimated ISM coefficient when the pause analysis is detected by the pauseanalysis module 304. In an exemplary embodiment, if the current speechthreshold is 600ms, then the listening time may be adjusted from 0 to 2*speechthreshold, i.e., from 0 to 1200ms. In another exemplary embodiment, theadjusted listening time may be derived based on the following formula:Adjusted Listening Time = Speech Threshold * (1+ISM Coefficient)In some embodiments, the listening time may only be adjusted based on acomparison with a pause duration. For example, the listening time may beadjusted if the pause duration is less than 180 seconds. If the pause duration ismore than 180 seconds, than the voice assistant may be configured to stoplistening to user voice inputs.
[0064] In one embodiment, the dynamically adjustment of the predefinedlistening time of the voice assistant device is based on the specific time durationof the pause to continue listening for voice commands in the conversation afterthe pause. In another embodiment, the dynamically adjusting predefinedlistening time of the voice assistant device is based on the ISM coefficient and aspeech threshold value. Further, in some embodiments, the voice assistant devicemay be configured to adjust the predefined listening time for listening to uservoice inputs till detection of a subsequent pause.
[0065] More specifically, in one embodiment, the listening time adjustmentmodule 308 may be configured to eliminate a need of repeatedly waking-up thevoice assistant device by suspending a requirement of receiving the hot word tocontinue listening for the voice commands.
[0066] In one embodiment, the training module 310 may be configured to traina machine learning (ML) or neural network model used for the listening timeadjustment module 308 to modulate / adjust the listening time based on ISMcoefficient. In one exemplary embodiment, the trained neural network modelprovides ISM coefficient and the adjusted listening time may be determined asspeech threshold * (1 + ISM coefficient). The speech threshold may be a pre-defined value (e.g., defined during manufacturing of VA, or modified during itsoperation) for the VA considering different age groups of users and deviceenvironments around the VA, at which the VA provides an optimal recognitionperformance without pausing it and abruptly leading to speech halt.
[0067] Figure 4 illustrates an exemplary process of determining user'sintention to speak more based on user's inhalation and exhalation breathingpattern while the user is speaking to a voice assistant, according to oneembodiment of the present invention. In one embodiment, the breath detectormodel and breath signal analyzer may be a part of breath analysis module 306,as discussed previously. As depicted, the breath detector model and breath signalanalyzer may be configured to determine an ISM coefficient based on inhalationand exhalations in the user's breathing pattern, to adjust the listening time of thevoice assistant device.
[0068] As depicted, user's inhalation and exhalations may be analyzed duringhis / her utterances or commands provided to the voice assistant device. Forexample, the user may provide a command "how is the weather today" duringexhalation phase of his / her breathing pattern. Such exhalation or command maytake a duration of 3 seconds. An amplitude vs time graph is depicted to highlightthe analysis performed by the breath analysis module 306.
[0069] After the exhalation and the first utterance, a pause is detected andbreath analysis is performed. As depicted, the breath analysis module 306 mayanalyze the breathing pattern in response to the aforementioned first utterance.The breathing pattern may include a duration of inhalation, an inhalation pause,an exhalation duration, and an exhalation pause. In one embodiment, thebreathing analysis module 306 may compare the breathing pattern of the specificuser with a pre-stored breathing pattern which may be captured over time by thesystem. Based on the comparison, if it is detected that the user's current breathingpattern is similar to the normal or pre-stored breathing pattern of the user, thenthe breathing analysis module 306 may conclude that the user is breathingnormally. In operation, an ISM coefficient may be derived based on the capturedvarious parameters such as speech, non-speech components, environmentalparameters, and breathing sounds, as discussed previously. The ISM coefficientmay be derived based on a machine learning or neural network model stored atthe system 203. In this exemplary scenario, the ISM coefficient may correspondto -0.75 indicating that the user is breathing normally, and hence, the user doesnot desire to provide any continuation or follow-up utterance. Thus, no dynamicadjustment to the predefined listening time of the voice assistant device may berequired.
[0070] Figure 5 illustrates an exemplary process of determining user'sintention to speak more based on user's quick exhalation or inhalation while theuser is speaking to a voice assistant, according to one embodiment of the presentinvention. In the present example, the breath analysis module 306 may comprisea breath detector model and a breath signal to ISM coefficient mapping moduleto perform the functions of breath detection and ISM coefficient determination,respectively, as discussed previously. For the sake of brevity, these are notdiscussed in detail here again.
[0071] As depicted, in the instance scenario, while the user is uttering "how isthe weather today", the breath analysis module 306 may be configured to analyzebreathing pattern of the user comprising cycles of inhalation and exhalations. Itmay be analyzed that the user uttered "how is" during speech exhalation phaseof 0-3 seconds. Further, after the initial utterance of "how is", a successivesequence of quick exhalation, an inhalation pause, and a quick inhalation phasemay be detected. The term "quick" specifies that the duration of inhalation andexhalation phases in the breathing pattern of the user are relatively faster thanthe normal breathing pattern pre-stored for the specific user, under a specific setof environmental conditions (indicated by one or more attributes related toenvironment). Since the user's breathing pattern indicates quickinhalation / exhalation after the exhalation speech utterance of "how is", thebreath analysis module 306 may determine user is trying to store breath to speakmore, thereby indicating that the user has not finished speaking yet. In operation,the breath signal to ISM time mapping module of the breath analysis module 306may be configured to determine the ISM as 0.70, indicating user's intention tospeak more. Thus, according to various embodiments, the voice assistant devicemay eliminate the requirement of wake-up or trigger word to receive userutterance as a part of continued conversation or command. The elimination ofthe requirement may be performed for an adjusted amount of time.
[0072] Figure 6 illustrates an exemplary process of determining user'sintention to speak more based on user's breath hold while the user is speaking toa voice assistant, according to one embodiment of the present invention. In thepresent example, a breath pause may be detected after an initial utterance of"send mom". For example, while the initial utterance may be detected from 0-3.2 seconds, the breath pause may be detected from 3.2 to 5.2 seconds. Based ondetection and analysis of the breath pause by the breath analysis module 306, anISM coefficient may be determined as 0.90. Based on the value of ISMcoefficient, it may be concluded or determined that after continuous exhalation,the user is holding breath implying that the user is thinking about next command,and hence, the user has not finished speaking yet. Thus, according to variousembodiments, the voice assistant device may subsequently eliminate therequirement of wake-up or trigger word to receive user utterance as a part ofcontinued conversation or command.
[0073] Figure 7 illustrates an exemplary process of determining user'sintention to speak more during a continuation utterance based on a combinationof different breathing patterns of the user while the user is speaking to a voiceassistant, according to one embodiment of the present invention.
[0074] In the present example, the various scenarios explained in Figures 5 and6 may be detected simultaneously, and a combined ISM coefficient may bederived. For example, during speech exhalation, the user speaks "How is" andthen performs quick exhale and inhale followed by a breath hold. At "Point A"the ISM Coefficient value may be 0.70, which is > 0 so the mic will continuelistening for increased adjusted time. Again, during speech exhalation, the userspeaks "the weather today". After some time, user starts normal breathing. At"Point B", ISM value may be -0.80, which is < 0. Hence, the user won't speakmore, and mic will stop listening after decreased adjusted time. Thus, in someembodiments, the listening time may also be decreased based on an analysis ofthe breathing pattern of user.
[0075] Figure 8 illustrates an exemplary process of determining user'sintention to speak more during a follow-up utterance based on a combination ofdifferent breathing patterns of the user while the user is speaking to a voiceassistant, according to one embodiment of the present invention.
[0076] In the present example, during speech exhalation, the user may speak"Send mom a message" then performs breath hold which means he will speakmore. At "Point A", the ISM Coefficient value may be 0.90, which is > 0.Accordingly, the mic will continue listening for an increased adjusted time.Again, during speech exhalation, the user speaks follow-up utterance "I will bethere in 10 minutes". Then, the user may perform normal breathing implying thatthe user finished speaking. At "Point B", the ISM value may be -0.70 which is <0 so the user won't speak more, and the mic will stop listening after decreasedadjusted time.
[0077] Figure 9 illustrates various exemplary scenarios of user's breathingpatterns in different surroundings while the user is speaking to a voice assistant,according to one embodiment of the present invention. The present invention iscapable of understanding the distinction between various environmentalscenarios. Fig. 9 illustrates three different environmental scenarios detectedbased on user's non-speech components and / or attributes associated withenvironmental parameters. The various exemplary scenarios detected for the usermay include a silent room, walking in crowded places, and travelling inmetro / bus. The present invention is capable of analyzing the user's environmentbased on breathing pattern and environmental attributes captures from thesurrounding environment of the user. The breath analysis module 306 may beconfigured to adjust the model to derive ISM value to correctly identify the user'sintention in order to adjust the predefined listening time of the voice assistantdevice.
[0078] Figure 10 illustrates an exemplary process flow depicting a method1000 for dynamically adjusting a predefined listening time of a voice assistantdevice, according to one embodiment of the present invention. In oneembodiment, the steps of the method 1000 may be performed by the system 203,which may be integrated within the voice assistant device 201 or providedseparately.
[0079] At step 1002, the method 1000 includes receiving, by the voice assistantdevice, an audio input. In one embodiment, the audio input may include a hotword indicating initiation of a conversation.
[0080] At step 1004, the method 1000 includes extracting one or more of aspeech component and a non-speech artifact from the audio input. In oneembodiment, the speech component comprises one or more of phonemes andgraphemes. The non-speech artifact comprises one or more of breathing soundsand non-lexical speaker noise.
[0081] At step 1006, the method 1000 includes determining a user breathingpattern based on the one or more of the speech component and the non-speechartifact. In one embodiment, the user breathing pattern is determined using oneof an audio spectrogram and a waveform analysis. Further, the user breathingpattern may be determined based on the at least one speech component and theat least one non-speech artifact using a neural network model. In oneembodiment, the user breathing pattern identification may include identifying aduration and an amplitude of at least one inhalation and exhalation of the userwhile providing the audio input to the voice assistant device.
[0082] At step 1008, the method 1000 includes identifying one or moreattributes that impact the user breathing pattern based on at least one non-speechcomponent, captured from the surrounding environment and the voice assistantdevice. In one embodiment, the at least one non-speech component comprisesone or more of environment noise, microphone channel noise, and non-speechartifacts (e.g., non-lexical speaker noise such as hum, hmm, aaa, bb, etc. alongwith breathing sounds). Further, the one or more attributes comprise one or moreof environmental parameters, user data, and voice assistant device data.
[0083] At step 1010, the method 1000 includes detecting a pause of a specifictime duration after receiving the audio input.
[0084] At step 1012, the method 1000 includes determining, after detecting apause in the audio input, whether a user's intention is to continue theconversation based on an analysis of the user breathing pattern and the one ormore attributes. In one embodiment, the determining comprises determining anintention-to-speak more (ISM) coefficient based on the analysis of the userbreathing pattern and the one or more attributes. In one embodiment, thedetermination of whether the user's intention is to continue conversation is basedon the identified duration and the amplitude of the at least one inhalation andexhalation of the user while providing the input to the voice assistant device.
[0085] At step 1014, the method 1000 includes dynamically adjusting thepredefined listening time of the voice assistant device to continue listening forvoice commands in the conversation based on a determination that the user'sintention is to continue the conversation. The predefined listening time of thevoice assistant device is one of a default listening time set during manufacturingof the voice assistant device, and a dynamically set listening time different fromthe default listening time.
[0086] In one embodiment, the dynamically adjusting the predefined listeningtime of the voice assistant device is based on the specific time duration of thepause to continue listening for voice commands in the conversation after thepause. In another embodiment, the dynamically adjusting predefined listeningtime of the voice assistant device is based on the ISM coefficient and a speechthreshold value.
[0087] At step 1016, the method 1000 includes eliminating a need ofrepeatedly waking-up the voice assistant device by suspending a requirement ofreceiving the hot word and continuing listening for the voice commands.
[0088] While the above discussed steps in Fig. 10 are shown and described ina particular sequence, the steps may occur in variations to the sequence inaccordance with various embodiments.
[0089] The present invention provides for various technical advancementsbased on the key features discussed above. The listening time of the voiceassistant device may be automatically adjusted, leading to elimination of anywake-up command or trigger or hot word requirement for waking up the deviceagain and again. This leads to smooth conversation of users with voice assistantdevices. Further, in addition to various use cases discussed above related topauses, the present invention may also be useful for different age groups. Forexample, usually kids have short vocal cords, they produce short air waves,shorter breaths, and have multiple longer pauses while speaking. Age impactspauses. The invention takes this into consideration during breathing patternanalysis and accordingly adjusts the listening time during pause.
[0090] While specific language has been used to describe the present subjectmatter, any limitations arising on account thereto, are not intended. As would beapparent to a person in the art, various working modifications may be made tothe method in order to implement the inventive concept as taught herein. Thedrawings and the foregoing description give examples of embodiments. Thoseskilled in the art will appreciate that one or more of the described elements maywell be combined into a single functional element. Alternatively, certainelements may be split into multiple functional elements. Elements from oneembodiment may be added to another embodiment.
Claims
1. A method (1000) of dynamically adjusting a predefined listening time of a voice assistant device (201), the method (1000) comprising: receiving (1002), by the voice assistant device (201), an audio input; extracting (1004) one or more of a speech component and a non-speech artifact from the audio input; determining (1006) a user breathing pattern based on the one or more of the speech component and the non-speech artifact; identifying (1008) one or more attributes that impact the user breathing pattern based on at least one non-speech component, captured from the surrounding environment and the voice assistant device; determining (1012), after detecting a pause in the audio input, whether a user's intention is to continue the conversation based on an analysis of the user breathing pattern and the one or more attributes; and dynamically adjusting (1014) the predefined listening time of the voice assistant device to continue listening for voice commands in the conversation based on a determination that the user's intention is to continue the conversation.
2. The method (1000) as claimed in claim 1, comprising: detecting (1010) the pause of a specific time duration after receiving the audio input; and dynamically adjusting (1014) the predefined listening time of the voice assistant device (201) based on the specific time duration of the pause to continue listening for voice commands in the conversation after the pause.
3. The method (1000) as claimed in claim 1, comprising: determining (1006) the user breathing pattern using one of an audio spectrogram and a waveform analysis.
4. The method (1000) as claimed in claim 1, comprising: determining an intention-to-speak more (ISM) coefficient based on the analysis of the user breathing pattern and the one or more attributes; and dynamically adjusting (1014) predefined listening time of the voice assistant device (201) based on the ISM coefficient and a speech threshold value.
5. The method (1000) as claimed in claim 1, wherein the at least one non-speech component comprises environment noise, microphone channel noise, and non-speech artifacts.
6. The method (1000) as claimed in claim 1, comprising: eliminating a need of repeatedly waking-up the voice assistant device by suspending a requirement of receiving a hot word to continue listening for the voice commands, in response to dynamically adjusting the predefined listening time of the voice assistant device (201).
7. The method (1000) as claimed in claim 1, wherein the one or more attributes comprise environmental parameters, user data, and voice assistant device data.
8. The method (1000) as claimed in claim 1, wherein the speech component comprises one or more of phonemes and graphemes, and wherein the non-speech artifact comprises one or more of breathing sounds and non-lexical speaker noise.
9. The method (1000) as claimed in claim 1 comprising: determining (1006) the user breathing pattern based on the speech component and the non-speech artifact using a neural network model.
10. The method (1000) as claimed in claim 1 comprising: identifying a duration and an amplitude of at least one inhalation and exhalation of the user while providing the audio input to the voice assistant device; and determining (1012) whether the user's intention is to continue conversation based on the identified duration and the amplitude.
11. The method (1000) as claimed in claim 1, wherein the predefined listening time of the voice assistant device (201) is one of a default listening time set during manufacturing of the voice assistant device (201), and a dynamically set listening time different from the default listening time.
12. A system (203) for dynamically adjusting a predefined listening time of a voice assistant device (201), the system (203) comprising: a receiving module (302) configured to receive an audio input; a breath analysis module (306) configured to: extract one or more of a speech component and a non-speech artifact from the audio input; determine a user breathing pattern based on the one or more of the speech component and the non-speech artifact; and identify one or more attributes that impact the user breathing pattern based on at least one non-speech component, captured from the surrounding environment and the voice assistant device (201); and a listening time adjustment module (308) configured to: determine, after detecting a pause in the audio input, whether a user's intention is to continue the conversation based on an analysis of the user breathing pattern and the one or more attributes; and dynamically adjust the predefined listening time of the voice assistant device (201) to continue listening for voice commands in the conversation based on a determination that the user's intention is to continue the conversation.
13. The system (203) as claimed in claim 12 comprising a pause analysis module (304) configured to detect the pause of a specific time duration after receiving the audio input, wherein the listening time adjustment module (308) is configured to dynamically adjust the predefined listening time of the voice assistant device (201) based on the specific time duration of the pause to continue listening for voice commands in the conversation after the pause.
14. The system (203) as claimed in claim 12, wherein the breath analysis module (306) is configured to determine an intention-to-speak more (ISM) coefficient based on the analysis of the user breathing pattern and the one or more attributes, and wherein the listening time adjustment module (308) is configured to dynamically adjust predefined listening time of the voice assistant device (201) based on the ISM coefficient and a speech threshold value.
15. The system (203) as claimed in claim 12, wherein the listening time adjustment module (308) is configured to: eliminate a need of repeatedly waking-up the voice assistant device (201) by suspending a requirement of receiving a hot word to continue listening for the voice commands, in response to dynamically adjusting the predefined listening time of the voice assistant device (201).
16. The system (203) as claimed in claim 12, wherein the speech component comprises one or more of phonemes and graphemes, and wherein the non-speech artifact comprises one or more of breathing sounds and non-lexical speaker noise.
17. The system (203) as claimed in claim 12, wherein the breath analysis module (306) is configured to: identify a duration and an amplitude of at least one inhalation and exhalation of the user while providing the audio input to the voice assistant device (201); and determine whether the user's intention is to continue conversation based on the identified duration and the amplitude.
18. The system (203) as claimed in claim 12, wherein the predefined listening time of the voice assistant device (201) is one of a default listening time set during manufacturing of the voice assistant device (201), and a dynamically set listening time different from the default listening time.
