Voice command parsing method and device based on non-voice sound in IoT environment

Abstract

A voice command parsing device includes: a memory configured to store instructions; and a processor configured to execute the instructions so as to: recognize a user's voice command in an input sound, analyze non-voice sounds included in the input sound, and determine at least one target Internet of Things (IoT) device related to the execution of the voice command based on the analysis result of the non-voice sounds.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Indian Patent Application No. 202041005143 filed in the Indian Patent Office on February 6, 2020, and Korean Patent Application No. 10-2020-0183698 filed in the Korean Intellectual Property Office on December 24, 2020, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to Internet of Things (IoT) devices, and more particularly, to a method and apparatus for parsing voice commands based on non-voice sounds in an IoT environment. Background Art

[0004] As technology advances, different types of devices can be interconnected to form a system of devices, which can be referred to as the Internet of Things (IoT). These devices can be controlled by voice or text input from the user. Voice assistant solutions allow users to perform various tasks, such as booking a taxi, playing music, and creating an alarm, through voice or text commands. For the assigned tasks, the voice assistant responds to the user through a natural language dialogue generated by a natural language generator (NLG) module in the IoT system. Currently, the response of natural language dialogue and NLG does not take into account the surrounding non-voice sounds to achieve the best resolution of voice commands. Moreover, the voice assistant cannot recognize the possible actions that the user can perform based on the surrounding audio scenes and / or activities. When the voice assistant cannot perform the required actions on the query requested by the user, the user may feel uncomfortable. For example, if the user is watching TV in the living room, there may be a variety of background sounds due to IoT devices such as TV, air conditioner, and robot vacuum cleaner. When the user receives a call, the user may ask the voice assistant to be quiet to answer the call. In this case, the user may want IoT devices such as TV, air conditioner, and robot vacuum cleaner to not make any sound. However, in practice, since the voice assistant does not take into account background non-voice sounds, the voice assistant may silence the phone while leaving all other sound-emitting IoT devices active. In addition, in order to switch to another task, the user needs to explicitly cancel the current conversation.

[0005] Unlike existing rule-based intelligent systems, artificial intelligence (AI) systems are systems that learn, judge, and become more intelligent. As AI systems are used, their recognition rates are improved, and they can understand users' hobbies more accurately, so existing rule-based intelligent systems are gradually being replaced by deep learning-based AI systems. AI technology includes machine learning (e.g., deep learning) and element technology using machine learning. Machine learning is an algorithm technology that self-classifies / learns the characteristics of input data, and element technology using machine learning algorithms (e.g., deep learning) includes technical fields such as language understanding, visual understanding, deduction / prediction, knowledge representation, and motion control. Summary of the invention

[0006] A method and device for parsing voice commands based on non-voice sounds in an Internet of Things (IoT) environment are provided to accurately and efficiently parse the voice commands of users.

[0007] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the embodiments presented in the disclosure.

[0008] According to one aspect of the present disclosure, a voice command parsing device includes: a memory configured to store instructions; and a processor configured to execute the instructions so as to: recognize a user's voice command in an input sound, analyze non-voice sounds included in the input sound, and determine at least one target Internet of Things (IoT) device related to the execution of the voice command based on the analysis results of the non-voice sounds.

[0009] The processor may also be configured to execute instructions to analyze non-voice sounds when the voice command does not include at least one target IoT device required to execute the voice command.

[0010] The processor may also be configured to execute instructions to determine an operation to be performed by at least one target IoT device related to the execution of the voice command based on the analysis result of the non-voice sound.

[0011] The processor can also be configured to execute instructions to: determine multiple target IoT devices related to the execution of a voice command based on the analysis results of the non-voice sound; determine operations to be performed by multiple target IoT devices related to the execution of the voice command based on the analysis results of the non-voice sound; and control multiple target IoT devices to perform operations simultaneously.

[0012] The processor may also be configured to execute instructions to compare the non-speech sound to predefined sounds for the IoT device.

[0013] The processor may also be configured to execute instructions to detect ambient sounds not related to the operation of the IoT device from non-speech sounds.

[0014] The processor may also be configured to execute instructions to: detect IoT device control parameters in voice commands, and analyze non-voice sounds based on the IoT device control parameters.

[0015] The processor may also be configured to execute instructions to analyze non-voice sounds based on detecting IoT device control parameters in the voice command.

[0016] The processor may also be configured to execute instructions to analyze non-speech sounds based on one or more IoT devices pre-matched with IoT device control parameters.

[0017] The processor can also be configured to execute instructions to determine at least one target IoT device based on a correlation of at least two of voice commands, IoT device control parameters, analysis results of non-voice sounds, or IoT device information received from a repository associated with the voice command parsing device.

[0018] The processor may also be configured to execute instructions to determine at least one target IoT device based on a correlation between IoT device control parameters, analysis results of non-voice sounds, and IoT device information received from a repository associated with the voice command parsing device.

[0019] The processor may also be configured to execute instructions to: determine one or more possible neighboring IoT devices, and analyze non-speech sounds based on the one or more possible neighboring IoT devices.

[0020] The processor may also be configured to execute instructions to determine one or more possible neighboring IoT devices based on the non-speech sound.

[0021] The processor may also be configured to execute instructions to determine one or more possible neighboring IoT devices based on the location of the voice command parsing device.

[0022] The processor may also be configured to execute instructions to determine a location of the voice command parsing device based on the non-speech sound.

[0023] The processor may also be configured to execute instructions to: determine one or more possible neighboring active IoT devices based on the location of the voice command parsing device and IoT device information received from a repository associated with the voice command parsing device; and analyze non-voice sounds based on the one or more possible neighboring active IoT devices.

[0024] The processor may also be configured to execute instructions to determine one or more active IoT devices present around at least one of the voice command parsing apparatus or the user based on one or more possible neighboring active IoT devices and non-voice sounds.

[0025] According to one aspect of the present disclosure, an operating method of a voice command parsing device includes: recognizing a user's voice command in an input sound; analyzing non-voice sounds included in the input sound; and determining at least one target Internet of Things (IoT) device related to the execution of the voice command based on the analysis results of the non-voice sounds.

[0026] The operating method may also include: detecting an IoT device control parameter in the voice command; and determining one or more possible neighboring IoT devices, wherein the non-voice sound is analyzed based on the IoT device control parameter and the one or more possible neighboring IoT devices.

[0027] According to one aspect of the present disclosure, a non-transitory computer-readable recording medium having recorded thereon instructions, which, when executed by at least one processor, causes the at least one processor to perform the following operations: recognizing a user's voice command in an input sound; analyzing non-voice sounds included in the input sound; and determining, based on the analysis results of the non-voice sounds, at least one target Internet of Things (IoT) device related to the execution of the voice command. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings, in which:

[0029] Figure 1 An environment for command parsing based on non-speech sounds in an Internet of Things (IoT) environment according to an embodiment of the present disclosure is shown;

[0030] Figure 2 is a detailed block diagram of a virtual assistant device according to an embodiment;

[0031] Figure 3 is a view showing a target IoT device list on a user interface of an electronic device associated with a user according to an embodiment;

[0032] Figure 4A is a flowchart illustrating a command parsing method based on non-voice sound in an IoT environment according to an embodiment;

[0033] Figure 4B is a flowchart illustrating a command parsing method based on non-voice sound in an IoT environment according to an embodiment;

[0034] Figure 5is a flowchart illustrating a method of obtaining a list of IoT devices having relevance to non-speech sounds according to an embodiment;

[0035] Figure 6 is a block diagram showing a configuration of a voice command parsing apparatus according to an embodiment;

[0036] Figure 7 is a flowchart showing the flow of an operation method of a voice command parsing device based on non-voice sound in an IoT environment according to an embodiment;

[0037] Figure 8 is a detailed flowchart showing the flow of an operation method of a voice command parsing device based on non-voice sound in an IoT environment according to an embodiment; and

[0038] Fig. 9 Detailed flowchart showing the process of an operation method of a voice command parsing device based on non-voice sound in an IoT environment according to an embodiment. DETAILED DESCRIPTION

[0039] Embodiments of the present disclosure will now be described more fully with reference to the accompanying drawings. In the description of the present disclosure, when it is considered that the subject matter of the present disclosure may be unnecessarily obscured, some detailed descriptions of related known functions or elements are omitted. In the accompanying drawings, elements with substantially the same function are represented by the same reference numerals or symbols. For ease of explanation, the device and method will be described together when necessary. The operation of the present disclosure does not necessarily have to be performed in the order described, and can be performed in parallel, selectively or individually. Throughout the specification, it will be understood that when an element is referred to as "connected" to another element, it can be "directly connected" to the other element or "electrically connected" to the other element with an intervening element between them.

[0040] Throughout the disclosure, the expression "at least one of a, b, or c" means: only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0041] Embodiments of the present disclosure relate to a command parsing method and device based on non-voice sounds in an Internet of Things (IoT) environment. In an IoT environment, IoT devices can be connected to communicate with each other and perform specific tasks. IoT devices are controlled by using voice or text commands. In the case of voice commands, when the user's voice command is clear, the IoT device can understand the voice command well and perform the required task. However, in the case where the user's voice command is unclear, the voice command is incomplete, or the ambient sound is mixed with the user's voice command, the IoT device may not be able to perform the task or deliver the output. In order to overcome this situation, the embodiment can control the operation of at least one target IoT device by simultaneously detecting and processing the user's voice command and the non-voice sound from the user's surrounding environment in the received voice command. Even when the user's voice command is unclear, the voice command is incomplete, or the ambient sound is mixed with the voice command, the embodiment can operate and deliver the desired task desired by the user from the IoT device.

[0042] Figure 1 An environment of command parsing based on non-voice sounds in an IoT environment according to an embodiment of the present disclosure is shown.

[0043] Reference Figure 1, environment 100 includes IoT device 101-1, IoT device 101-2, ... and IoT device 101-N (which may be collectively referred to as multiple IoT devices 101), storage library 103, communication network 105 and virtual auxiliary device 107. Multiple IoT devices 101 may be connected to virtual auxiliary device 107 via communication network 105. In an embodiment of the present disclosure, multiple IoT devices 101 may include, but are not limited to, electronic products, electronic devices, or any object embedded with electronic circuits, sensors, and Internet connections. For example, multiple IoT devices 101 may be refrigerators, chimneys, ventilators, speakers, televisions, air conditioners, vacuum cleaners, mobile phones, light bulbs, etc. Multiple IoT devices 101 may run on the Universal Plug and Play (UPnP) protocol. It will be understood by those of ordinary skill in the art that any IoT device not explicitly mentioned may also be used as an IoT device in the present disclosure. Here, all inputs from the user may be received through the user interface of one or more IoT devices or any user device. The plurality of IoT devices 101 may provide real-time output data to the virtual auxiliary device 107 via the communication network 105, and then, the plurality of IoT devices 101 may receive control signals from the virtual auxiliary device 107 via the communication network 105. The real-time output data may be an operating state of the plurality of IoT devices 101. The operating state may include an IoT device state of the plurality of IoT devices 101. The communication network 105 may include, but is not limited to, direct interconnection, an e-commerce network, a peer-to-peer (P2P) network, a local area network (LAN), a wide area network (WAN), a wireless network such as a network using a wireless application protocol (WAP), the Internet, Wi-Fi, Bluetooth, and the like.

[0044] In an embodiment of the present disclosure, the virtual assistant device 107 may receive a voice command from a user, and may control the operation of at least one target IoT device based on the received voice command. The target IoT device may be an IoT device that the user desires to control through a voice command. The target IoT device may be an IoT device that the user desires to control through a voice command, or an IoT device that is capable of performing an operation related to the execution of the user's voice command. The voice command may contain at least one non-voice sound from the user's surroundings. Here, the voice command may be the user's voice sound. The virtual assistant device 107 may include an input / output (I / O) interface 111, a memory 113, and a processor 115. The I / O interface 111 may be configured to receive real-time output data from a plurality of IoT devices 101. Similarly, the I / O interface 111 may be configured to send a control signal to a plurality of IoT devices 101. The I / O interface 111 may use a communication protocol / method such as Bluetooth, cellular, such as code division multiple access (CDMA), high speed packet access (HSPA+), global system for mobile communications (GSM), long term evolution (LTE), or global microwave access interoperability (WiMax), etc. The virtual auxiliary device 107 may be a part of at least one IoT device among the plurality of IoT devices 101 .

[0045] The voice commands received from the user through the I / O interface 111 can be stored in the memory 113. The memory 113 can be communicatively coupled to the processor 115 of the virtual assistant device 107. The memory 113 can also store processor commands. Accordingly, the processor 115 can execute commands for controlling the operation of at least one target IoT device. The memory 113 may include a memory drive, a removable disk drive, etc. The memory drive may include a magnetic drum, a magnetic disk drive, a magneto-optical drive, an optical disk drive, a redundant array of independent disks (RAID), a solid-state storage device, a solid-state drive, etc.

[0046] The processor 115 may include at least one data processor for controlling the operation of at least one target IoT device. The processor 115 may include a dedicated processing unit, such as an integrated system (bus) controller, a memory management control unit, a floating point unit, a graphics processing unit, and a digital signal processing unit.

[0047] In an embodiment of the present disclosure, the virtual assistant device 107 may exchange data with the repository 103 directly or through the communication network 105. The repository 103 may be a database or a cloud server. Data including at least one of the following may be initially loaded or stored in the repository 103 during training: sounds related to the operation of multiple IoT devices 101, ambient sounds from the user's surroundings, IoT device control parameters, or associations of each IoT device control parameter with one or more IoT devices. The data may include the operating status of multiple IoT devices 101 stored in the repository 103. Here, the operating status of multiple IoT devices 101 may be maintained in the repository 103 in real time. Sounds related to the operation of multiple IoT devices 101 may include, for example, the vibration sound of a washing machine, the sound from a chimney fan, and the sound of a refrigerator. Ambient sounds from the user's surroundings may include, for example, the sound of water from a faucet and the sound of a car engine. The IoT device control parameter may be a predefined keyword related to the control of one or more IoT devices. The IoT device control parameter may be a predefined word indicating the operation of one or more IoT devices. For example, the IoT device control parameters may include "speed", "speed increase / decrease", "quiet", "volume", "volume increase / decrease", "mute", "start / stop", "brightness", "bright / dark", "warm / cool", "open / closed", and "play next / previous". According to an embodiment of the present disclosure, the IoT device control parameters may include at least one of a command, a component type, or a target device. For example, when identifying the IoT device control parameters, when the user gives a voice command to "reduce the speed of the chimney", the virtual assistant device 107 may determine that the command is "reduce speed", the component type is "IoT", because the chimney is an IoT device, and the target device is "chimney". The association of each IoT device control parameter with one or more IoT devices may include, for example, associating IoT device control parameters such as speed and speed increase or decrease with IoT devices such as chimneys and / or ceiling fans. The operating status of multiple IoT devices 101 may include the current operating status of multiple IoT devices 101. For example, when the speaker is turned on and the volume is 5, the operating status of the speaker may be 5. The repository 103 may store a list of multiple IoT devices 101. The list may include: static IoT devices that are fixed to a specific location (which may mean, for example, that the location is fixed), such as televisions, refrigerators, and air conditioners; and dynamic IoT devices that are not fixed to any location or are actually movable (which may mean, for example, that the location is not fixed), such as mobile terminals / phones and vacuum cleaners.

[0048] The repository 103 may be updated at any time by the virtual assistant 107, by the user, or by both. The update may be related to at least one of: sounds related to the operation of the plurality of IoT devices 101, ambient sounds from the user's surroundings, IoT device control parameters, association of each IoT device control parameter with one or more IoT devices, or operating status of the plurality of IoT devices 101.

[0049] Figure 2 is a detailed block diagram of a virtual assistant device according to an embodiment of the present disclosure.

[0050] In addition to the I / O interface 111 and the processor 115, the virtual assistant device 107 may also include data 200 and one or more units 211, which will be described in detail herein. In an embodiment of the present disclosure, the data 200 may be stored in the memory 113. The data 200 may include, for example, voice data 201 and other data 203.

[0051] Voice data 201 may be data about input sound. Input sound may be sound input through a microphone. Input sound may include a user's voice command and non-voice sound. Voice data 201 may include a user's voice command. Voice data 201 may include at least one non-voice sound from the user's surroundings. Voice commands may contain at least one non-voice sound from the user's surroundings. Non-voice sounds from the user's surroundings may include at least one of the following: sounds related to the operation of multiple IoT devices 101 in the user's surroundings, or ambient sounds from the user's surroundings other than sounds related to the operation of multiple IoT devices 101. Here, ambient sounds may include the sound of water from a faucet, the sound of a car engine during starting or driving, the sound of a pet, the sound of a conversation between two or more people, and the like.

[0052] The other data 203 may store data generated by the unit 211 for executing various functions of the virtual assistant 107 , including temporary data and temporary files.

[0053] In an embodiment of the present disclosure, the data 200 in the memory 113 is processed by one or more units 211 present in the memory 113 of the virtual assistant 107. In an embodiment of the present disclosure, one or more units 211 may be implemented as a dedicated hardware unit. For example, the term "unit" may refer to an application-specific integrated circuit (ASIC), an electronic circuit, a field programmable gate array (FPGA), a programmable system on chip (PSoC), a combinational logic circuit, and / or other suitable components that provide the above functions. In an embodiment of the present disclosure, one or more units 211 may be communicatively coupled to the processor 115 to perform one or more functions of the virtual assistant 107. The unit 211 configured with the functions defined in the present disclosure may be novel hardware. In addition, the term "unit" may refer to a unit that performs at least one function or operation, and may be implemented by hardware, software, or a combination of hardware and software. For example, part or all of at least one of the units 211 may be implemented as software stored in the memory 113 and executed by the processor 115.

[0054] In an embodiment of the present disclosure, the one or more units 211 may include, but are not limited to, a receiver unit 213, an automatic speech recognition unit 215, a non-speech sound classifier unit 217, a control parameter identifier unit 219, an IoT query service unit 221, a correlation engine unit 223, a control unit 225, a hyperparameter selection unit 227, and a list unit 229. The one or more units 211 may also include other units 231 for performing various other functions of the virtual assistant 107.

[0055] The receiver unit 213 may receive a voice command from the user via the I / O interface 111. The voice command may contain at least one non-voice sound from the user's surroundings. The user's voice command may be a voice sound. The non-voice sound from the user's surroundings may include at least one of the following: a sound related to the operation of multiple IoT devices 101 in the user's surroundings, or an ambient sound from the user's surroundings other than the sound related to the operation of multiple IoT devices 101. For example, when the user says "reduce speed", the receiver unit 213 may receive "reduce speed" as a voice command together with ambient sounds from a refrigerator and a chimney fan in the user's surroundings.

[0056] The automatic speech recognition unit 215 can detect the user's voice command. The automatic speech recognition unit 215 can use any speech recognition technology, such as automatic speech recognition (ASR) technology or speech to text (STT) technology, to detect the user's voice command. After detection, the automatic speech recognition unit 215 can provide a user command hypothesis by performing a speech to text operation. The output of the automatic speech recognition unit 215, such as the user command hypothesis, can be in text format. For example, the automatic speech recognition unit 215 can process the voice command for user command recognition. For example, when the user says "reduce speed", the receiver unit 213 can receive "reduce speed" as a voice command together with ambient sounds from refrigerators and chimney fans in the user's surroundings. The automatic speech recognition unit 215 can only detect the user's voice command, i.e., "reduce speed". In the "reduce speed" example above, the user's voice command may be an incomplete voice command because the voice command does not mention at least one target IoT device.

[0057] The non-speech sound classifier unit 217 can be trained with non-speech data from various IoT devices and ambient sounds from the user's surroundings in addition to non-speech data from various IoT devices to perform effective IoT device type detection. The non-speech sound classifier unit 217 can detect at least one non-speech sound from the user's surroundings in the voice command. The non-speech sound classifier unit 217 can apply sound processing technology to extract logarithmic Mel features such as Mel frequency cepstral coefficients (MFCC) from the voice command, and then can detect non-speech data by using a convolutional neural network (CNN) algorithm for the logarithmic Mel features. The above detection method is only an example. Non-speech sound classification is not limited to CNN algorithms based on MFCC. After detection, the non-speech sound classifier unit 217 can classify the non-speech sound to identify one or more appropriate IoT devices. Moreover, the non-speech sound classifier unit 217 can detect the location where the user's voice command is detected, and can use the detected location to identify one or more appropriate IoT devices. For example, the non-speech sound classifier unit 217 can process the non-speech sound to identify the relevant IoT device. For example, when the user says "reduce speed", the receiver unit 213 may receive "reduce speed" as a voice command together with ambient sounds from a refrigerator and a chimney fan in the user's surroundings. The non-voice sound classifier unit 217 may detect ambient sounds other than sounds related to the operation of IoT devices. The non-voice sound classifier unit 217 may detect only ambient sounds from a refrigerator and a chimney fan, and may identify the refrigerator and the chimney fan as appropriate IoT devices.

[0058] The automatic speech recognition unit 215 and the non-speech sound classifier unit 217 can process the corresponding data of the received speech command in parallel. For example, the received speech command can be provided to the automatic speech recognition unit 215 and the non-speech sound classifier unit 217 for parallel processing at the same time.

[0059] The control parameter identifier unit 219 may receive a user command hypothesis from the automatic speech recognition unit 215, and may identify one or more IoT device control parameters in the user command hypothesis. The control parameter identifier unit 219 may be part of a natural language processing (NLP) system. For example, the control parameter identifier unit 219 may process the user's voice command by using NLP techniques such as sentence segmentation, tokenization, part of speech tags, and entity recognition to identify the IoT device control parameters. For example, in the user's voice command, i.e., "reduce speed", the control parameter identifier unit 219 may identify "reduce" and "speed" as control parameters.

[0060] The IoT query service unit 221 can receive one or more pieces of IoT device information identified by the non-speech sound classifier unit 217. Based on the identified IoT device information, the IoT query service unit 221 can obtain the operating status of the corresponding IoT device. The operating status may include the IoT device status. The IoT device status may mean that the IoT device is turned on or off, and the operating status of the IoT device may mean the operating mode. For example, a chimney fan may run at a fan speed of 4, while the possible fan speeds are 1 to 5. The IoT query service unit 221 can obtain the operating status by directly querying the IoT device or by querying the repository 103, which stores the operating status of all available IoT devices in real time. For example, once the non-speech sound classifier unit 217 identifies the refrigerator and the chimney fan as appropriate IoT devices, the IoT query service unit 221 can obtain from the repository 103 that the operating status of the refrigerator is "off" and the operating status of the chimney fan is "on" and "4".

[0061] The correlation engine unit 223 may receive the IoT device control parameters from the control parameter identifier unit 219, and receive the operating status of one or more IoT devices from the IoT query service unit 221. The correlation engine unit 223 may receive a voice command from the automatic speech recognition unit 215. The correlation engine unit 223 may receive a non-speech sound classification result from the non-speech sound classifier unit 217. The correlation engine unit 223 may identify the correlation of at least two of the voice command, the non-speech sound classification result, the operating status of one or more IoT devices, or the IoT device control parameters. The correlation engine unit 223 may identify the correlation between the operating status of one or more IoT devices and the IoT device control parameters. The correlation engine unit 223 may identify the correlation between the non-speech sound classification result, the operating status of one or more IoT devices, and the IoT device control parameters. Next, the correlation engine unit 223 may select at least one target IoT device from the one or more IoT devices based on the correlation. For example, when receiving from the IoT query service unit 221 that the operating state of the refrigerator is "off" and the operating state of the chimney fan is "on" and "4" and receiving from the control parameter identifier unit 219 that the control parameters are "reduced" and "speed", the correlation engine unit 223 may determine that: "speed" is related to the chimney fan. In this case, because the refrigerator is in the "off" state, the correlation engine unit 223 may not consider the refrigerator. The correlation engine unit 223 may be part of the NLP system.

[0062] In addition to the correlation engine unit 223 , the virtual assist device 107 may also include a multi-context loader unit and an intent parser and response generator unit.

[0063] The multi-context loader unit may receive at least one target IoT device selected from the correlation engine unit 223 or in the candidate list. The multi-context loader unit may load a concurrent context stack for each selected IoT device for user command matching. The multi-context loader unit may maintain the current operating state of each selected IoT device for accurate user command determination. The multi-context loader unit may be part of an NLP system. For example, when IoT devices such as a television, an air conditioner, and a vacuum cleaner are selected by the correlation engine unit 223, the multi-context loader unit may load the context of the selected IoT device into the NLP system.

[0064] The intent parser and response generator unit can parse the user command intent to one or more most appropriate IoT devices. The intent parser and response generator unit can generate a response for performing the required task by specifying the task of the IoT device. The intent parser and response generator unit can be part of an NLP system. For example, consider a case where a user issues the command "Hi Bixby! Please be quiet. I have to answer the phone." In addition to TVs, air conditioners, and vacuum cleaners, there may be a variety of IoT devices (e.g., light bulbs and door handles) that are confirmed to be operating near the user command. In this case, the intent parser and response generator unit can parse the user command request "quiet" only to devices that can produce loud noises (such as TVs, air conditioners, and vacuum cleaners).

[0065] The control unit 225 may perform an operation of controlling the operation of at least one selected target IoT device received from the correlation engine unit 223. For example, when the user command is "mute the device" and the correlation engine provides at least one target IoT device such as a TV, an air conditioner, and a vacuum cleaner and their respective operating states "on", "fan on", and "on", the control unit 225 may mute the TV, turn off the fan of the air conditioner, and turn off the vacuum cleaner.

[0066] The hyperparameter selection unit 227 may obtain a list of IoT devices by determining the positioning and / or location of the virtual assistant device 107 according to whether the virtual assistant device 107 is fixed or movable. For example, there are two categories of IoT devices. One category involves dynamic / movable IoT devices in a house, such as a mobile terminal and a vacuum cleaner, and the other category involves fixed / static IoT devices in a house, such as a speaker, a TV, and a refrigerator. When the virtual assistant device 107 as a part of the IoT device is fixed, the hyperparameter selection unit 227 may know the location of the IoT device and may obtain the IoT device that can be used in the room / scene of the house. When the virtual assistant device 107 as a part of the IoT device is movable, the hyperparameter selection unit 227 may check the location of the IoT device. When the hyperparameter selection unit 227 may know the location of the IoT device, the hyperparameter selection unit 227 may obtain the IoT device that can be used in the room / scene of the house. When the hyperparameter selection unit 227 may not know the location of the IoT device, the hyperparameter selection unit 227 may obtain all IoT devices that can be used in the house. In order to generate or obtain one or more appropriate IoT devices, a list of IoT devices may be provided from the hyperparameter selection unit 227 to the non-speech sound classifier unit 217 .

[0067] In the case where the correlation engine unit 223 selects one or more target IoT devices, the list unit 229 may receive the output of the correlation engine unit 223. The list unit 229 may provide a list including target IoT devices to the user so that the user can select via an electronic device related to the user, such as Figure 3 For example, the list may include target IoT devices, such as a Samsung television (TV), an air conditioner, and a vacuum cleaner, which may be included in the plurality of IoT devices 101.

[0068] Figure 4A and Figure 4B is a flowchart illustrating a command parsing method based on non-voice sounds in an IoT environment according to an embodiment of the present disclosure.

[0069] Reference Figure 4A and Figure 4B , the command parsing method 400 may include one or more operations for performing command parsing based on non-speech sounds in an IoT environment, and may be described in the general context of computer-executable instructions. In general, computer-executable instructions may include routines, programs, objects, components, data structures, processes, units, and functions that perform specific functions or implement specific abstract data types.

[0070] The order in which the operations of command parsing method 400 are described should not be construed as a limitation, and any number of the described operations may be combined in any order to implement command parsing method 400. Additionally, individual operations may be omitted from command parsing method 400 without departing from the scope of the subject matter described herein. Furthermore, command parsing method 400 may be implemented in any suitable hardware, software, firmware, or a combination thereof.

[0071] In operation S401, a voice command from a user may be received by the receiver unit 213 through the I / O interface 111. The voice command may include at least one IoT device control parameter.

[0072] In operation S403, the automatic speech recognition unit 215 and the non-speech sound classifier unit 217 may detect the user's voice command and at least one non-speech sound from the user's surroundings in the received voice command. The analysis of at least one non-speech sound from the user's surroundings may be triggered based on the detection of the presence of at least one IoT device control parameter in the user's voice command.

[0073] The automatic speech recognition unit 215 can process the user's voice command to identify the IoT device control parameter. Here, each IoT device control parameter can be associated with one or more IoT devices. In addition, the IoT device control parameter can include at least one of a command, a component type, or a target IoT device.

[0074] The processing of the user's voice command and the analysis of at least one non-voice sound from the user's surrounding environment can be performed in parallel.

[0075] In operation S405, the non-speech sound classifier unit 217 may identify one or more IoT devices in the user's surroundings based on at least one of the detected non-speech sounds or the location where the user's voice command is detected. The non-speech sound from the user's surroundings may include at least one of the following: a sound related to the operation of the IoT device in the user's surroundings, or an ambient sound from the user's surroundings other than the sound related to the operation of the IoT device. At least one non-speech sound may be detected from the user's surroundings or from the surroundings of the virtual assistant 107.

[0076] In operation S407, the correlation engine unit 223 may select at least one target IoT device from the identified one or more IoT devices based on the correlation between the current operating state of each of the identified one or more IoT devices and the identified IoT device control parameters. The operating state of the one or more IoT devices may be received from the IoT query service unit 221, and the IoT device control parameters may be received from the control parameter identifier unit 219.

[0077] In operation S409, the control unit 225 may execute the received IoT device control parameter on the selected at least one target IoT device. The control unit 225 may control the operation of the selected at least one target IoT device based on the non-voice sound to implement command parsing.

[0078] When the correlation engine unit 223 selects two or more target IoT devices, it can perform Figure 4B The control method 411 may include operations S413 to S417.

[0079] In operation S413, the list unit 229 may provide a list of at least one target IoT device to the user via an electronic device related to the user. The list may be provided when the correlation engine unit 223 selects two or more target IoT devices from the identified one or more IoT devices.

[0080] In operation S415, the receiver unit 213 may receive an input from a user via an electronic device to control the operation of at least one target IoT device. The input from the user may include a selection of at least one target IoT device. The electronic device may include a mobile terminal, a speaker, a television, a refrigerator, a washing machine, a chimney, or any other device having IoT functionality.

[0081] In operation S417 , the control unit 225 may control the operation of at least one target IoT device based on the user input.

[0082] Figure 5 is a flowchart illustrating a method of obtaining a list of IoT devices having correlation with non-speech sounds according to an embodiment of the present disclosure.

[0083] like Figure 5 As shown, the method 500 for obtaining a list according to an embodiment of the present disclosure may include one or more operations for obtaining a list of IoT devices associated with non-speech sounds, and may be described in the general context of computer-executable instructions. In general, computer-executable instructions may include routines, programs, objects, components, data structures, processes, units, and functions that perform specific functions or implement specific abstract data types.

[0084] The order in which the operations of method 500 are described should not be construed as a limitation, and any number of the described operations may be combined in any order to implement method 500. Additionally, individual operations may be omitted from method 500 without departing from the scope of the subject matter described herein. Furthermore, method 500 may be implemented in any suitable hardware, software, firmware, or a combination thereof.

[0085] In operation S501, the hyperparameter selection unit 227 may determine the location where the user voice command is received to identify one or more IoT devices present near the location. IoT devices may be divided into two groups: (i) static IoT devices that are not moved or fixed to a specific location, such as a TV, a washing machine, and a refrigerator, and (ii) dynamic IoT devices that are moved or not fixed to any location, such as a mobile terminal and a vacuum cleaner in a house. The locations of static IoT devices may be stored in the repository 103. This operation may be performed when the virtual auxiliary device 107 is fixed, i.e., immovable. For example, when the virtual auxiliary device 107 is a fixed device such as a speaker, a TV, or a refrigerator, the virtual auxiliary device 107 may know the location of the IoT device from the repository 103, and may obtain the available IoT devices in the room / scene.

[0086] In operation S503, the hyperparameter selection unit 227 may determine the location of the virtual assistant device 107 in the IoT environment. This operation may be performed when the virtual assistant device 107 is dynamic, i.e., movable. For example, when the virtual assistant device 107 is a dynamic device such as a mobile terminal or a vacuum cleaner, the virtual assistant device 107 may determine its location in the room / scene. The IoT query service unit 221 may assist the hyperparameter selection unit 227 in determining the location. In an embodiment of the present disclosure, the location may be determined based on any specific identified IoT device, a combination of an IoT device and a known location, or an IoT scene. For example, during user command processing, the non-speech sound classifier unit 217 may identify that the TV is turned on with the help of the IoT query service unit 221. This confirms that the virtual assistant device 107 may be in the same location as the TV. As another example, during user command processing, the non-speech sound classifier unit 217 may identify the presence of two or more IoT devices belonging to the same IoT scene with the help of the IoT query service unit 221. For example, the sounds of a refrigerator and a chimney are identified in the same IoT scene. This potentially means that the virtual assistant 107 is also in the same IoT scene. In this case, the scene is the "kitchen IoT scene".

[0087] In operation S505, the hyperparameter selection unit 227 may determine one or more IoT devices that exist near the location of the virtual assistant device 107. This operation is performed when the location of the virtual assistant device 107 is known. For example, when the location of the virtual assistant device 107 is known, the virtual assistant device 107 may obtain a list of available IoT devices in the room / scene. In this case, the IoT query service unit 221 may provide the list of available IoT devices in the same location or IoT scene to the virtual assistant device 107 together with the status of each IoT device. One way to implement this is to have the user create an IoT room / scene and add one or more immovable IoT devices to the room / scene. When the virtual assistant device 107 is a movable device, the room / scene of the virtual assistant device 107 may be identified as long as the sound of the nearest fixed IoT device is detected.

[0088] In operation S507, when the location of the virtual assistant 107 is unknown, the hyperparameter selection unit 227 may determine all IoT devices connected to the virtual assistant 107. Information about all IoT devices connected to the virtual assistant 107 may be obtained from the repository 103, which stores a list of a plurality of IoT devices 101, which may include both static IoT devices and dynamic IoT devices. For example, when the location of the virtual assistant 107 is unknown, the virtual assistant 107 may obtain all available static IoT devices and dynamic IoT devices in the house.

[0089] The information from operation S501 to operation S507 may be provided from the hyperparameter selection unit 227 to the non-speech sound classifier unit 217 to generate or obtain one or more appropriate IoT devices.

[0090] The following is based on Figure 4A , Figure 4B and Figure 5 Show some examples.

[0091] Example 1: Consider a scenario in which a user is in a kitchen listening to a recipe on an electronic device. The user cannot listen to the recipe due to the ambient or environmental sounds of a refrigerator and a chimney fan. In order to clearly listen to the recipe, the user issues a command, "Hey, Bixby! Can't hear you. Lower the speed." The virtual assistant device 107 recognizes the non-voice data, i.e., the sounds of the refrigerator and the chimney fan. The virtual assistant device 107 also recognizes from the refrigerator and the chimney fan that lowering the speed is related to the chimney fan and not the refrigerator, and lowers the speed of the chimney fan. Now, the user is listening to the recipe without any sound from the chimney fan. This example relates to a case in which a single device, i.e., a chimney fan, is controlled based on multiple unique non-voice data, i.e., the ambient sounds of the refrigerator and the chimney fan.

[0092] Example 2: Consider a scenario where a user receives a call through his mobile phone in the living room. A vacuum cleaner, a TV, and an air conditioner are running in the living room. In order to make a call without any interruptions, the user issues a command "Hi Bixby! Keep quiet. I want to answer the call". The virtual assistant device 107 of the present disclosure recognizes non-voice data, i.e., the sounds of the vacuum cleaner, the TV, and the air conditioner. The virtual assistant device 107 also recognizes that keeping quiet is related to all devices in the living room (i.e., the vacuum cleaner, the TV, and the air conditioner) except for the ringing phone itself. In this case, the virtual assistant device 107 stops the vacuum cleaner, mutes the TV, and turns off the fan of the air conditioner. The user now answers the call without having to worry about any sounds from the devices in the living room. This example relates to the following situation: controlling multiple devices, i.e., a vacuum cleaner, a TV, and an air conditioner based on multiple unique non-voice data, i.e., the ambient sounds of the vacuum cleaner, the TV, and the air conditioner.

[0093] Example 3: Consider a scenario in which a user is washing dishes in a sink in a kitchen. In the kitchen, there are lights near the sink and dishwasher. In order to illuminate the kitchen, the user issues a command "Hey, Bixby! Turn on the lights". The virtual assistant device 107 of the present disclosure recognizes non-voice data, i.e., the sounds of running water and a dishwasher. Based on the non-voice data, the virtual assistant device 107 recognizes that the location is the kitchen, and then recognizes that the devices in the kitchen are lights and a dishwasher. The virtual assistant device 107 recognizes that turning on the lights is related to the lights near the sink in the kitchen, but not to lights elsewhere. In this case, the virtual assistant device 107 turns on the lights near the sink in the kitchen. This example relates to a case where one or more devices (i.e., lights) are controlled based on the location of a non-voice sound (i.e., the ambient sound of running water and a dishwasher). Moreover, this example relates to a case where a device that does not operate or make a sound is controlled.

[0094] Example 4: Consider a scenario where a user is driving a car with the sunroof open. In order to make a call, the user issues a command "Hi, Bixby! Call Jacob". The virtual assistant device 107 of the present disclosure recognizes non-voice data, i.e., noise outside the car. Based on the non-voice data, the virtual assistant device 107 recognizes that the sunroof of the car is open. The virtual assistant device 107 closes the sunroof and makes a call to contact Jacob from the phone book. This example relates to a case where one or more devices, i.e., the sunroof of the car, are controlled during a session with the virtual assistant device 107. In addition, this example relates to a case where an IoT device capable of performing operations related to the execution of a user's voice command is controlled, rather than the direct target IoT device of the user's voice command. In addition, this example relates to a case where an IoT device capable of performing operations related to the execution of a user's voice command is controlled when the user's voice command is complete, i.e., when the target IoT device of the voice command is specified.

[0095] Example 5: Consider another scenario in which a user is listening to a recipe on an electronic device in the kitchen. The user cannot listen to the recipe due to the ambient or environmental sounds of the refrigerator and chimney fan. In order to clearly listen to the recipe, the user issues a command "Hey, Bixby! Can't hear you. Lower the speed?". The virtual assistant device 107 of the present disclosure recognizes non-voice data, i.e., the sounds of the refrigerator and the chimney fan. The virtual assistant device 107 also recognizes from the refrigerator and the chimney fan that lowering the speed is related to the chimney fan, not the refrigerator, and asks the user "Do you want me to lower the chimney fan speed?". When receiving a user voice confirmation such as "yes", the virtual assistant device 107 lowers the speed of the chimney fan. The user continues to listen to the recipe without any sound from the chimney fan. This example relates to the following situation: controlling a single device, i.e., a chimney fan, after confirmation from the user (rather than receiving confirmation via the user interface of the electronic device).

[0096] Example 6: Consider a scenario in which a user is trying to put the user's baby to sleep in a bedroom. A vacuum cleaner, a television, and an air conditioner are running in the bedroom. To reduce the sounds in the bedroom, the user issues a command, "Hey Bixby! Keep quiet. My baby is going to sleep." The virtual assistant device 107 of the present disclosure recognizes non-voice data, i.e., the sounds of the vacuum cleaner, the television, and the air conditioner. The virtual assistant device 107 also recognizes that keeping quiet is related to all the devices in the bedroom (i.e., the vacuum cleaner, the television, and the air conditioner). In this case, the virtual assistant device 107 provides a list of devices including the vacuum cleaner, the television, and the air conditioner to the user interface of the user's mobile device, along with operations to keep the devices quiet (i.e., mute the TV, reduce the speed of the air conditioner, and stop the vacuum cleaner). Figure 3 A similar example is shown. Because the user wants to use the air conditioner, the user issues a voice command "stop the vacuum cleaner and TV", or selects the vacuum cleaner and TV through a touch command on the user interface of the user's mobile device. When receiving the confirmation, the virtual assistant device 107 stops the vacuum cleaner and TV while the air conditioner is still running. This example relates to the following case: after receiving a confirmation from the user via the user interface of the electronic device or a voice command, multiple devices, namely, the vacuum cleaner and the TV, are controlled.

[0097] Example 7: Consider a scenario in which a user in a living room wants to listen to messages received on his or her mobile phone. In the living room, a vacuum cleaner, a television, and an air conditioner are running, and there is the user's mobile phone. In order to listen to the messages on the mobile phone uninterruptedly, the user issues a command "Hi Bixby! Keep quiet. I want to listen to messages". The virtual assistant device 107 of the present disclosure recognizes non-voice data, i.e., the sounds of the vacuum cleaner, the television, and the air conditioner. The virtual assistant device 107 also recognizes that keeping quiet is related to all devices in the living room except the mobile phone (i.e., the vacuum cleaner, the television, and the air conditioner) because the message is to be read from the mobile phone. In this case, the virtual assistant device 107 presents a list of devices including the vacuum cleaner, the television, and the air conditioner to the user interface of the user's mobile device together with the operation to keep the devices quiet (i.e., mute the TV, reduce the speed of the air conditioner, and stop the vacuum cleaner). Figure 3 A similar example is shown. Because the user wants to use the air conditioner and the vacuum cleaner, the user issues a voice command of "mute the TV" or selects the TV through a touch command on the user interface of the user's mobile device. When receiving the confirmation, the virtual assistant device 107 stops the TV while the air conditioner and the vacuum cleaner are still running, and plays the message received on the mobile phone. This example involves the following situation: controlling one or more devices, namely, the air conditioner, the vacuum cleaner, and the mobile phone, by resolving the user command intention conflict, namely, avoiding muting the mobile phone together with the TV.

[0098] The above examples may include an incomplete voice command without mentioning at least one target IoT device, or may include a user's voice command lacking any target IoT device.

[0099] Embodiments of the present disclosure may provide a command parsing method in an Internet of Things (IoT) environment, the command parsing method comprising: receiving, by a virtual assistant device, a voice command from a user, wherein the voice command comprises at least one IoT device control parameter; detecting, by the virtual assistant device, at least one non-voice sound while receiving the voice command; identifying, by the virtual assistant device, one or more IoT devices in the user's surrounding environment based on the detected at least one non-voice sound; selecting, by the virtual assistant device, at least one target IoT device from the identified one or more IoT devices based on a correlation between a current operating state of each of the identified one or more IoT devices and the identified IoT device control parameter; and executing, by the virtual assistant device, the received IoT device control parameter on the selected at least one target IoT device.

[0100] Embodiments of the present disclosure may provide a virtual assistant device for command parsing in an Internet of Things (IoT) environment, the virtual assistant device comprising: a processor; and a memory communicatively coupled to the processor, wherein the processor is configured to execute instructions stored in the memory so as to: receive a voice command from a user, wherein the voice command includes at least one IoT device control parameter; while receiving the voice command, detect at least one non-voice sound; based on the detected at least one non-voice sound, identify one or more IoT devices in the user's surrounding environment; based on the correlation between the current operating state of each of the one or more identified IoT devices and the identified IoT device control parameter, select at least one target IoT device from the identified one or more IoT devices; and execute the received IoT device control parameter on the selected at least one target IoT device.

[0101] In an embodiment of the present disclosure, processing of a voice command from a user and analysis of at least one non-voice sound may be performed in parallel.

[0102] In an embodiment of the present disclosure, analysis of at least one non-voice sound may be triggered based on detection of the presence of at least one IoT device control parameter in a voice command.

[0103] In an embodiment of the present disclosure, the non-voice sound from the user's surrounding environment may include at least one of the following: a sound related to the operation of an IoT device in the user's surrounding environment, or an ambient sound from the user's surrounding environment other than a sound related to the operation of an IoT device.

[0104] In an embodiment of the present disclosure, sounds related to the operation of the IoT device and environmental sounds from the user's surroundings may be stored in a repository related to the virtual assistant device.

[0105] In an embodiment of the present disclosure, each IoT device control parameter may be associated with one or more IoT devices, wherein the association between each IoT device control parameter and the one or more IoT devices is stored in a repository associated with the virtual auxiliary device.

[0106] In an embodiment of the present disclosure, when the location of the virtual assistant device is fixed, the virtual assistant device may enable the processor to determine the location where the user voice command is received to identify one or more IoT devices existing near the location.

[0107] In an embodiment of the present disclosure, when the location of the virtual assistant device is movable, the virtual assistant device may cause the processor to determine the location of the virtual assistant device in the IoT environment.

[0108] In an embodiment of the present disclosure, when the location of the virtual assistant device is known, the virtual assistant device may cause the processor to detect one or more IoT devices existing near the location of the virtual assistant device.

[0109] In an embodiment of the present disclosure, when the location of the virtual assistant device is unknown, the virtual assistant device may cause the processor to determine all IoT devices connected to the virtual assistant device.

[0110] In an embodiment of the present disclosure, the operating status of one or more identified IoT devices may be received from a repository associated with a virtual auxiliary device, wherein the operating status of the one or more identified IoT devices is maintained in real time in the repository associated with the virtual auxiliary device.

[0111] In an embodiment of the present disclosure, the user's voice command may include an incomplete voice command from the user that does not mention at least one target IoT device.

[0112] In an embodiment of the present disclosure, controlling the operation of at least one selected target IoT device may include: when more than one target IoT device is selected from one or more identified IoT devices, providing a list of one or more target IoT devices to a user via an electronic device associated with the user by a virtual assistant device; receiving input from the user via the electronic device by the virtual assistant device to control the operation of the one or more target IoT devices, wherein the input from the user includes a selection of the one or more target IoT devices; and controlling the operation of the one or more target IoT devices based on the user input by the virtual assistant device.

[0113] In an embodiment of the present disclosure, at least one non-speech sound may be detected from the user's surroundings or from the surroundings of the virtual assistant.

[0114] In an embodiment of the present disclosure, executing the identified IoT device control parameter on the selected at least one target IoT device may include: controlling the operation of the selected at least one target IoT device based on non-voice sound by the virtual assistant device to implement command parsing.

[0115] In an embodiment of the present disclosure, the IoT device control parameter may include at least one of a command, a component type, or a target IoT device.

[0116] Figure 6 is a block diagram showing the configuration of a voice command parsing device according to an embodiment of the present disclosure. Figure 6According to an embodiment of the present disclosure, the speech command parsing device 600 may include a processor 601 and a memory 603, wherein the memory 603 stores one or more instructions executable by the processor 601. One or more processors 601 may be provided. One or more memories 603 may be provided. Figure 7 An example of the operations performed by the voice command parsing apparatus 600 when the processor 601 executes one or more instructions stored in the memory 603 is described in detail. The voice command parsing apparatus 600 may be the virtual assistant 107 or a part of the virtual assistant 107.

[0117] Figure 7 1 is a flowchart showing the process of an operation method of a voice command parsing device based on non-voice sound in an IoT environment according to an embodiment of the present disclosure. Figure 7 , the processor 601 of the voice command parsing device 600 may identify the user voice command in the input sound in operation S701, and may analyze the non-voice sound in the input sound in operation S702. The non-voice sound may be a sound generated simultaneously or approximately simultaneously with the voice command. For example, the non-voice sound may be a sound generated within a certain period of time from the start time point and / or the end time point of the voice command.

[0118] The operation S701 of recognizing a voice command and the operation S702 of analyzing a non-voice sound may be performed in parallel or simultaneously. When two operations are performed simultaneously, it does not mean that the two operations have the same starting time point, but may mean that the execution time of the two operations may overlap each other.

[0119] In operation S705, the processor 601 may determine at least one target IoT device related to the execution of the voice command based on the analysis result of the non-voice sound. The target IoT device is determined based on the analysis result of the non-voice sound, but is not necessarily a device that is currently running or making a sound. The processor 601 may determine the target IoT device based on the correlation between the analysis result of the non-voice sound and the voice command.

[0120] The processor 601 may execute the voice command on the determined target IoT device. The processor 601 may control the determined target IoT device to execute the voice command. The processor 601 may determine the operation to be performed by the target IoT device related to the execution of the voice command based on the analysis result of the non-voice sound, and may control the target IoT device to perform the operation. The processor 601 may determine the operation to be performed by the target IoT device based on the correlation between the analysis result of the non-voice sound and the voice command.

[0121] Processor 601 may analyze non-speech sounds when the recognized voice command is incomplete. Processor 601 may analyze non-speech sounds when the target IoT device of the voice command is not fully specified in the recognized voice command. Processor 601 may analyze non-speech sounds based on the recognized voice command not including at least one target IoT device required to execute the voice command. Processor 601 may analyze non-speech sounds only when the recognized voice command does not include at least one target IoT device required to execute the voice command.

[0122] The processor 601 may analyze the non-voice sound when the recognized voice command is complete. The processor 601 may analyze the non-voice sound even when the recognized voice command includes all target IoT devices required to execute the voice command. In this case, the processor 601 may cause the target IoT device determined based on the analysis result of the non-voice sound to perform operations related to the execution of the voice command.

[0123] The processor 601 may determine multiple target IoT devices related to the execution of the voice command based on the analysis result of the non-voice sound. The processor 601 may determine the operations to be performed by the determined multiple target IoT devices related to the execution of the voice command based on the analysis result of the non-voice sound, and may control the multiple target IoT devices to perform the operations respectively. The processor 601 may control the multiple target IoT devices to simultaneously perform the operations to be performed related to the execution of the voice command.

[0124] When analyzing non-speech sounds, the processor 601 may compare the non-speech sounds with predefined sounds of IoT devices. The processor 601 may determine one or more IoT devices currently running around by comparing the non-speech sounds with predefined sounds of IoT devices. The processor 601 may determine at least one of the IoT devices currently running around as a target IoT device. The processor 601 may determine devices other than the IoT devices currently running around as target IoT devices based on the IoT devices currently running around. The processor 601 may determine the operation to be performed by the target IoT device based on the IoT devices currently running around. The IoT device currently running around may be the IoT device that is making a sound.

[0125] When analyzing non-speech sounds, the processor 601 may detect environmental sounds other than sounds related to the operation of the IoT device from the non-speech sounds. The processor 601 may determine the target IoT device based on the detected environmental sounds. The processor 601 may determine the operation to be performed by the target IoT device based on the detected environmental sounds.

[0126] Figure 8Detailed flowchart showing the process of the operation method of the voice command parsing device based on non-voice sound in IoT environment according to an embodiment of the present disclosure. Operations such as operation S701, operation S703 and operation S705 may correspond to the above reference Figure 7 Similar operations are described above, and repeated description of these operations has been omitted.

[0127] Reference Figure 8 In operation S801, the processor 601 may detect an IoT device control parameter in a voice command (e.g., a voice command hypothesis). The processor 601 may analyze non-voice sounds based on the detected IoT device control parameters. The IoT device control parameters may be directly included in the voice command, or may be extracted by analyzing the voice command. For example, when the user's voice command is "Hi Bixby! Keep quiet", "quiet" included in the voice command may be detected as a control parameter, and when the user's voice command is "Hi Bixby! It's too loud", "quiet" may be extracted as a control parameter by analyzing the voice command.

[0128] The processor 601 may analyze non-voice sounds based on detecting IoT device control parameters in the voice command. The processor 601 may analyze non-voice sounds only when IoT device control parameters are detected in the voice command. The processor 601 may analyze non-voice sounds when the target IoT device for operations related to the IoT device control parameters is not fully specified in the voice command. The processor 601 may analyze non-voice sounds when the recognized voice command does not include at least one target IoT device required to perform operations related to the detected IoT device control parameters. The processor 601 may analyze non-voice sounds when no IoT device control parameters are detected in the voice command.

[0129] The processor 601 may analyze the non-speech sound based on one or more IoT devices pre-matched with the detected IoT device control parameters. For example, when comparing the non-speech sound with the pre-defined sound of the IoT device, the processor 601 may compare the non-speech sound only with the pre-defined sound of the IoT device pre-matched with the detected IoT device control parameters, thereby greatly reducing computing resources and improving processing speed.

[0130] The processor 601 may select at least one filter bank to be used for analyzing non-speech sounds based on the detected IoT device control parameters. Thus, computing resources may be reduced and processing speed may be increased. The filter bank may be, for example, a Mel filter bank.

[0131] The processor 601 may determine the target IoT device and / or the operation to be performed by the target IoT device based on the correlation of at least two of the recognized voice commands, the detected IoT device control parameters, the analysis results of the non-voice sounds, or the IoT device information received from the repository 103 associated with the voice command parsing device 600. The repository 103 may be an IoT server, or a database associated with the IoT server. The processor 601 may determine the target IoT device and / or the operation to be performed by the target IoT device based on the correlation between the detected IoT device control parameters and the analysis results of the non-voice sounds. The processor 601 may determine the target IoT device and / or the operation to be performed by the target IoT device based on the correlation between the detected IoT device control parameters, the analysis results of the non-voice sounds, and the IoT device information received from the repository 103.

[0132] The processor 601 may determine one or more possible adjacent IoT devices, for example, one or more IoT devices that may exist in the area of ​​the voice command parsing device 600 or the area of ​​the user who issues the voice command, and the processor 601 may analyze the non-voice sound based on the one or more possible adjacent IoT devices. The processor 601 may only consider the IoT devices that may exist around the voice command parsing device 600 or the user who issues the voice command, rather than considering all IoT devices within the IoT service, thereby reducing computing resources and improving processing speed. For example, when comparing a non-voice sound with a predefined sound of an IoT device, the processor 601 may compare the non-voice sound only with the predefined sound of one or more possible adjacent IoT devices, thereby greatly reducing computing resources and improving processing speed. The processor 601 may determine one or more possible adjacent IoT devices based on the non-voice sound. One or more possible adjacent IoT devices may be devices that are currently running or making a sound. One or more possible adjacent IoT devices may not be devices that are currently running or making a sound.

[0133] In operation S803, the processor 601 may determine one or more possible neighboring IoT devices based on the location information. The location information may be information related to the location of the voice command parsing apparatus 600 and / or the location of the user who issued the voice command.

[0134] The processor 601 may determine one or more possible neighboring IoT devices based on the location of the voice command parsing apparatus 600. The location of the voice command parsing apparatus 600 may include the location of a microphone that receives input sound.

[0135] The processor 601 may determine one or more possible neighboring IoT devices based on the location of the user issuing the voice command. The location of the user may be determined based on the location of the voice command parsing apparatus 600. The location of the user may be determined based on the direction in which the user's voice command is heard by using beamforming, etc. The processor 601 may determine the location of the user based on non-voice sounds heard in the direction in which the user's voice command is heard. The processor 601 may determine one or more possible neighboring IoT devices based on the direction in which the non-voice sounds are heard. Although the description will focus on the location of the voice command parsing apparatus 600, the description may also be applied to the location of the user issuing the voice command.

[0136] When the voice command parsing device 600 is a movable device, the voice command parsing device 600 can determine its own position by using any indoor / outdoor positioning technology or IoT service. The voice command parsing device 600 can determine its own position based on non-voice sounds. When the voice command parsing device 600 is a fixed device, the position of the voice command parsing device 600 can be specified by the user. For example, the user can specify fixed IoT devices (e.g., TVs, air conditioners, refrigerators, and virtual auxiliary devices) installed in each place (e.g., a living room, room, or kitchen) by using the user interface (UI) of the IoT service. Even when the voice command parsing device 600 is a fixed device, the voice command parsing device 600 can determine its own position based on positioning technology, IoT services, non-voice sounds, etc.

[0137] When the location of the voice command parsing device 600 is known, the processor 601 may determine the fixed IoT devices present around the voice command parsing device 600 as one or more possible adjacent IoT devices. The processor 601 may determine all movable IoT devices within the IoT service as one or more possible adjacent IoT devices. The processor 601 may determine the devices whose locations are unknown among the movable IoT devices within the IoT service that are not around the voice command parsing device 600 as one or more possible adjacent IoT devices. The processor 601 may determine all IoT devices within the IoT service and whose locations within the IoT service are unknown as one or more possible adjacent IoT devices. The case where the location of the voice command parsing device 600 is known may include the case where the voice command parsing device 600 is a fixed device. In addition, the case where the location of the voice command parsing device 600 is known may include the case where the voice command parsing device 600 is a movable device and its location is determined.

[0138] When the location of the voice command parsing apparatus 600 is unknown, the processor 601 may determine all IoT devices within the IoT service as one or more possible neighboring IoT devices. The situation where the location of the voice command parsing apparatus 600 is unknown may include the situation where the voice command parsing apparatus 600 is a movable device and its location is uncertain.

[0139] The processor 601 may determine one or more possible neighboring IoT devices based on the IoT device information received from the repository 103, for example, the location of the IoT device or the operating status of the IoT device. The processor 601 may determine one or more possible neighboring active IoT devices based on the IoT device information received from the repository 103, that is, one or more operating IoT devices that may exist around. The processor 601 may determine one or more possible neighboring active IoT devices based on the location of the voice command parsing device 600 and the IoT device information received from the repository 103.

[0140] The processor 601 can analyze the non-voice sound based on possible adjacent active IoT devices. The processor 601 can determine one or more active IoT devices around the voice command parsing device 600 based on one or more possible adjacent active IoT devices and non-voice sounds, that is, IoT devices currently actually running around the voice command parsing device 600 or the user who issued the voice command.

[0141] The analysis results of the non-voice sound may include one or more possible adjacent IoT devices, one or more possible adjacent active IoT devices, or one or more active IoT devices existing around the voice command parsing device 600 or the user who issued the voice command. The processor 601 may determine the target IoT device based on one or more possible adjacent IoT devices, one or more possible adjacent active IoT devices, or one or more active IoT devices existing around the voice command parsing device 600 or the user who issued the voice command. The processor 601 may determine the target IoT device and / or the operation to be performed by the target IoT device based on the correlation between one or more possible adjacent IoT devices, one or more possible adjacent active IoT devices, or one or more active IoT devices existing around the voice command parsing device 600 or the user who issued the voice command, the recognized voice command, the detected IoT device control parameter, the analysis results of the non-voice sound, or the IoT device information received from the repository 103.

[0142] The processor 601 may determine at least one of the one or more possible neighboring IoT devices as a target IoT device. The processor 601 may determine at least one of the one or more possible neighboring active IoT devices as a target IoT device. The processor 601 may determine at least one of the one or more active IoT devices existing around the voice command parsing apparatus 600 or the user who issued the voice command as a target IoT device.

[0143] The processor 601 may receive the operating status of one or more possible neighboring IoT devices from the repository 103. The processor 601 may determine the target IoT device and / or the operation to be performed by the target IoT device based on the correlation between the operating status of at least one IoT device that may exist in the surroundings and the detected IoT device control parameters received from the repository 103.

[0144] In an embodiment of the present disclosure, the processor 601 may determine one or more possible neighboring active IoT devices based on the location of the voice command parsing device 600 and the IoT device information received from the repository 103, may determine one or more active IoT devices existing around the voice command parsing device 600 or the user who issued the voice command based on the one or more possible neighboring active IoT devices and non-voice sounds, and may determine at least one target IoT device related to the execution of the voice command based on the one or more active IoT devices existing around the voice command parsing device 600 or the user who issued the voice command. The processor 601 may determine the target IoT device and / or the operation to be performed by the target IoT device based on the correlation between the one or more active IoT devices existing around the voice command parsing device 600 or the user who issued the voice command and the recognized voice command or the detected IoT device control parameter. The processor 601 may determine the target IoT device and / or the operation to be performed by the target IoT device based on the correlation between the one or more active IoT devices existing around the voice command parsing device 600 or the user who issued the voice command, the recognized voice command or the detected IoT device control parameter, and the IoT device information received from the repository 103. The determined target IoT device may be one of the one or more active IoT devices present around the voice command parsing apparatus 600 or the user who issues the voice command. The determined target IoT device may not be one of the one or more active IoT devices present around the voice command parsing apparatus 600 or the user who issues the voice command, but may be a device related to the execution of the recognized voice command.

[0145] In an embodiment of the present disclosure, the processor 601 may determine the location of the voice command parsing device 600 based on non-voice sounds, and may determine at least one target IoT device related to the execution of the voice command based on the determined location of the voice command parsing device 600.

[0146] Fig. 9 Detailed flowchart showing the process of an operation method of a voice command parsing device based on non-voice sound in an IoT environment according to an embodiment of the present disclosure.

[0147] Some advantages of the present disclosure are as follows.

[0148] Embodiments of the present disclosure provide an effective command parsing method based on non-voice sounds in an IoT environment in addition to a user's voice command.

[0149] Embodiments of the present disclosure provide a method that can understand an incomplete voice command from a user.

[0150] Embodiments of the present disclosure use the operating status of one or more IoT devices identified by a non-speech sound classifier unit to more accurately select one or more IoT devices.

[0151] Embodiments of the present disclosure allow a user to not remember the complete and exact commands used to perform a task to reduce the user's cognitive overload.

[0152] The proposed method can improve the end-user experience and thus enhance user reliability regarding artificial intelligence (AI) assistants.

[0153] Embodiments of the present disclosure may be implemented as computer executable code stored in a computer-readable recording medium. Examples of computer-readable recording media include all recording media, such as magnetic media, optical media, read-only memory (ROM), and random access memory (RAM). Computer-readable recording media may be provided as non-transitory storage media. Here, "non-transitory" means that the storage medium does not include a signal and is tangible, but it is not possible to distinguish whether the data is stored semi-permanently or temporarily on the storage medium. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data.

[0154] According to an embodiment of the present disclosure, the method according to various embodiments of the present disclosure may be provided in a computer program product. A computer program product is a product that can be purchased between a seller and a buyer. The computer program product may be stored and distributed in a computer-readable recording medium, or may be downloaded through an application store (e.g., Play Store TM) online distribution (e.g., download or upload), or directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product (e.g., downloadable application) may be temporarily generated or at least temporarily stored in a computer-readable recording medium, such as a manufacturer's server, an application store's server, or a memory of a relay server.

[0155] Furthermore, the term “unit” used herein may be a hardware component such as a processor or a circuit and / or a software component executed by a hardware component such as a processor.

[0156] Furthermore, throughout the disclosure, the expression "at least one of a, b, or c" means: only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c, or variations thereof.

[0157] According to the functions related to artificial intelligence (AI) disclosed in the present invention, a processor and a memory are used for execution. The processor may include one or more processors. In this case, the one or more processors may include a general-purpose processor such as a central processing unit (CPU), an application processor (AP) or a digital signal processor (DSP), a graphics processor such as a graphics processing unit (GPU) or a visual processing unit (VPU), or an AI processor such as a neural processing unit (NPU). One or more processors control the input data to be processed according to predefined operating rules or an A1 model stored in the memory. In an embodiment, when one or more processors are AI processors, the AI ​​processor may be designed in a hardware structure dedicated to processing a specific AI model.

[0158] Predefined operating rules or AI models can be created by learning. Here, "created by learning" means that as the basic AI model is trained according to the learning algorithm by using multiple training data, a predefined operating rule or AI model set to perform the desired characteristics (or purposes) is created. Such learning can be performed on a device in which the AI-based voice command parsing method according to the present disclosure is performed, or it can be performed by a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0159] The AI ​​model may include multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and the neural network operation is performed by the operation between the operation result of the previous layer and the multiple weight values. The weight value of the neural network layer can be optimized by the learning result of the AI ​​model. For example, multiple weight values ​​can be updated to reduce or optimize the loss value or cost value obtained by the AI ​​model during the learning process. The artificial neural network may include a deep neural network (DNN) such as a convolutional neural network (CNN), a recursive neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive deep neural network (BRDNN), or a deep Q network, but is not limited thereto.

[0160] Although the present disclosure has been specifically shown and described with reference to the embodiments of the present disclosure, these embodiments are non-limiting examples of the present disclosure and should be considered only in a descriptive sense and not for the purpose of limitation. It will be understood by those of ordinary skill in the art that the embodiments of the present disclosure can be easily modified in other specific forms without changing the technical spirit or basic features of the present disclosure. For example, each component described in a single type can be executed in a distributed manner, and the components described in a distributed type can be executed in a combined type. Although the present disclosure has been described by using specific terms, the terms are only used to illustrate the present disclosure and should not be interpreted as limiting the concept or scope of the present disclosure defined by the claims.

[0161] The scope of the present disclosure is indicated by the claims rather than the specific embodiments of the present disclosure, and it should be understood that the claims and all modifications or modified forms derived from the concepts and scopes of the claims and equivalents are included in the scope of the present disclosure. In addition, such equivalents are intended to include currently known equivalents and equivalents developed in the future, that is, any element developed to perform the same function, regardless of its structure.

Claims

1. A voice command parsing device, comprising: a memory configured to store instructions; as well as A processor configured to execute the instructions to: Recognize the user's voice command in the input sound, analyzing non-speech sounds included in the input sound, and Based on the analysis result of the non-voice sound, at least one target Internet of Things (IoT) device related to the execution of the voice command is determined.

2. The voice command parsing device according to claim 1, wherein: The processor is further configured to execute the instructions to analyze the non-voice sound when the voice command does not include at least one target IoT device required to execute the voice command.

3. The voice command parsing device according to claim 1, wherein: The processor is also configured to execute the instructions to determine an operation to be performed by the at least one target IoT device related to the execution of the voice command based on the analysis result of the non-voice sound.

4. The voice command parsing device according to claim 1, wherein: The processor is further configured to execute the instructions to: Based on the analysis result of the non-voice sound, determining a plurality of target IoT devices related to the execution of the voice command, determining operations to be performed by the plurality of target IoT devices related to the execution of the voice command based on the analysis result of the non-voice sound, and Control the multiple target IoT devices to perform the operations simultaneously.

5. The voice command parsing device according to claim 1, wherein: The processor is further configured to execute the instructions to compare the non-speech sound with a predefined sound of an IoT device.

6. The voice command parsing device according to claim 1, wherein: The processor is further configured to execute the instructions to detect environmental sounds that are not related to the operation of the IoT device from the non-speech sounds.

7. The voice command parsing device according to claim 1, wherein: The processor is further configured to execute the instructions to: detecting IoT device control parameters in the voice command, and The non-speech sound is analyzed based on the IoT device control parameter.

8. The voice command parsing device according to claim 7, wherein: The processor is further configured to execute the instructions to analyze the non-voice sound based on detecting the IoT device control parameter in the voice command.

9. The voice command parsing device according to claim 7, wherein: The processor is further configured to execute the instructions to analyze the non-speech sound based on one or more IoT devices pre-matched with the IoT device control parameters.

10. The voice command parsing device according to claim 7, wherein: The processor is also configured to execute the instructions to determine the at least one target IoT device based on a correlation between at least two of the voice command, the IoT device control parameter, the analysis result of the non-voice sound, or IoT device information received from a repository associated with the voice command parsing device.

11. The voice command parsing device according to claim 7, wherein: The processor is also configured to execute the instructions to determine the at least one target IoT device based on a correlation between the IoT device control parameters, the analysis results of the non-voice sound, and the IoT device information received from a repository associated with the voice command parsing device.

12. The voice command parsing device according to claim 1, wherein: The processor is configured to execute the instructions to: Identify one or more possible neighboring IoT devices, and The non-speech sound is analyzed based on the one or more possible neighboring IoT devices.

13. The voice command parsing device according to claim 12, wherein: The processor is further configured to execute the instructions to determine the one or more possible neighboring IoT devices based on the non-speech sound.

14. The voice command parsing device according to claim 12, wherein: The processor is further configured to execute the instructions to determine the one or more possible neighboring IoT devices based on the location of the voice command parsing device.

15. The voice command parsing device according to claim 14, wherein: The processor is further configured to execute the instructions to determine a location of the voice command parsing device based on the non-speech sound.

16. The voice command parsing device according to claim 1, wherein: The processor is further configured to execute the instructions to: determining one or more possible neighboring active IoT devices based on the location of the voice command parsing device and IoT device information received from a repository associated with the voice command parsing device, and The non-speech sound is analyzed based on the one or more possible neighboring active IoT devices.

17. The voice command parsing device according to claim 16, wherein: The processor is also configured to execute the instructions to determine one or more active IoT devices present around at least one of the voice command parsing device or the user based on the one or more possible neighboring active IoT devices and the non-voice sound.

18. An operating method of a voice command parsing device, the operating method comprising: Recognizing a user's voice command in input sound; analyzing non-speech sounds included in the input sound; as well as Based on the analysis result of the non-voice sound, at least one target Internet of Things (IoT) device related to the execution of the voice command is determined.

19. The operating method according to claim 18, further comprising: Detecting IoT device control parameters in the voice command; as well as Identify one or more possible neighboring IoT devices, Wherein, the non-speech sound is analyzed based on the IoT device control parameter and the one or more possible adjacent IoT devices.

20. A non-transitory computer-readable recording medium having recorded thereon instructions, the instructions, when executed by at least one processor, causing the at least one processor to perform the following operations: Recognizing a user's voice command in input sound; analyzing non-speech sounds included in the input sound; as well as Based on the analysis result of the non-voice sound, at least one target Internet of Things (IoT) device related to the execution of the voice command is determined.

Images