Method and apparatus for voice command resolution based on non-speech Sound in IoT Environment

Abstract

A voice command interpreter is disclosed, comprising a processor and a memory for storing instructions, wherein the processor recognizes a user's voice command in an input sound by executing the instructions, analyzes a non-voice sound in the input sound, and determines at least one target IoT device related to the execution of the voice command based on the analysis result of the non-voice sound.

Description

Technology Field

[0001] The present disclosure generally relates to Internet of Things (IoT) devices, and more specifically to a method and device for interpreting voice commands based on non-voice sound in an IoT environment. Background Technology

[0002] As technology advances, various types of devices can be interconnected to form a system of devices known as the Internet of Things (IoT). These devices can be controlled via user voice or text input. Voice assistant solutions enable users to perform various tasks, such as booking a taxi, playing music, and setting alarms, through voice or text commands. For assigned tasks, the voice assistant responds to the user through natural conversation generated by the IoT system's Natural Language Generator (NLG) module. Currently, NLG and natural conversation responses do not take into account surrounding non-verbal sounds for the optimal interpretation of voice commands. Furthermore, voice assistants fail to recognize possible actions the user can perform based on the surrounding audio scene and / or activities. Users may experience inconvenience if the voice assistant is unable to perform the desired action in response to a query. For example, suppose a user is watching television in the living room. Various background sounds are active from IoT devices such as the TV, air conditioner, and robot vacuum cleaner. When a call comes in, the user asks the voice assistant to be quiet so they can answer the call. In this situation, the user wants IoT devices such as TVs, air conditioners, and robot vacuums not to make any sound. However, in reality, the voice assistant does not take background non-audio sounds into account, muting the phone while leaving all other sound-producing IoT devices undisturbed. Additionally, the user must explicitly cancel the current conversation to switch to another task.

[0003] Meanwhile, unlike existing rule-based smart systems, Artificial Intelligence (AI) systems are systems in which machines learn, make judgments, and become smarter on their own. As AI systems improve their recognition rates and understand user preferences more accurately with use, existing rule-based smart systems are gradually being replaced by deep learning-based AI systems. AI technology consists of machine learning (deep learning) and component technologies that utilize machine learning. Machine learning is an algorithmic technology that independently classifies and learns the characteristics of input data, while component technologies are technologies that utilize machine learning algorithms such as deep learning, comprising fields such as linguistic understanding, visual understanding, reasoning / prediction, knowledge representation, and motion control. The problem to be solved

[0004] One embodiment of the present disclosure provides a method and device for interpreting voice commands based on non-voice sounds in an IoT environment, enabling accurate and efficient interpretation of a user's voice commands. means of solving the problem

[0005] As a technical means for achieving the technical problem described above, one embodiment of the present disclosure may provide a voice command interpreter comprising a processor and a memory for storing instructions, wherein the processor recognizes a user's voice command from an input sound by executing the instructions, analyzes a non-voice sound from the input sound, and determines at least one target IoT device related to the execution of the voice command based on the analysis result of the non-voice sound.

[0006] In one embodiment, the processor may provide a voice command interpreter that performs analysis of the non-voice sound when the voice command does not include at least one target IoT device required for the execution of the voice command.

[0007] In one embodiment, the processor may provide a voice command interpreter that determines an action to be performed by at least one target IoT device in relation to the execution of the voice command based on the analysis result of the non-voice sound.

[0008] In one embodiment, the processor may provide a voice command interpreter that determines a plurality of target IoT devices related to the execution of the voice command based on the analysis result of the non-voice sound, determines operations to be performed by each of the plurality of target IoT devices in relation to the execution of the voice command based on the analysis result of the non-voice sound, and controls the plurality of target IoT devices to simultaneously perform operations to be performed in relation to the execution of the voice command.

[0009] In one embodiment, the processor may provide a voice command interpreter that compares the non-voice sound with the sounds of predefined IoT devices when performing analysis of the non-voice sound.

[0010] In one embodiment, the processor may provide a voice command interpreter that detects ambient sounds other than sounds related to the operation of the IoT device from the non-voice sound while performing analysis of the non-voice sound.

[0011] In one embodiment, the processor may provide a voice command interpreter that detects IoT device control parameters in the voice command and performs analysis of the non-voice sound based on the IoT device control parameters.

[0012] In one embodiment, the processor may provide a voice command interpreter that performs analysis of the non-voice sound when an IoT device control parameter is detected in the voice command.

[0013] In one embodiment, the processor may provide a voice command interpreter that performs analysis of the non-voice sound based on one or more IoT devices pre-matched to the IoT device control parameters.

[0014] In one embodiment, the processor may provide a voice command interpreter that determines the target IoT device based on at least two correlations among the voice command, the IoT device control parameter, the analysis result of the non-voice sound, or IoT device information received from a repository associated with the voice command interpreter.

[0015] In one embodiment, the processor may provide a voice command interpreter that determines the target IoT device based on the correlation between the IoT device control parameters, the analysis result of the non-voice sound, and IoT device information received from a repository associated with the voice command interpreter.

[0016] In one embodiment, the processor may provide a voice command interpreter that determines one or more IoT devices that may be in the vicinity and performs an analysis of the non-voice sound based on the one or more IoT devices that may be in the vicinity.

[0017] In one embodiment, the processor may provide a voice command interpreter that determines one or more IoT devices that may be in the vicinity based on the non-voice sound.

[0018] In one embodiment, the processor may provide a voice command interpreter that determines one or more IoT devices that may be in the vicinity based on the location of the voice command interpreter.

[0019] In one embodiment, the processor may provide a voice command interpreter that determines the location of the voice command interpreter based on the non-voice sound.

[0020] In one embodiment, the processor may provide a voice command interpreter that determines one or more operating IoT devices that may be in the vicinity based on the location of the voice command interpreter and IoT device information received from a storage associated with the voice command interpreter, and performs an analysis of the non-voice sound based on the one or more operating IoT devices that may be in the vicinity.

[0021] In one embodiment, the processor may provide a voice command interpreter that, in performing the analysis of the non-voice sound, determines one or more operating IoT devices that may be in the vicinity and one or more operating IoT devices in the vicinity based on the non-voice sound.

[0022] One embodiment of the present disclosure may provide a method of operation of a voice command interpretation device, comprising the steps of: recognizing a user’s voice command from an input sound; analyzing a non-voice sound from the input sound; and determining at least one target IoT device related to the execution of the voice command based on the analysis result of the non-voice sound.

[0023] In one embodiment, a method of operation of a voice command interpretation device may be provided, further comprising an operation of detecting IoT device control parameters in the voice command and an operation of determining one or more IoT devices that may be in the vicinity, wherein the operation of analyzing the non-voice sound in the input sound includes an operation of analyzing the non-voice sound based on the IoT device control parameters and one or more IoT devices that may be in the vicinity.

[0024] One embodiment of the present disclosure includes a program product stored on a computer-readable recording medium to enable the method according to one embodiment of the present disclosure to be executed on a computer.

[0025] One embodiment of the present disclosure includes a computer-readable recording medium having a program recorded thereon for executing a method according to one embodiment of the present disclosure on a computer. Brief explanation of the drawing

[0026] FIG. 1 illustrates an exemplary environment for interpreting commands based on non-voice sound in an IoT environment according to one embodiment of the present disclosure. FIG. 2 illustrates a detailed block diagram of a virtual assistant device according to one embodiment of the present disclosure. FIG. 3 illustrates a list including target IoT devices on a user interface of an electronic device associated with a user according to one embodiment of the present disclosure. FIGS. 4a and 4b illustrate a flowchart illustrating a method for interpreting commands based on non-voice sound in an IoT environment according to one embodiment of the present disclosure. FIG. 5 illustrates a flowchart showing a method for defining a list of IoT devices having a correlation with non-voice sound according to one embodiment of the present disclosure. FIG. 6 shows a block diagram schematically illustrating the configuration of a voice command interpretation device according to one embodiment of the present disclosure. FIG. 7 illustrates a flowchart schematically showing the flow of an operation method of a voice command interpretation device based on non-voice sound in an IoT environment according to one embodiment of the present disclosure. FIG. 8 illustrates a flowchart specifically showing the flow of an operation method of a voice command interpretation device based on non-voice sound in an IoT environment according to one embodiment of the present disclosure. FIG. 9 illustrates a flowchart specifically showing the flow of an operation method of a voice command interpretation device based on non-voice sound in an IoT environment according to one embodiment of the present disclosure. Specific details for implementing the invention

[0027] To clarify the technical concept of the present disclosure, embodiments of the present disclosure will be described in detail with reference to the attached drawings. In describing the present disclosure, detailed descriptions of related known functions or components will be omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the present disclosure. Components having substantially the same functional configuration among the drawings have been assigned the same reference numerals and symbols as much as possible, even if they are shown in different drawings. For convenience of explanation, devices and methods are described together where necessary. Each operation of the present disclosure does not necessarily need to be performed in the order described and may be performed in parallel, selectively, or individually. Throughout the specification, when a part is described as being "connected" to another part, this may include not only cases where they are "directly connected" but also cases where they are "electrically connected" with other elements interposed between them.

[0028] Embodiments of the present disclosure relate to a method and apparatus for interpreting non-voice-based commands in an IoT environment. In today's IoT environment, IoT devices can be connected and communicate with each other to perform specific tasks. IoT devices are controlled using voice or text commands. In the case of voice commands, when the user's voice command is clear, the IoT device understands the voice command well and operates to perform the desired task. However, in situations where the user's voice command is unclear, incomplete, or mixed with ambient sound, the IoT device cannot perform the task or deliver the output. To overcome these situations, the present disclosure controls the operation of at least one target IoT device by simultaneously detecting and processing the user's voice command and non-voice sound surrounding the user in a received voice command. The present disclosure operates even when the user's voice command is unclear, incomplete, or mixed with ambient sound to deliver the task desired by the user from the IoT device.

[0029] FIG. 1 illustrates an exemplary environment for interpreting commands based on non-voice sound in an IoT environment according to one embodiment of the present disclosure.

[0030] Referring to FIG. 1, the environment (100) may include an IoT device (101-1), an IoT device (101-2), ..., an IoT device (101-N) {collectively referred to as a plurality of IoT devices (101)}, a storage (103), a communication network (105), and a virtual assistant device (107). The plurality of IoT devices (101) may be connected to the virtual assistant device (107) via the communication network (105). In one embodiment, the plurality of IoT devices (101) may include, but are not limited to, any object with an embedded electronic product, electronic device, or electronic circuit, sensor, and internet connection. For example, the plurality of IoT devices (101) may be a refrigerator, a ventilation fan (hood), a speaker, a television, an air conditioner, a vacuum cleaner, a mobile phone, a light bulb, etc. The plurality of IoT devices (101) may be executed on the Universal Plug and Play (UPnP) protocol. Those skilled in the art will understand that any IoT device not explicitly mentioned may be used as an IoT device in this disclosure. Here, all input from a user may be received through a user interface (not shown) of one or more IoT devices or any user device. A plurality of IoT devices (101) may provide real-time output data to a virtual assistant device (107) via a communication network (105), and accordingly, the plurality of IoT devices (101) may receive control signals from the virtual assistant device (107) via the communication network (105). The real-time output data may be the operating state of the plurality of IoT devices (101). The operating state may include the IoT device state of the plurality of IoT devices (101).The communication network (105) may include, but is not limited to, direct interconnection, e-commerce network, P2P network (Peer-to-Peer network), LAN (Local Area Network), WAN (Wide Area Network), wireless network (e.g., using WAP (Wireless Application Protocol)), Internet, Wi-Fi, Bluetooth, etc.

[0031] In one embodiment, the virtual assistant device (107) may receive a voice command from a user and control the operation of at least one target IoT device based on the received voice command. The target IoT device may be an IoT device that the user intends to control via a voice command. In addition to the IoT device that the user intends to control via a voice command, the target IoT device may be an IoT device capable of performing an operation related to the execution of the user's voice command. The voice command may include at least one non-voice sound heard around the user. Here, the voice command may be the sound of the user speaking. The virtual assistant device (107) may include an input / output (I / O) interface (111), a memory (113), and a processor (115). The I / O interface (111) may be configured to receive real-time output data from a plurality of IoT devices (101). Similarly, the I / O interface (111) may be configured to transmit control signals to a plurality of IoT devices (101). The I / O interface (111) can use communication protocols / methods such as Bluetooth, cellular (e.g., CDMA (Code-Division Multiple Access), HSPA+ (High-Speed ​​Packet Access), GSM (Global System for Mobile Communications), LTE (Long-Term Evolution)), and WiMAX (Worldwide interoperability for Microwave access). The virtual assistant device (107) may be part of one or more of the multiple IoT devices (101).

[0032] Voice commands from a user received via an I / O interface (111) can be stored in memory (113). Memory (113) can be communicably coupled to a processor (115) of a virtual assistant device (107). Memory (113) can also store processor commands. This allows the processor (115) to execute commands to control the operation of at least one target IoT device. Memory (113) may include a memory drive, a removable disk drive, etc. The memory drive may include a drum, a magnetic disk drive, an optical magnetic drive, an optical drive, a RAID (Redundant Array of Independent Discs), a solid-state memory device, a solid-state drive, etc.

[0033] The processor (115) may include at least one data processor for controlling the operation of at least one target IoT device. The processor (115) may include special processing units such as an integrated system (bus) controller, a memory management control unit, a floating-point unit, a graphics processing unit, a digital signal processing unit, etc.

[0034] In one embodiment, the virtual assistant device (107) may exchange data with the storage (103) directly or through a communication network (105). The storage (103) may be a database or a cloud server. The storage (103) may be initially populated with or stored during training with data including at least one of sounds related to the operation of a plurality of IoT devices (101), ambient sounds around the user, IoT device control parameters, or associations between each IoT device control parameter and one or more IoT devices. The data may include the operating status of the plurality of IoT devices (101) stored in the storage (103). Here, the operating status of the plurality of IoT devices (101) may be maintained in real time in the storage (103). Sounds related to the operation of the plurality of IoT devices (101) may include, for example, the vibration sound of a washing machine, the sound coming from a ventilation fan, the sound of a refrigerator, etc. Ambient sounds around the user may include, for example, the sound of water coming from a faucet, the sound of a car engine, etc. IoT device control parameters may be predefined keywords related to the control of one or more IoT devices. IoT device control parameters may be predefined words representing the operation of one or more IoT devices. For example, IoT device control parameters may include 'speed', 'speed increase / decrease', 'quiet', 'volume', 'volume increase / decrease', 'mute', 'start / stop', 'brightness', 'bright / dark', 'warm / cool', 'open / close', 'next / previous play', etc. According to an embodiment, IoT device control parameters may include at least one of a command, a component type, or a target device.For example, when a user issues a voice command such as 'decrease fan speed,' the virtual assistant device (107) may identify the command as 'decrease speed' when identifying IoT device control parameters, identify the component type as 'IoT' since the fan is an IoT device, and identify the target device as 'fan.' The association between each IoT device control parameter and one or more IoT devices may include, for example, associating IoT device control parameters such as speed and speed increase or decrease with IoT devices such as fans and / or ceiling fans. The operating state of multiple IoT devices (101) may include the current operating state of multiple IoT devices (101). For example, if the speaker is turned on and the volume is 5, the operating state of the speaker may be 5. The storage (103) may store a list of multiple IoT devices (101). This list may include static IoT devices that are fixed in one place (i.e., fixed in position), such as televisions, refrigerators, and air conditioners, and dynamic IoT devices that are not fixed in any place or are inherently movable (i.e., not fixed in position), such as mobile terminals / telephones and vacuum cleaners.

[0035] The repository (103) can be updated at any time by the virtual assistant device (107), by the user, or by both. Such updates may be related to at least one of sounds related to the operation of multiple IoT devices (101), ambient sounds around the user, IoT device control parameters, associations between each IoT device control parameter and one or more IoT devices, or the operating status of multiple IoT devices (101).

[0036] FIG. 2 illustrates a detailed block diagram of a virtual assistant device according to one embodiment of the present disclosure.

[0037] In addition to the I / O interface (111) and processor (115) described above, the virtual assistant device (107) may include data (200) and one or more units (211) described in detail herein. In one embodiment, the data (200) may be stored in memory (113). The data (200) may include, for example, voice data (201) and other data (203).

[0038] Voice data (201) may be data regarding input sound. The input sound may be sound input through a microphone. The input sound may include a user's voice command and non-voice sound. Voice data (201) may include a user's voice command. Voice data (201) may include at least one non-voice sound around the user. A voice command may include at least one non-voice sound around the user. The non-voice sound around the user may include at least one of ambient sounds occurring around the user other than sounds related to the operation of a plurality of IoT devices (101) around the user and sounds related to the operation of a plurality of IoT devices (101). Here, the ambient sound may include the sound of water falling from a faucet, the sound of a car engine starting or driving, the sound of a pet, the sound of two or more people talking, etc.

[0039] Other data (203) can store data including temporary data and temporary files generated by the units (211) to perform various functions of the virtual assistant device (107).

[0040] In one embodiment, data (200) in memory (113) is processed by one or more units (211) present in memory (113) of the virtual assistant device (107). In one embodiment, one or more units (211) may be implemented as dedicated hardware units. The term unit as used herein may refer to an Application Specific Integrated Circuit (ASIC), electronic circuit, Field-Programmable Gate Array (FPGA), Programmable System-on-Chip (PSoC), combinational logic circuit, and / or other suitable components providing the described functions. In one embodiment, one or more units (211) may be communicably coupled to a processor (115) to perform one or more functions of the virtual assistant device (107). When configured with the functions defined in this disclosure, the units (211) may be novel hardware.

[0041] In one embodiment, one or more units (211) may include, but are not limited to, a receiver unit (213), an automated speech recognition unit (215), a non-speech sound classifier unit (217), a control parameter identifier unit (219), an IoT query service unit (221), a correlation engine unit (223), a control unit (225), a hyper-parameter selection unit (227), and a list unit (229). One or more units (211) may also include other units (231) for performing various other functions of the virtual assistant device (107).

[0042] The receiver unit (213) can receive voice commands from a user through the I / O interface (111). The voice command may include at least one non-voice sound around the user. The user's voice command may be speech. The non-voice sound around the user may include at least one of sounds related to the operation of multiple IoT devices (101) around the user and ambient sounds occurring around the user other than sounds related to the operation of multiple IoT devices (101). For example, if the user says 'decrease speed', the receiver unit (213) can receive 'decrease speed' as a voice command along with ambient sounds coming from a refrigerator and a ventilation fan around the user.

[0043] The automatic speech recognition unit (215) can detect a user's voice command. The automatic speech recognition unit (215) may use any speech recognition technology, such as Automated Speech Recognition (ASR) technology or Speech to Text (STT) technology, to detect the user's voice command. After detection, the automatic speech recognition unit (215) may perform a speech-to-text conversion operation to provide a user command hypothesis. The output, that is, the user command hypothesis of the automatic speech recognition unit (215), may be in text format. In short, the automatic speech recognition unit (215) can process voice commands for user command recognition. For example, if the user says 'speed down,' the receiver unit (213) may receive 'speed down' as a voice command along with ambient sounds of the refrigerator and ventilation fan around the user. The automatic speech recognition unit (215) may detect only the user's voice command, that is, 'speed down.' In the aforementioned example of 'speed reduction,' the user's voice command is not a complete voice command because the voice command does not mention at least one target IoT device.

[0044] The non-voice sound classifier unit (217) can be trained with non-voice data from various IoT devices and ambient sounds around the user other than the non-voice data from various IoT devices for effective detection of IoT device types. The non-voice sound classifier unit (217) can detect at least one non-voice sound around the user from a voice command. The non-voice sound classifier unit (217) can detect non-voice data by applying a sound processing technique that extracts Log Mel features, such as Mel-Frequency Cepstral Coefficients (MFCC), from the voice command, and then using a Convolutional Neural Network (CNN) algorithm on the Log Mel features. The detection method described above is merely an example. Non-voice sound classification is not limited to MFCC-based CNN algorithms. After detection, the unit can classify the non-voice sound to recognize one or more appropriate IoT devices. Additionally, the non-voice sound classifier unit (217) can detect the location where the user's voice command was detected and use the detected location to recognize one or more appropriate IoT devices. In short, the non-voice sound classifier unit (217) can process non-voice sounds to recognize related IoT devices. For example, if a user says "speed down," the receiver unit (213) can receive "speed down" as a voice command along with ambient sounds of the refrigerator and ventilation fan around the user. The non-voice sound classifier unit (217) can detect ambient sounds other than sounds related to the operation of IoT devices. The non-voice sound classifier unit (217) can detect only the ambient sounds of the refrigerator and ventilation fan to recognize the refrigerator and ventilation fan as appropriate IoT devices.

[0045] The automatic voice recognition unit (215) and the non-voice sound classifier unit (217) can process each data of the received voice command in parallel. That is, the received voice command can be supplied simultaneously to the automatic voice recognition unit (215) and the non-voice sound classifier unit (217) for parallel processing.

[0046] The control parameter identifier unit (219) can receive a user command estimate from the automatic speech recognition unit (215) and can identify the presence of one or more IoT device control parameters in the user command estimate. This unit may be part of a natural language processing (NLP) system. Simply put, the control parameter identifier unit (219) can process the user's voice command using NLP techniques such as sentence segmentation, tokenization, some voice tagging, and entity recognition to identify IoT device control parameters. For example, in the user's voice command, namely 'reduce speed', the control parameter identifier unit (219) can identify 'reduce' and 'speed' as control parameters.

[0047] The IoT query service unit (221) can receive information on one or more IoT devices recognized by the non-voice sound classifier unit (217). The IoT query service unit (221) can obtain the operating status of the IoT devices based on the information on one or more recognized IoT devices. The operating status may include the state of the IoT device. The state of the IoT device may mean that the IoT device is turned on or off, and the operating status of the IoT device may mean the mode of operation. For example, a ventilation fan may operate at speed 4 out of fan speeds 1 to 5. The IoT query service unit (221) can obtain the operating status by directly querying the IoT devices or by querying a storage (103) that stores the operating status of all available IoT devices in real time. For example, when the refrigerator and the ventilation fan are recognized as appropriate IoT devices by the non-voice sound classifier unit (217), the IoT query service unit (221) can obtain the operating status of the refrigerator as 'OFF' and the operating status of the ventilation fan as 'ON' and '4' from the storage (103).

[0048] The correlation engine unit (223) can receive IoT device control parameters from the control parameter identifier unit (219) and receive the IoT device operation status from the IoT query service unit (221). The correlation engine unit (223) can receive voice commands from the automatic voice recognition unit (215). The correlation engine unit (223) can receive non-voice sound classification results from the non-voice sound classifier unit (217). The correlation engine unit (223) can identify at least two correlations among voice commands, non-voice sound classification results, the operation status of one or more IoT devices, or IoT device control parameters. The correlation engine unit (223) can identify the correlation between the operation status of one or more IoT devices and the IoT device control parameters. The correlation engine unit (223) can identify the correlation between the non-voice sound classification results, the operation status of one or more IoT devices, and the IoT device control parameters. Subsequently, the correlation engine unit (223) can select at least one target IoT device from one or more IoT devices based on the correlation. For example, if the operating status of the refrigerator is received as 'OFF' and the operating status of the ventilation fan as 'ON' and '4' from the IoT query service unit (221), and the control parameters are received as 'decrease' and 'speed' from the control parameter identifier unit (219), the correlation engine unit (223) can identify that 'speed' is related to the ventilation fan. In this case, since the refrigerator is in the 'OFF' state, the correlation engine unit (223) may not consider this. The correlation engine unit (223) may be part of an NLP system.

[0049] In addition to the correlation engine unit (223), the virtual assistant device (107) may include a multiple context loader unit (not shown) and an intent resolver and response generator unit (not shown).

[0050] The multi-context loader unit may receive at least one target IoT device selected from the correlation engine unit (223) or from a candidate list. The multi-context loader unit may load a concurrent context stack for each selected IoT device for user command matching. The multi-context loader unit may maintain the current operational state of each selected IoT device for accurate user command determination. This unit may be part of an NLP system. For example, when an IoT device such as a television, air conditioner, or vacuum cleaner is selected by the correlation engine unit (223), the multi-context loader unit may load the context of the selected IoT device from the NLP system.

[0051] The intent resolver and response generator unit can interpret the intent of a user command for one or more of the most appropriate IoT devices. This unit can generate a response to perform the desired task by specifying the action(s) of the IoT device(s). This unit may be part of an NLP system. For example, consider a situation where a user issues the command, "Hi Bixby, be quiet. I need to answer a call." In addition to the television, air conditioner, and vacuum cleaner identified as operating in the vicinity where the user command was issued, there may be other IoT devices such as light bulbs and door handles. In this situation, the intent resolver and response generator can interpret the user command request for "be quiet" as referring only to devices capable of making loud noises, such as the TV, air conditioner, and vacuum cleaner.

[0052] The control unit (225) can perform an operation to control the operation of at least one selected target IoT device received from the correlation engine unit (223). For example, if the user command is 'device mute' and the correlation engine provides the respective operation states 'ON', 'FAN ON', 'ON' with at least one target IoT device such as a TV, air conditioner, and vacuum cleaner, the control unit (225) can mute the TV, turn off the fan of the air conditioner, and turn off the vacuum cleaner.

[0053] The hyperparameter selection unit (227) can limit the list of IoT devices by determining the location and / or position of the virtual assistant device (107) depending on whether the virtual assistant device (107) is fixed or movable. For example, there are two categories of IoT devices. One category consists of dynamic / mobile IoT devices within the home, such as mobile terminals and vacuum cleaners, and the other category consists of fixed / static IoT devices within the home, such as speakers, televisions, and refrigerators. When the virtual assistant device (107) that is part of the IoT devices is fixed, the hyperparameter selection unit (227) can determine the location of the IoT device and obtain IoT devices available in the corresponding room / scene within the home. When the virtual assistant device (107) that is part of the IoT devices is movable, the hyperparameter selection unit (227) can determine the location of the IoT device. Once the location of the IoT device is known, the hyperparameter selection unit (227) can obtain IoT devices available in that room / scene within the home. When the location of the IoT device is unknown, the hyperparameter selection unit (227) may obtain all available IoT devices in the home. To create or limit one or more suitable IoT devices, a list of IoT devices may be provided from the hyperparameter selection unit (227) to the non-voice sound classifier unit (217).

[0054] The list unit (229) can receive the output of the correlation engine unit (223) in a situation where the correlation engine unit (223) selects two or more target IoT devices. The list unit (229) can provide the user with a list containing target IoT devices for user selection through an electronic device associated with the user, as illustrated in FIG. 3.

[0055] FIGS. 4a and 4b illustrate a flowchart illustrating a method for interpreting commands based on non-voice sound in an IoT environment according to one embodiment of the present disclosure.

[0056] Referring to FIGS. 4a and 4b, a command interpretation method (400) according to one embodiment of the present disclosure includes one or more operations for command interpretation based on non-voice sound in an IoT environment and can be described in the general context of computer-executable instructions. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, procedures, units, and functions that perform specific functions or implement specific abstract data types.

[0057] The order in which the command interpretation method (400) according to one embodiment of the present disclosure is described is not intended to be interpreted as a limitation, and any number of method operations may be combined in any order to implement the method. Additionally, individual operations may be omitted from the method without departing from the scope of the subject matter described herein. Furthermore, the method may be implemented in any suitable hardware, software, firmware, or combination thereof.

[0058] In operation S401, a voice command from a user can be received by a receiver unit (213) through an I / O interface (111). The voice command may include at least one IoT device control parameter.

[0059] In operation S403, the automatic voice recognition unit (215) and the non-voice sound classifier unit (217) can detect the user's voice command and at least one non-voice sound around the user in the received voice command. Analysis of at least one non-voice sound around the user can be triggered upon detection of the presence of at least one IoT device control parameter within the user's voice command.

[0060] The automatic voice recognition unit (215) can process a user's voice command to identify IoT device control parameters. Here, each IoT device control parameter may be associated with one or more IoT devices. Additionally, the IoT device control parameter may include at least one of a command, a component type, or a target IoT device.

[0061] The processing of the user's voice command and the analysis of at least one non-voice sound around the user can be performed in parallel.

[0062] In operation S405, the non-voice sound classifier unit (217) can identify one or more IoT devices around the user based on at least one of the detected non-voice sound and the location where the user's voice command was detected. The non-voice sound around the user may include at least one of sounds related to the operation of the IoT device around the user and ambient sounds around the user other than sounds related to the operation of the IoT device. At least one non-voice sound may be detected around the user or around the virtual assistant device (107).

[0063] In operation S407, the correlation engine unit (223) can select at least one target IoT device from the identified IoT devices based on the correlation between the current operating state of each of the identified IoT devices and the identified IoT device control parameters. The operating state of the one or more IoT devices can be received from the IoT query service unit (221), and the IoT device control parameters can be received from the control parameter identifier unit (219).

[0064] In operation S409, the control unit (225) can execute received IoT device control parameters for at least one selected target IoT device. The control unit (225) can control the operation of at least one selected target IoT device to achieve non-voice-based command interpretation.

[0065] Operations S413 to S417 can be performed when two or more target IoT devices are selected by the correlation engine unit (223).

[0066] In operation S413, the list unit (229) may provide the user with a list of at least one target IoT device through an electronic device associated with the user. The list may be provided when two or more target IoT devices are selected by the correlation engine unit (223) from among one or more identified IoT devices.

[0067] In operation S415, the receiver unit (213) may receive input from a user via an electronic device to control the operation of at least one target IoT device. The input from the user may include the selection of at least one target IoT device. The electronic device may include a mobile terminal, a speaker, a television, a refrigerator, a washing machine, a fan, or other IoT-enabled device.

[0068] In operation S417, the control unit (225) can control the operation of at least one target IoT device based on user input.

[0069] FIG. 5 illustrates a flowchart showing a method for defining a list of IoT devices having a correlation with non-voice sound according to one embodiment of the present disclosure.

[0070] As illustrated in FIG. 5, a list limitation method (500) according to one embodiment of the present disclosure comprises one or more operations for limiting a list of IoT devices having a correlation with non-voice sound and may be described in the general context of computer-executable instructions. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, procedures, units, and functions that perform specific functions or implement specific abstract data types.

[0071] The order in which the list limiting method (500) according to one embodiment of the present disclosure is described is not intended to be interpreted as a limitation, and any number of method operations may be combined in any order to implement the method. Additionally, individual operations may be omitted from the method without departing from the scope of the subject matter described herein. Furthermore, the method may be implemented in any suitable hardware, software, firmware, or combination thereof.

[0072] In operation S501, the hyperparameter selection unit (227) can determine the location where the user's voice command was received to identify one or more IoT devices present nearby. IoT devices can be classified into two sets: (i) static IoT devices that are fixed in place or do not move, such as televisions, washing machines, and refrigerators, and (ii) dynamic IoT devices that are not fixed in place or move, such as mobile terminals or vacuum cleaners in the house. The location of the static IoT devices can be stored in the storage (103). This operation can be performed when the virtual assistant device (107) is fixed, that is, when it cannot move. For example, if the virtual assistant device (107) is a fixed device such as a speaker, television, or refrigerator, the virtual assistant device (107) can know the location of the IoT device from the storage (103) and obtain the available IoT device in that room / site.

[0073] In operation S503, the hyperparameter selection unit (227) can determine the location of the virtual assistant device (107) in an IoT environment. This operation can be performed when the virtual assistant device (107) is dynamic, that is, when it is movable. For example, if the virtual assistant device (107) is a dynamic device such as a mobile terminal, a vacuum cleaner, etc., the virtual assistant device (107) can determine its location in that room / site. The IoT query service unit (221) can assist the hyperparameter selection unit (227) in determining the location. In one embodiment, the location may be determined based on a specific identified IoT device, a combination of known locations and IoT devices, or an IoT site. For example, while processing a user command, the non-voice sound classifier unit (217) can identify that the television is turned on with the help of the IoT query service unit (221). This confirms that the virtual assistant device (107) is potentially in the same location as the television. As another example, while processing a user command, the non-voice sound classifier unit (217) can identify the presence of two or more IoT devices belonging to the same IoT site with the help of the IoT query service unit (221). For example, the sounds of a refrigerator and a ventilation fan can be identified in the same IoT site. This potentially implies that the virtual assistant device (107) is in the same IoT site. In this case, the site is the 'kitchen IoT site'.

[0074] In operation S505, the hyperparameter selection unit (227) can determine one or more IoT devices that exist near the location of the virtual assistant device (107). This operation is performed when the location of the virtual assistant device (107) is known. For example, if the location of the virtual assistant device (107) is known, the virtual assistant device (107) can obtain a list of available IoT devices in that room / site. In this situation, the IoT query service unit (221) can provide the virtual assistant device (107) with a list of available IoT devices in the same location or IoT site, along with the status of each IoT device. One way to achieve this is for the user to create an IoT room / site and add one or more stationary IoT devices to that room / site. If the virtual assistant device (107) is a mobile device, the room / site of the virtual assistant device (107) can also be identified as soon as the sound of the nearest stationary IoT device is detected.

[0075] In operation S507, the hyperparameter selection unit (227) can determine all IoT devices connected to the virtual assistant device (107) when the location of the virtual assistant device (107) is unknown. Information regarding all IoT devices connected to the virtual assistant device (107) can be obtained from a storage (103) having a list of stored multiple IoT devices (static and dynamic IoT devices). For example, when the location of the virtual assistant device (107) is unknown, the virtual assistant device (107) can obtain all available static and dynamic IoT devices within the house.

[0076] Information from operation S501 to operation S507 is provided from the hyperparameter selection unit (227) to the non-voice sound classifier unit (217) to create or limit one or more suitable IoT devices.

[0077] Below, several examples based on FIGS. 4a, 4b, and 5 are disclosed.

[0078] Example 1: Consider a scenario in which a user is listening to a recipe on an electronic device in a kitchen. The user cannot hear the recipe due to ambient or surrounding sounds from the refrigerator and the fan. To hear the recipe clearly, the user gives the command, "Hi Bixby, I can't hear it, so slow down." The virtual assistant device (107) of the present disclosure identifies non-auditory data, namely the sounds of the refrigerator and the fan. The virtual assistant device (107) further identifies that the speed reduction command is about the fan rather than the refrigerator, and reduces the speed of the fan. Now the user hears the recipe without the sound of the fan. This example covers a situation where one device, namely the fan, is controlled based on several unique non-auditory data, namely the ambient sounds of the refrigerator and the fan.

[0079] Example 2: Consider a scenario in which a user receives a phone call on their mobile phone in the living room. A vacuum cleaner, a TV, and an air conditioner are operating in the living room. To make a call without any interruptions, the user gives the command, "Hi Bixby, be quiet. I'm going to answer the call." The virtual assistant device (107) of the present disclosure identifies non-voice data, namely the sounds of the vacuum cleaner, the television, and the air conditioner. The virtual assistant device (107) further identifies that the command to be quiet refers to all devices in the living room, namely the vacuum cleaner, the television, and the air conditioner, excluding the ringing phone itself. In this situation, the virtual assistant device (107) stops the vacuum cleaner, mutes the television, and turns off the fan of the air conditioner. Now the user answers the call without worrying about the sounds of the devices in the living room. This example covers a situation in which multiple devices, such as a vacuum cleaner, a television, and an air conditioner, are controlled based on various unique non-voice data, namely ambient sounds of the vacuum cleaner, the TV, and the air conditioner.

[0080] Example 3: Consider a scenario in which a user is washing dishes at a kitchen sink. There are lights in the kitchen near the sink and the dishwasher. To illuminate the kitchen, the user gives the command, "Hi Bixby, turn on these lights." The virtual assistant device (107) of the present disclosure identifies non-verbal data, namely the sounds of running water and the dishwasher. Based on the non-verbal data, the virtual assistant device (107) identifies the location as the kitchen and then identifies that the devices in the kitchen are lights and the dishwasher. The virtual assistant device (107) identifies that the command to turn on these lights is related to the lights near the kitchen sink, rather than lights located elsewhere. In this situation, the virtual assistant device (107) turns on the lights in the kitchen. This example covers a situation in which one or more devices, namely lights, are controlled based on the location of non-verbal sounds, namely the sounds of running water and the dishwasher. Additionally, this example covers a situation in which devices that are not operating or making sounds are controlled.

[0081] Example 4: Consider a scenario in which a user is driving a car with the sunroof open. To make a phone call, the user gives the command, "Hi Bixby, call Jacob." The virtual assistant device (107) of the present disclosure identifies non-voice data, namely noise from outside the car. Based on the non-voice data, the virtual assistant device (107) identifies that the car's sunroof is open. The virtual assistant device (107) closes the sunroof and makes a call to contact Jacob. This example covers a situation in which one or more devices, namely the car's sunroof, are controlled during a conversation with the virtual assistant device (107). Additionally, this example covers a situation in which an IoT device capable of performing actions related to the execution of the user's voice command is controlled, rather than the direct target IoT device of the user's voice command. Additionally, this example covers a situation in which an IoT device capable of performing actions related to the execution of the user's voice command is controlled when the user's voice command is complete, that is, when the target IoT device of the voice command is specified.

[0082] Example 5: Consider another scenario in which a user is listening to a recipe on an electronic device in a kitchen. The user cannot hear the recipe due to surrounding or ambient sounds from the refrigerator and the fan. To hear the recipe clearly, the user gives the command, "Hi Bixby, I can't hear it, so slow down." The virtual assistant device (107) of the present disclosure identifies non-voice data, namely the sounds of the refrigerator and the fan. The virtual assistant device (107) further identifies that the speed reduction command is about the fan rather than the refrigerator and asks the user, "Would you like to slow down the fan?" Upon receiving the user's voice confirmation of "Yes," the virtual assistant device (107) slows down the fan. The user continues to listen to the recipe without the sound of the fan. This example covers a situation where one device, namely the fan, is controlled after the user's confirmation (without receiving confirmation through the user interface of the electronic device).

[0083] Example 6: Consider a scenario in which a user is trying to put a baby to sleep in a bedroom. A vacuum cleaner, a TV, and an air conditioner are operating in the bedroom. To reduce the noise in the bedroom, the user issues a command: "Hi Bixby, quiet down. My baby is going to sleep." The virtual assistant device (107) of the present disclosure identifies non-voice data, namely the sounds of the vacuum cleaner, the television, and the air conditioner. The virtual assistant device (107) also identifies that the command to quiet down refers to all devices in the bedroom, namely the vacuum cleaner, the television, and the air conditioner. In this situation, the virtual assistant device (107) provides a list including the vacuum cleaner, the television, and the air conditioner to the user interface of the user's mobile device, along with actions to quiet the devices, namely muting the television, slowing down the air conditioner, and stopping the vacuum cleaner (a similar example is shown in FIG. 3). Since the user wants to use the air conditioner, they issue a voice command saying "stop vacuum cleaner and television" or select the vacuum cleaner and television via a touch command on the user interface of the user's mobile device. Upon receiving such confirmation, the virtual assistant device (107) stops the vacuum cleaner and the television while the air conditioner is still operating. This example covers a situation in which multiple devices, such as a vacuum cleaner and a TV, are controlled after receiving confirmation from the user via the user interface or voice command of an electronic device.

[0084] Example 7: Consider a scenario in which a user in the living room attempts to listen to a message received on their mobile phone. In the living room, a vacuum cleaner, a TV, and an air conditioner are in operation, and the user's mobile phone is present. To listen to the message on the mobile phone without interruption, the user issues the command, "Hi Bixby, be quiet. I want to listen to the message." The virtual assistant device (107) of the present disclosure identifies non-voice data, namely the sounds of the vacuum cleaner, the television, and the air conditioner. The virtual assistant device (107) further identifies that the command to be quiet applies to all devices in the living room excluding the mobile phone, namely the vacuum cleaner, the television, and the air conditioner, because the message will be read on the mobile phone. In this situation, the virtual assistant device (107) provides a list including the vacuum cleaner, the television, and the air conditioner to the user interface of the user's mobile device, along with actions to quiet the devices, namely muting the television, slowing down the air conditioner, and stopping the vacuum cleaner (a similar example is illustrated in FIG. 3). Because the user wants to use the air conditioner and vacuum cleaner, they issue a voice command saying "Mute TV" or select the television via a touch command on the user interface of the user's mobile device. Upon receiving this confirmation, the virtual assistant device (107) stops the television while the air conditioner and vacuum cleaner are still operating and plays the message received from the mobile phone. This example resolves a conflict of user command intent, that is, prevents the mobile phone from being muted along with the television, thereby covering a situation where one or more devices, namely the air conditioner, vacuum cleaner, and mobile phone, are controlled.

[0085] The examples mentioned above may include incomplete voice commands that do not mention at least one target IoT device, or voice commands from a user that does not have a target IoT device.

[0086] One embodiment of the present disclosure may provide a method for interpreting commands in an Internet of Things (IoT) environment, comprising: receiving a voice command from a user by a virtual assistant device, wherein the voice command includes at least one IoT device control parameter; detecting at least one non-voice sound by the virtual assistant device while receiving the voice command; identifying one or more IoT devices around the user based on the detected at least one non-voice sound by the virtual assistant device; selecting at least one target IoT device from the identified one or more IoT devices based on the correlation between the current operating state of each of the identified one or more IoT devices and the identified IoT device control parameter by the virtual assistant device; and executing the received IoT device control parameter for the selected at least one target IoT device by the virtual assistant device.

[0087] One embodiment of the present disclosure may provide a virtual assistant device for resolving commands in an Internet of Things (IoT) environment, comprising a processor and a memory communicably coupled to the processor, wherein the processor receives a voice command from a user by executing instructions stored in the memory, the voice command includes at least one IoT device control parameter, detects at least one non-voice sound while receiving the voice command, identifies one or more IoT devices around the user based on the detected at least one non-voice sound, selects at least one target IoT device from the identified one or more IoT devices based on the correlation between the current operating state of each of the identified one or more IoT devices and the identified IoT device control parameter, and executes the received IoT device control parameter for the selected at least one target IoT device.

[0088] In one embodiment, the processing of the voice command from the user and the analysis of the at least one non-voice sound can be performed in parallel.

[0089] In one embodiment, the analysis of the at least one non-voice sound may be triggered by the detection of the presence of at least one IoT device control parameter in a user voice command.

[0090] In one embodiment, the non-voice sound around the user may include at least one of a sound related to the operation of an IoT device around the user and ambient sounds around the user other than the sound related to the operation of the IoT device.

[0091] In one embodiment, sounds related to the operation of the IoT device and ambient sounds around the user may be stored in a storage associated with the virtual assistant device.

[0092] In one embodiment, each IoT device control parameter is associated with one or more IoT devices, and the association between each IoT device control parameter and one or more IoT devices may be stored in a storage associated with a virtual assistant device.

[0093] In one embodiment, when the virtual assistant device is fixed, the virtual assistant device can determine the location where the user's voice command was received in order to identify one or more IoT devices existing around the location where the user's voice command was received.

[0094] In one embodiment, if the virtual assistant device is movable, the location of the virtual assistant device in the IoT environment can be determined by the virtual assistant device.

[0095] In one embodiment, when the location of the virtual assistant device is known, the virtual assistant device can detect one or more IoT devices existing near the location of the virtual assistant device.

[0096] In one embodiment, if the location of the virtual assistant device is unknown, the virtual assistant device can determine all IoT devices connected to the virtual assistant device.

[0097] In one embodiment, the operating states of one or more identified IoT devices are received from a storage associated with the virtual assistant device, wherein the operating states of one or more identified IoT devices can be maintained in real time in the storage associated with the virtual assistant device.

[0098] In one embodiment, the voice command of the user may include an incomplete voice command from the user that does not mention the at least one target IoT device.

[0099] In one embodiment, the operation of controlling the operation of at least one selected target IoT device may include, when more than one target IoT device is selected from the identified one or more IoT devices, the operation of providing the user with a list of the one or more target IoT devices through an electronic device associated with the user by the virtual assistant device, the operation of receiving input from the user through the electronic device to control the operation of the one or more target IoT devices by the virtual assistant device—the input from the user includes the selection of the one or more target IoT devices—and the operation of the one or more target IoT devices by the virtual assistant device based on the user input.

[0100] In one embodiment, the at least one non-voice sound may be detected from the surroundings of the user or the surroundings of the virtual assistant device.

[0101] In one embodiment, the operation of executing the identified IoT device control parameters for the selected at least one target IoT device may include the operation of controlling the operation of the selected at least one target IoT device by the virtual assistant device to achieve command interpretation based on non-voice sound.

[0102] In one embodiment, the IoT device control parameter may include at least one of a command, a component type, or a target IoT device.

[0103] FIG. 6 is a block diagram schematically illustrating the configuration of a voice command interpreter according to one embodiment of the present disclosure. Referring to FIG. 6, a voice command interpreter (600) according to one embodiment of the present disclosure may include a processor (601) and a memory (603) that stores one or more instructions that can be executed by the processor (601). The processor (601) may be a single processor or a plurality of processors. The memory (603) may be a single memory or a plurality of memories. The operation of the voice command interpreter (600) performed by the processor (601) executing one or more instructions stored in the memory (603) will be described in detail below with reference to FIG. 7, etc. The voice command interpreter (600) may be a virtual assistant device (107) or a part thereof.

[0104] FIG. 7 is a flowchart schematically illustrating the flow of an operation method of a voice command interpretation device based on non-voice sound in an IoT environment according to an embodiment of the present disclosure. Referring to FIG. 7, the processor (601) of the voice command interpretation device (600) recognizes a user's voice command from an input sound in operation S701 and analyzes a non-voice sound from the input sound in operation S702. Here, the non-voice sound may be a sound that occurs simultaneously with or within a short period of time with the voice command. The non-voice sound may be a sound that occurs within a certain period of time from the start time and / or end time of the voice command.

[0105] The operation S701 for recognizing voice commands and the operation S702 for analyzing non-voice sounds may be performed in parallel or simultaneously. Here, the fact that the two operations are performed simultaneously does not necessarily mean that the start times of the two operations are the same, but may mean that the time of one operation and the time of the other operation may overlap.

[0106] The processor (601) can determine at least one target IoT device involved in the execution of a voice command based on the analysis result of a non-voice sound in operation S703. Here, the target IoT device is determined based on the analysis result of the non-voice sound, but it is not necessarily a device that is currently operating or making a sound. The processor (601) can determine the target IoT device based on the correlation between the analysis result of the non-voice sound and the voice command.

[0107] The processor (601) can execute a voice command targeting a determined target IoT device. The processor (601) can control the determined target IoT device to execute the voice command. Based on the results of non-voice sound analysis, the processor (601) can determine the action to be performed by the target IoT device in relation to the execution of the voice command and control the target IoT device to perform the action. The processor (601) can determine the action to be performed by the target IoT device based on the correlation between the results of non-voice sound analysis and the voice command.

[0108] The processor (601) may perform analysis of non-voice sounds when the recognized voice command is incomplete. The processor (601) may perform analysis of non-voice sounds when the target IoT device of the voice command is not fully specified in the recognized voice command. The processor (601) may perform analysis of non-voice sounds based on the fact that the recognized voice command does not include at least one target IoT device required for the execution of the voice command. The processor (601) may perform analysis of non-voice sounds only when the recognized voice command does not include at least one target IoT device required for the execution of the voice command.

[0109] The processor (601) may perform analysis of non-voice sounds when the recognized voice command is complete. The processor (601) may also perform analysis of non-voice sounds when the recognized voice command includes all target IoT devices necessary for the execution of the voice command. In this case, the processor (601) may cause the target IoT device determined based on the analysis results of the non-voice sounds to perform an action related to the execution of the voice command.

[0110] The processor (601) can determine multiple target IoT devices related to the execution of a voice command based on the results of non-voice sound analysis. Based on the results of non-voice sound analysis, the processor (601) can determine the actions to be performed by each of the determined multiple target IoT devices in relation to the execution of the voice command, and control each of the multiple target IoT devices to perform the corresponding actions. The processor (601) can control the multiple target IoT devices to perform the actions to be performed in relation to the execution of the voice command simultaneously.

[0111] In performing an analysis of non-voice sounds, the processor (601) may compare the non-voice sound with the sounds of predefined IoT devices. By comparing the non-voice sound with the sounds of predefined IoT devices, the processor (601) may determine one or more IoT devices currently operating in the vicinity. The processor (601) may determine at least one of the IoT devices currently operating in the vicinity as the target IoT device. Based on the IoT devices currently operating in the vicinity, the processor (601) may determine a device that is not an IoT device currently operating in the vicinity as the target IoT device. The processor (601) may determine the action to be performed by the target IoT device based on the IoT devices currently operating in the vicinity. An IoT device currently operating in the vicinity may be an IoT device that is emitting sound.

[0112] In performing an analysis of non-voice sounds, the processor (601) can detect ambient sounds other than sounds related to the operation of the IoT device from the non-voice sounds. The processor (601) can determine the target IoT device based on the detected ambient sounds. The processor (601) can determine the operation to be performed by the target IoT device based on the detected ambient sounds.

[0113] FIG. 8 is a flowchart specifically illustrating the flow of an operation method of a voice command interpretation device based on non-voice sound in an IoT environment according to an embodiment of the present disclosure. Referring to FIG. 8, a processor (601) can detect IoT device control parameters from a voice command (voice command estimation) in operation S801. The processor (601) can perform analysis of non-voice sound based on the detected IoT device control parameters. Here, the IoT device control parameters may be directly included in the voice command or may be extracted by analyzing the voice command. For example, when a user's voice command is "Hi Bixby, be quiet," the word "quiet" included in this voice command may be detected as a control parameter, and when a user's voice command is "Hi Bixby, it's too loud," the control parameter "quiet" may be extracted by analyzing it.

[0114] The processor (601) can perform analysis of non-voice sounds based on the detection of IoT device control parameters in a voice command. The processor (601) can perform analysis of non-voice sounds only when IoT device control parameters are detected in a voice command. The processor (601) can perform analysis of non-voice sounds when the target IoT device for the action related to the IoT device control parameters in the voice command is not fully specified. The processor (601) can perform analysis of non-voice sounds when the recognized voice command does not include at least one target IoT device required for the execution of the action related to the detected IoT device control parameters. The processor (601) may also perform analysis of non-voice sounds when IoT device control parameters are not detected in a voice command.

[0115] The processor (601) can perform analysis of non-voice sounds based on one or more IoT devices that are pre-matched to the detected IoT device control parameters. For example, when comparing non-voice sounds with the sounds of pre-defined IoT devices, the processor (601) can significantly reduce computational resources and improve processing speed by comparing only the pre-defined sounds of the IoT devices that are pre-matched to the detected IoT device control parameters with the non-voice sounds.

[0116] The processor (601) can select at least one filter bank to be used for the analysis of non-voice sounds based on detected IoT device control parameters. This can reduce computational resources and improve processing speed. Here, the filter bank may be a Mel filter bank.

[0117] The processor (601) can determine the action to be performed by the target IoT device and / or the target IoT device based on at least two correlations among the recognized voice command, the detected IoT device control parameter, the non-voice sound analysis result, or the IoT device information received from the storage (103) associated with the voice command interpretation device (600). The storage (103) may be an IoT server or a database associated with the IoT server. The processor (601) can determine the action to be performed by the target IoT device and / or the target IoT device based on the correlation between the detected IoT device control parameter and the non-voice sound analysis result. The processor (601) can determine the action to be performed by the target IoT device and / or the target IoT device based on the correlation between the detected IoT device control parameter, the non-voice sound analysis result, and the IoT device information received from the storage (103).

[0118] The processor (601) can determine one or more IoT devices that may be in the vicinity and perform an analysis of non-voice sounds based on the one or more IoT devices that may be in the vicinity. Here, the vicinity may be the vicinity of the voice command interpreter (600) or the vicinity of the user issuing the voice command. The processor (601) can reduce computational resources and improve processing speed by considering only the IoT devices that may be in the vicinity, rather than all IoT devices within the IoT service. For example, when comparing non-voice sounds with predefined sounds of IoT devices, the processor (601) can significantly reduce computational resources and improve processing speed by comparing only the predefined sounds of one or more IoT devices that may be in the vicinity with the non-voice sounds. The processor (601) may also determine one or more IoT devices that may be in the vicinity based on the non-voice sounds. The one or more IoT devices that may be in the vicinity may be devices that are currently operating or making sounds. The one or more IoT devices that may be in the vicinity may not be devices that are currently operating or making sounds.

[0119] In operation S803, the processor (601) can determine one or more IoT devices that may be in the vicinity based on location information. Here, the location information may be information regarding the location of the voice command interpreter (600) and / or the location of the user issuing the voice command.

[0120] The processor (601) can determine one or more IoT devices that may be in the vicinity based on the location of the voice command interpreter (600). The location of the voice command interpreter (600) may include the location of a microphone that receives input sound.

[0121] The processor (601) can determine one or more IoT devices that may be in the vicinity based on the location of the user issuing the voice command. The user's location can be determined based on the location of the voice command interpreter (600). The user's location can be determined based on the direction from which the user's voice command is heard using beamforming technology, etc. The processor (601) can determine the user's location based on non-voice sounds heard from the direction from which the user's voice command is heard. The processor (601) can determine one or more IoT devices that may be in the vicinity based on the direction from which the non-voice sounds are heard. Although the following description focuses on the location of the voice command interpreter (600), the description is also applicable to the location of the user issuing the voice command.

[0122] If the voice command interpreter (600) is a mobile device, the voice command interpreter (600) can determine its location using various indoor / outdoor positioning technologies or IoT services. The voice command interpreter (600) can also determine its location based on non-voice sounds. If the voice command interpreter (600) is a stationary device, its location can be specified by the user. For example, the user can specify stationary IoT devices (e.g., TV, air conditioner, refrigerator, virtual assistant device) installed in each location (e.g., living room, room, kitchen) using the UI of an IoT service. Even if the voice command interpreter (600) is a stationary device, the voice command interpreter (600) can determine its location based on positioning technologies, IoT services, non-voice sounds, etc.

[0123] If the location of the voice command interpreter (600) is known, the processor (601) may determine one or more IoT devices that may be around the voice command interpreter (600) as fixed IoT devices. The processor (601) may determine one or more IoT devices that may be around all mobile IoT devices within the IoT service. The processor (601) may determine one or more IoT devices that may be around mobile IoT devices within the IoT service whose location is unknown and which is not currently around the voice command interpreter (600). The processor (601) may determine one or more IoT devices that may be around all IoT devices whose location is unknown within the IoT service. Cases where the location of the voice command interpreter (600) is known may include cases where the voice command interpreter (600) is a fixed device. Cases where the location of the voice command interpreter (600) is known may include cases where the voice command interpreter (600) is a mobile device and its location is determined.

[0124] If the location of the voice command interpreter (600) is unknown, the processor (601) may determine all IoT devices within the IoT service as one or more IoT devices that may be nearby. The case where the location of the voice command interpreter (600) is unknown may include the case where the voice command interpreter (600) is a mobile device and its location is not determined.

[0125] The processor (601) can determine one or more IoT devices that may be in the vicinity based on IoT device information (e.g., location of the IoT device, operating status of the IoT device) received from the storage (103). The processor (601) can determine one or more currently operating IoT devices that may be in the vicinity based on IoT device information received from the storage (103). The processor (601) can determine one or more operating IoT devices that may be in the vicinity based on the location of the voice command interpreter (600) and IoT device information received from the storage (103).

[0126] The processor (601) can perform an analysis of non-voice sounds based on one or more operating IoT devices that may be in the vicinity. The processor (601) can determine one or more operating IoT devices in the vicinity, that is, IoT devices that are actually currently operating in the vicinity, based on the non-voice sounds and one or more operating IoT devices that may be in the vicinity.

[0127] The non-voice sound analysis result may include one or more IoT devices that may be in the vicinity, one or more operating IoT devices that may be in the vicinity, or one or more operating IoT devices in the vicinity. The processor (601) may determine a target IoT device based on one or more IoT devices that may be in the vicinity, one or more operating IoT devices that may be in the vicinity, or one or more operating IoT devices in the vicinity. The processor (601) may determine a target IoT device and / or an action to be performed by the target IoT device based on a correlation between one or more IoT devices that may be in the vicinity, one or more operating IoT devices that may be in the vicinity, or one or more operating IoT devices in the vicinity, and a recognized voice command, a detected IoT device control parameter, a non-voice sound analysis result, or IoT device information received from a storage (103).

[0128] The processor (601) can determine at least one of one or more IoT devices that may be in the vicinity as the target IoT device. The processor (601) can determine at least one of one or more operating IoT devices that may be in the vicinity as the target IoT device. The processor (601) can determine at least one of one or more operating IoT devices in the vicinity as the target IoT device.

[0129] The processor (601) can receive the operating status of one or more IoT devices that may be in the vicinity from the storage (103). The processor (601) can determine the target IoT device and / or the operation to be performed by the target IoT device based on the operating status of at least one IoT device that may be in the vicinity received from the storage (103) and the correlation of the detected IoT device control parameters.

[0130] In one embodiment, the processor (601) may determine one or more operating IoT devices that may be in the vicinity based on the location of the voice command interpreter (600) and IoT device information received from the storage (103), determine one or more operating IoT devices that may be in the vicinity based on non-voice sound, and determine at least one target IoT device related to the execution of a voice command based on one or more operating IoT devices in the vicinity. The processor (601) may determine the target IoT device and / or the action to be performed by the target IoT device based on the correlation between one or more operating IoT devices in the vicinity and a recognized voice command or a detected IoT device control parameter. The processor (601) may determine the target IoT device and / or the action to be performed by the target IoT device based on the correlation between one or more operating IoT devices in the vicinity, a recognized voice command or a detected IoT device control parameter, and IoT device information received from the storage (103). The determined target IoT device may be one of one or more operating IoT devices in the vicinity. The determined target IoT device is not one of one or more active IoT devices in the vicinity, but may be a device involved in executing the recognized voice command.

[0131] In one embodiment, the processor (601) determines the location of a voice command interpreter (600) based on a non-voice sound, and can determine at least one target IoT device related to the execution of a voice command based on the determined location of the voice command interpreter (600).

[0132] FIG. 9 is a flowchart specifically illustrating the flow of an operation method of a voice command interpretation device based on non-voice sound in an IoT environment according to one embodiment of the present disclosure.

[0133] Some advantages of the present disclosure are as follows.

[0134] One embodiment of the present disclosure provides an efficient command interpretation method based on non-voice sounds in an IoT environment in addition to a user's voice command.

[0135] One embodiment of the present disclosure provides a method for understanding an incomplete voice command from a user.

[0136] One embodiment of the present disclosure utilizes the operating states of one or more IoT devices identified by a non-voice sound classification unit, which makes the selection of one or more IoT devices more accurate.

[0137] One embodiment of the present disclosure reduces the user's cognitive overload by allowing the user to perform tasks without having to remember complete and accurate commands.

[0138] The proposed method will improve the end-user experience and increase user trust in artificial intelligence (AI) assistants.

[0139] Embodiments of the present disclosure may be implemented as computer-executable code on a computer-readable recording medium. Computer-readable recording media include all recording media such as magnetic media, optical media, ROM, and RAM. Computer-readable recording media may be provided in the form of a non-transitory storage medium. Here, "non-transitory storage medium" simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily on the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is stored temporarily.

[0140] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed by being stored on a computer-readable recording medium, or in an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a computer-readable recording medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0141] Additionally, in this specification, "unit" may be a hardware component, such as a processor or circuit, and / or a software component executed by a hardware component, such as a processor.

[0142] Additionally, in this specification, "comprising at least one of a, b, or c" may mean "comprising only a, comprising only b, comprising only c, comprising a and b, comprising b and c, comprising a and c, or comprising all of a, b, and c."

[0143] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0144] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0145] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0146] The present disclosure has been described in detail with reference to the preferred embodiments illustrated in the drawings. These embodiments are merely illustrative and are not intended to limit the disclosure; they should be considered in an illustrative rather than a limiting sense. Those skilled in the art will understand that these embodiments can be easily modified into other specific forms without altering the technical spirit or essential features of the present disclosure. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form. Although specific terms have been used in this specification, they are used only for the purpose of explaining the concept of the present disclosure and are not intended to limit the meaning or the scope of the present disclosure as defined in the claims.

[0147] The true technical scope of protection of this disclosure shall be determined not by the foregoing description but by the technical concept of the appended claims, and all modifications or variations derived from the meaning and scope of the claims and the concept of equivalents shall be interpreted as being included within the scope of this disclosure. Equivalents shall be understood to include not only currently known equivalents but also equivalents to be developed in the future, that is, all components disclosed to perform the same function regardless of structure.

Claims

Claim 1 A voice command interpretation device comprising: a processor; and a memory for storing instructions, wherein the processor receives an input sound, recognizes a user's voice command included in the input sound, classifies a non-voice sound included in the input sound to obtain a non-voice sound classification result related to the operation of an IoT device, identifies a correlation between the voice command, the non-voice sound classification result, and the operation state or IoT device control parameter of one or more IoT devices, and determines at least one target IoT device related to the execution of the voice command based on the identified correlation. Claim 2 In claim 1, the processor is a voice command interpreter that performs classification of the non-voice sound when the voice command does not include at least one target IoT device required for the execution of the voice command. Claim 3 In claim 1, the processor is a voice command interpreter that determines an action to be performed by at least one target IoT device in relation to the execution of the voice command based on the non-voice sound classification result. Claim 4 A voice command interpretation device according to claim 1, wherein the processor determines a plurality of target IoT devices related to the execution of the voice command based on the non-voice sound classification result, determines the operations to be performed by each of the plurality of target IoT devices in relation to the execution of the voice command based on the non-voice sound classification result, and controls the plurality of target IoT devices to simultaneously perform the operations to be performed in relation to the execution of the voice command. Claim 5 In claim 1, the processor is a voice command interpreter that, in performing classification of the non-voice sound, compares the non-voice sound with the operation sounds of predefined IoT devices. Claim 6 In claim 1, the processor is a voice command interpretation device that, in performing classification of the non-voice sound, detects ambient sound other than sound related to the operation of the IoT device from the non-voice sound. Claim 7 In claim 1, the processor detects IoT device control parameters in the voice command and performs classification of the non-voice sound based on the IoT device control parameters. Claim 8 In claim 7, the processor is a voice command interpreter that performs classification of the non-voice sound when an IoT device control parameter is detected in the voice command. Claim 9 In claim 7, the processor is a voice command interpreter that performs classification of the non-voice sound based on one or more IoT devices pre-matched to the IoT device control parameters. Claim 10 In claim 7, the processor determines the target IoT device based on at least two correlations among the voice command, the IoT device control parameter, the non-voice sound classification result, or IoT device information received from a repository associated with the voice command interpreter. Claim 11 In claim 7, the processor determines the target IoT device based on the correlation between the IoT device control parameters, the non-voice sound classification result, and IoT device information received from a repository associated with the voice command interpreter. Claim 12 A voice command interpreter according to claim 1, wherein the processor determines one or more IoT devices that may be in the vicinity and performs classification of the non-voice sound based on the one or more IoT devices that may be in the vicinity. Claim 13 In paragraph 12, the processor is a voice command interpreter that determines one or more IoT devices that may be in the vicinity based on the non-voice sound. Claim 14 In paragraph 12, the processor determines one or more IoT devices that may be in the vicinity based on the location of the voice command interpreter. Claim 15 In paragraph 14, the processor determines the location of the voice command interpreter based on the non-voice sound, the voice command interpreter. Claim 16 A voice command interpreter according to claim 1, wherein the processor determines one or more operating IoT devices that may be in the vicinity based on the location of the voice command interpreter and IoT device information received from a storage associated with the voice command interpreter, and performs classification of the non-voice sound based on the one or more operating IoT devices that may be in the vicinity. Claim 17 In claim 16, the processor determines one or more operating IoT devices that may be in the vicinity and one or more operating IoT devices in the vicinity based on the non-voice sound in performing the classification of the non-voice sound. Claim 18 A method of operation of a voice command interpretation device comprising: receiving an input sound; recognizing a user's voice command included in the input sound; classifying non-voice sounds included in the input sound to obtain a non-voice sound classification result related to the operation of an IoT device; and identifying a correlation between the voice command, the non-voice sound classification result, and the operation state or IoT device control parameter of one or more IoT devices, and determining at least one target IoT device related to the execution of the voice command based on the identified correlation. Claim 19 A method of operation of a voice command interpretation device according to claim 18, wherein the method further comprises: an operation of detecting an IoT device control parameter in the voice command; and an operation of determining one or more IoT devices that may be in the vicinity, and wherein the operation of classifying the non-voice sound in the input sound comprises an operation of classifying the non-voice sound based on the IoT device control parameter and one or more IoT devices that may be in the vicinity. Claim 20 A computer-readable recording medium having a program recorded thereon for performing the method of paragraph 18 or 19 on a computer.

Images