System and method for remotely controlling electronic device

Abstract

The present disclosure relates to a system and a method for remotely controlling an electronic device by using bio-signals and / or speech of a user, and can provide the method for remotely controlling an electronic device, comprising the steps of: filtering a speech signal from a signal received from a speech signal sensor; filtering a bio-signal from a signal received from a bio-signal sensor; recognizing a first user gesture corresponding to the bio-signal and a first instruction corresponding to the speech signal; mapping, to the first user gesture, a first control signal corresponding to the first instruction; and transmitting the first control signal to an external device corresponding to the first control signal if a second user gesture corresponding to a subsequent bio-signal corresponds to the first user gesture.

Description

Electronic device remote control system and method

[0001] The present disclosure relates to a system and method for remotely controlling an electronic device using at least one of a user's bio-signal and voice.

[0002] Biometrics is a technology that uses automated devices to measure physical or behavioral characteristics to authenticate or identify individuals. Currently, biometric technologies utilizing fingerprints, faces, palm prints, hand geometry, retinas, irises, voice, and signatures are being developed and deployed. Biometric technologies utilizing blood vessels and DNA are also being developed. Accordingly, interest in biometrics is rapidly increasing, and standardization of the various technologies required for these biometric technologies is also progressing rapidly.

[0003] Among the above biometric technologies, the electromyography (EMG) signal has different frequencies and waveforms depending on the user's physical characteristics and also depending on the user's gestures (e.g., the degree of muscle contraction and relaxation), so the EMG signal can be detected and used for remote control of electronic devices.

[0004] However, if users are required to memorize various gestures assigned to control various functions of electronic devices, they may invest significant time and effort in controlling the device's functions. Furthermore, if users were allowed to register various gestures for controlling various functions of electronic devices, they could register gestures whose EMG signals are inaccurately distinguished, potentially leading to malfunctions in the EMG-based control of the electronic device.

[0005] The present disclosure is proposed to solve the above-mentioned problems, and its purpose is to provide a system and method for remotely controlling an electronic device, which uses a user's voice for labeling purposes when learning a gesture that generates a bio-signal such as an electromyography signal to control an electronic device, and enables remote control of the electronic device using at least one of the learned gestures and / or voices.

[0006] In order to achieve the above object, the present disclosure may provide a hub device including a voice signal filter for filtering a voice signal from a signal received from a voice signal sensor, a biosignal filter for filtering a biosignal from a signal received from a biosignal sensor, and a control command recognizer for recognizing a first user gesture corresponding to the biosignal and a first command corresponding to the voice signal, mapping a first control signal corresponding to the first command to the first user gesture, and controlling the transmission of the first control signal to an external device corresponding to the first control signal when a second user gesture corresponding to a subsequent biosignal corresponds to the first user gesture.

[0007] The above control command recognizer can determine whether a corresponding control signal is mapped to the first user gesture when the first user gesture is a pre-registered gesture, and if the corresponding control signal is not mapped to the first user gesture, control to map the first control signal to the first user gesture.

[0008] The above control command recognizer can control to transmit the second control signal to an external device corresponding to the second control signal when a second control signal is mapped to the first user gesture.

[0009] The above control command recognizer can be controlled to tune a gesture judgment model for judging whether the first user gesture is a pre-registered gesture using the first user gesture.

[0010] The above control command recognizer can be controlled to label a first user gesture using a first command.

[0011] The above control command recognizer can be controlled to learn the first user gesture input multiple times when the first user gesture labeled with the first command is input multiple times, and map the first control signal to the first user gesture based on the learning result.

[0012] The above control command recognizer can be controlled to register the first user gesture if the first user gesture is an unregistered gesture.

[0013] The above control command recognizer can store a first user gesture, learn the first user gesture inputted a plurality of times when the same or similar first user gestures are inputted a plurality of times or more, and control the first user gesture to be newly registered based on the learning result.

[0014] The above control command recognizer can use few-shot learning to learn the first user gesture inputted multiple times.

[0015] The above control command recognizer can be controlled to map a first control signal to the newly registered first user gesture when the newly registered first user gesture is recognized again.

[0016] The above control command recognizer can determine whether a trigger word existed before recognition of the first command, and if the trigger word existed, control to transmit a first control signal to the external device corresponding to the first control signal.

[0017] The first control signal includes an operation control signal and information on an electronic device to be controlled, and the control command recognizer can control to transmit the operation control signal to the external device corresponding to the information on the electronic device to be controlled.

[0018] The above control command recognizer can determine whether the operation control signal is valid for the electronic device to be controlled, and if it is determined to be valid, control to transmit the operation control signal to the external device corresponding to the information of the electronic device to be controlled.

[0019] The above hub device may further include a communication unit for short-range communication with a wearable device including the voice signal sensor and the biosignal sensor.

[0020] The above biosignal may be an electromyography signal.

[0021] In addition, to achieve the above object, the present disclosure may provide a method for remotely controlling an electronic device, including a step of filtering a voice signal from a signal received from a voice signal sensor, a step of filtering a biosignal from a signal received from a biosignal sensor, a step of recognizing a first user gesture corresponding to the biosignal and a first command corresponding to the voice signal, a step of mapping a first control signal corresponding to the first command to the first user gesture, and a step of transmitting the first control signal to an external device corresponding to the first control signal when a second user gesture corresponding to a subsequent biosignal corresponds to the first user gesture.

[0022] The effects of the electronic device remote control system and method according to the present disclosure are as follows.

[0023] According to one aspect of the present disclosure, there is an advantage in that a user's voice is used for labeling purposes in learning a gesture that generates a bio-signal such as an electromyogram signal to control an electronic device, and the electronic device can be remotely controlled using at least one of the learned gesture and / or voice.

[0024] FIG. 1 is a conceptual diagram of an electronic device remote control system according to one aspect of the present disclosure.

[0025] FIG. 2 is a block diagram of a wearable device (100) according to one aspect of the present disclosure.

[0026] FIG. 3 is a block diagram of an electronic device remote control system for explaining voice signals and bio-signals processed according to one aspect of the present disclosure.

[0027] Figure 4 is a conceptual diagram for explaining the use of learning in the control command recognizer of Figure 3.

[0028] Fig. 5 is a modified example of the electronic device remote control system of Fig. 3.

[0029] Figure 6 is a flowchart of control command recognition according to one aspect of the present disclosure.

[0030] Figure 7 is a detailed flowchart of step S613 of Figure 6.

[0031] FIG. 8 illustrates an example of the correlation between an electromyography signal and a voice signal according to one aspect of the present disclosure.

[0032] Figure 9 is a detailed flowchart of step S614 of Figure 6.

[0033] FIG. 10 illustrates an example of a gesture and a corresponding control target electronic device and motion control command according to one aspect of the present disclosure.

[0034] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0035] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0036] The terminology used in this application is solely for the purpose of describing specific embodiments and is not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0037] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0038] The suffixes "module" and "part" used for components in the following description are given or used interchangeably only for the convenience of writing specifications, and do not have distinct meanings or roles in themselves.

[0039] Referring to FIG. 1, an electronic device remote control system according to one aspect of the present disclosure will be described. FIG. 1 is a conceptual diagram of an electronic device remote control system according to one aspect of the present disclosure.

[0040] As illustrated in FIG. 1, the electronic device remote control system may include a wearable device (100), a hub device (200), a management server (300), and at least one electronic device (400-1, 400-2, ..., 400-N). If there is no need to distinguish between the at least one electronic device, the reference numeral for the electronic device may be 400.

[0041] The wearable device (100) can transmit a biosignal (e.g., an electromyography signal) sensed from the user to the hub device (200).

[0042] The above wearable device (100) can transmit a voice signal sensed from a user to the hub device (200).

[0043] The user's voice signal does not necessarily have to be sensed through the wearable device (100) and transmitted to the hub device (200). The hub device (200) may be equipped with a microphone for sensing the user's voice signal and may directly sense the user's voice.

[0044] The above hub device (200) can generate a control command for controlling one of the target electronic devices among the at least one electronic device (400-1, 400-2, ..., 400-N) using the bio-signal and the voice signal. That is, the hub device (200) can operate as a remote control relay device for the at least one electronic device.

[0045] When the above hub device (200) is installed in a home, it may be referred to as a home hub device. The hub device may also function as an access point to provide a home local area network (e.g., a Wi-Fi network).

[0046] The above-mentioned generated control command can be transmitted to the target electronic device (400) via the management server (300). Of course, the above-mentioned generated control command can also be transmitted to the target electronic device (400) without going through the management server (300).

[0047] Hereinafter, with reference to FIG. 2, the wearable device (100) of FIG. 1 will be examined in more detail. FIG. 2 is a block diagram of a wearable device (100) according to one aspect of the present disclosure.

[0048] The above wearable device (100) may include a wireless communication unit (110), a voice sensor (122), a user input unit (123), a biosignal sensor (143), an output unit (150), a memory (170), and a control unit (180).

[0049] The above wireless communication unit (110) may include at least one of a mobile communication module, a wireless Internet module, a short-range communication module, and a location information module.

[0050] The mobile communication module transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network built according to 5G, such as technology standards or communication methods for mobile communication (e.g., GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.).

[0051] The above wireless signal may include various forms of data such as voice call signals, video call signals, or text / multimedia message transmission and reception.

[0052] A wireless Internet module refers to a module for wireless Internet access, and can be built into or externally mounted on a wearable device (100). The wireless Internet module is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

[0053] Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), and LTE-A (Long Term Evolution-Advanced), and the wireless Internet module transmits and receives data according to at least one wireless Internet technology, including Internet technologies not listed above.

[0054] From the perspective that wireless Internet access through WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A, 5G, etc. is achieved through a mobile communication network, the wireless Internet module that performs wireless Internet access through the mobile communication network can be understood as a type of mobile communication module.

[0055] The short-range communication module is for short-range communication, and can support short-range communication using at least one of Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies. The short-range communication module can support wireless communication between the wearable device (100) and another external device or external server through a short-range wireless communication network (Wireless Area Networks). The short-range wireless communication network may be a short-range wireless personal area network (Wireless Personal Area Networks).

[0056] A location information module is a module for obtaining the location (or current location) of a wearable device. Representative examples thereof include a GPS (Global Positioning System) module or a WiFi (Wireless Fidelity) module. For example, if a wearable device utilizes a GPS module, the location of the wearable device can be obtained using signals transmitted from GPS satellites. As another example, if a wearable device utilizes a Wi-Fi module, the location of the wearable device can be obtained based on information from a wireless access point (AP) that transmits or receives wireless signals with the Wi-Fi module. If necessary, the location information module may perform the function of any of the other modules of the wireless communication unit (110) to obtain data regarding the location of the wearable device as a substitute or in addition. The location information module is a module used to obtain the location (or current location) of the wearable device, and is not limited to a module that directly calculates or obtains the location of the wearable device.

[0057] The voice sensor (122) processes external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application program) being performed on the wearable device (100). Meanwhile, the voice sensor (122) can implement various noise removal algorithms to remove noise generated during the process of receiving external acoustic signals.

[0058] The user input unit (123) is for receiving information from a user, and when information is input through the user input unit (123), the control unit (180) can control the operation of the wearable device (100) to correspond to the input information. The user input unit (123) may include a mechanical input means (or a mechanical key, for example, a button located on the front, back, or side of the mobile terminal (100), a dome switch, a jog wheel, a jog switch, etc.) and a touch input means. As an example, the touch input means may be composed of a virtual key, a soft key, or a visual key displayed on a touch screen through software processing, or a touch key placed on a part other than the touch screen. The virtual key or visual key may have various forms and be displayed on the touch screen, and may be composed of, for example, graphics, text, icons, videos, or a combination thereof.

[0059] The biosignal sensor (143) can sense various biosignals of the user. More specifically, the biosignal sensor (143) can sense various biosignals of the user wearing the wearable device (100). Here, the biosignal may refer to various biological signals that are generated by the user's physical activity and can be sensed through the user's body. For example, the biosignal sensor (143) may be worn or attached to the user's body and sense at least one of the user's muscle contraction degree, electromyography, skin conductance, brain waves, pulse, electrocardiogram, respiration, blood pressure, blood flow, body temperature, and heart rate as a biosignal. That is, the biosignal sensor (143) may include a muscle tension measurement sensor, an electromyography sensor, a muscle strength sensor, a skin conductance sensor, an brain wave sensor, an electrocardiogram sensor, a respiration sensor, a blood pressure sensor, a blood flow sensor, a body temperature measurement sensor, a heart rate sensor, and the like. The sensors described above may be included in a wearable device as individual components or integrated into at least one component. The biosignal sensor (143) may sense at least one biosignal described above and transmit the sensing result to the control unit (180).

[0060] The output unit (150) is for generating output related to visual, auditory, or tactile sensations, and may include at least one of a display unit, an audio output unit, a haptic module, and an optical output unit. The display unit may be formed as a layer structure with a touch sensor or formed as an integral part, thereby implementing a touch screen. This touch screen may function as a user input unit (123) that provides an input interface between the wearable device (100) and the user, and at the same time, may provide an output interface between the wearable device (100) and the user.

[0061] The memory (170) stores data that supports various functions of the wearable device (100). The memory (170) can store a plurality of application programs (or applications) running on the wearable device (100), data for the operation of the wearable device (100), and commands. At least some of these application programs can be downloaded from an external server via wireless communication. In addition, at least some of these application programs can exist on the wearable device (100) from the time of shipment for the basic functions of the wearable device (100) (e.g., call receiving and sending functions, message receiving and sending functions). Meanwhile, the application programs can be stored in the memory (170), installed on the wearable device (100), and driven by the control unit (180) to perform the operations (or functions) of the wearable device (100).

[0062] In addition to the operations related to the above application program, the control unit (180) typically controls the overall operation of the wearable device (100). The control unit (180) processes signals, data, information, etc. input or output through the components discussed above, or operates an application program stored in the memory (170), thereby providing or processing appropriate information or functions to the user.

[0063] The control unit (180) can sense a user's voice signal through the voice sensor (122), sense the user's bio-signal, and transmit the sensed voice signal and the sensed bio-signal to the hub device (200) through the wireless communication unit (110). The voice signal and the bio-signal can be transmitted to the hub device (200) through, for example, short-range communication. For example, the short-range communication can include Wi-Fi communication, Bluetooth communication, etc.

[0064] The processing of the voice signal and the biosignal transmitted to the hub device (200) will be further described with reference to FIG. 3. FIG. 3 is a block diagram of an electronic device remote control system for explaining the voice signal and biosignal processed according to one aspect of the present disclosure.

[0065] The hub device (200) may include a biosignal filter (243), a voice signal filter (222), and a control command recognizer (280). Of course, the hub device (200) may further include a communication unit (not shown) for receiving the voice signal and the biosignal from the wearable device (100). The communication unit may be connected to the wearable device (100) via short-range communication, such as Wi-Fi communication or Bluetooth communication.

[0066] Each of the components of the above hub device (200) may be composed of hardware or software. Additionally, two or more of the components of the above hub device (200) may be composed of one component.

[0067] The above voice signal filter (222) can filter the voice signal from the communication signal received from the wearable device (100).

[0068] The above biosignal filter (243) can filter the biosignal from the communication signal received from the wearable device (100).

[0069] The above filtered voice signal and the above biosignal can be transmitted to the control command recognizer (280).

[0070] The above control command recognizer (280) can analyze the biosignal and recognize a gesture corresponding to the biosignal if the biosignal is a valid biosignal.

[0071] If the recognized gesture is a registered gesture and a control command is mapped to the registered gesture, the control command recognizer (280) can transmit the matched control command to the management server (300) so that the matched control command can be transmitted to an electronic device (400) corresponding to the mapped control command.

[0072] The above control command recognizer (280) can analyze the voice signal and recognize a command corresponding to the voice signal.

[0073] The above control command recognizer (280) can determine whether a trigger word exists in the voice signal immediately before the command is received, if there is a control command mapped to the recognized command, and if the trigger word exists, can transmit the matched control command to the management server (300) so that the matched control command can be transmitted to the electronic device (400) corresponding to the mapped control command.

[0074] If the recognized gesture is a registered gesture but no control command is mapped to the registered gesture, the control command recognizer (280) can learn the gesture using the recognized command so that a control command intended by the user can be mapped to the registered gesture. That is, the control command recognizer (280) includes a learning processor (not shown) for learning an artificial intelligence neural network, and can utilize the recognized command as label information for learning a mapping between the registered gesture and the intended control command.

[0075] If the above recognized gesture is not a registered gesture, the control command recognizer (280) can learn to register the above recognized gesture.

[0076] Below, we will look more closely at artificial intelligence.

[0077] Artificial intelligence (AI) is the study of artificial intelligence or the methodologies for creating it, while machine learning (ML) defines various problems in the field of AI and studies the methodologies for solving them. Machine learning is also defined as an algorithm that improves performance on a task through consistent experience.

[0078] An artificial neural network (ANN) is a model used in machine learning. It can refer to a model with problem-solving capabilities, comprised of artificial neurons (nodes) formed by the connection of synapses. An ANN can be defined by the connection patterns between neurons in different layers, the learning process that updates model parameters, and the activation function that generates output values.

[0079] An artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer contains one or more neurons, and the artificial neural network may include synapses connecting neurons. In an artificial neural network, each neuron can output a function value of an activation function based on input signals, weights, and biases received through the synapses.

[0080] Model parameters are parameters determined through learning, including synaptic connection weights and neuron biases. Hyperparameters are parameters that must be set before learning in machine learning algorithms, including the learning rate, number of iterations, mini-batch size, and initialization function.

[0081] The goal of artificial neural network training can be seen as determining model parameters that minimize a loss function. The loss function can be used as an indicator for determining optimal model parameters during the artificial neural network training process.

[0082] Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning depending on the learning method.

[0083] Supervised learning refers to a method for training an artificial neural network when given labels for the training data. The labels can refer to the correct answer (or output value) that the artificial neural network must infer when the training data is input to the artificial neural network. Unsupervised learning can refer to a method for training an artificial neural network when the training data is not given labels. Reinforcement learning can refer to a learning method in which an agent defined within a given environment is trained to select actions or action sequences that maximize the cumulative reward in each state.

[0084] Machine learning implemented with a deep neural network (DNN) containing multiple hidden layers among artificial neural networks is also called deep learning, and deep learning is a subset of machine learning. Hereinafter, the term "machine learning" is used to encompass deep learning.

[0085] Object detection models using machine learning include the single-stage YOLO (You Only Look Once) model and the two-stage Faster R-CNN (Regions with Convolution Neural Networks) model.

[0086] The YOLO (You Only Look Once) model is a model that can predict objects and their locations within an image by looking at the image only once.

[0087] The YOLO (You Only Look Once) model divides the original image into grids of equal size. For each grid, it predicts the number of bounding boxes in a predefined shape centered around the grid center, and calculates a confidence level based on this prediction.

[0088] Afterwards, whether the image contains an object or is just a background is included, and locations with high object confidence are selected so that the object category can be identified.

[0089] The Faster R-CNN (Regions with Convolution Neural Networks) model is a model that can detect objects faster than the RCNN model and the Fast RCNN model.

[0090] This article specifically explains the Faster R-CNN (Regions with Convolution Neural Networks) model.

[0091] First, feature maps are extracted from the image using a Convolution Neural Network (CNN) model. Based on the extracted feature maps, multiple regions of interest (RoIs) are extracted. RoI pooling is performed for each region of interest.

[0092] RoI pooling is a process of setting a grid to a predetermined size of H x W for the feature map onto which the region of interest is projected, extracting the largest value for each cell included in each grid, and extracting a feature map with a size of H x W.

[0093] A feature vector is extracted from a feature map having a size of H x W, and object identification information can be obtained from the feature vector.

[0094] Meanwhile, in the present disclosure, few-shot learning can be utilized to train an artificial intelligence neural network using a limited amount of labeled data.

[0095] The utilization of the above learning in the present disclosure will be further described with reference to FIG. 4. FIG. 4 is a conceptual diagram for explaining the utilization of learning in the control command recognizer of FIG. 3.

[0096] During learning about the mapping between gestures and control commands, the electronic device (400) can be controlled based on the commands within the voice signal, as illustrated in (4-1) of FIG. 4. At this time, the commands can be utilized as label information during learning about the mapping between the gestures and the control commands. Therefore, according to the present disclosure, separate data labeling work is not required, making artificial intelligence learning easier.

[0097] When learning about the mapping between the above gesture and the above control command is completed, the electronic device (400) may be controlled based on the command in the voice signal, as shown in (4-2) of FIG. 4, or the electronic device (400) may also be controlled by a user gesture according to the biosignal.

[0098] A modification of the electronic device remote control system of Fig. 3 will be further described with reference to Fig. 5. Fig. 5 is a modified example of the electronic device remote control system of Fig. 3.

[0099] In FIG. 3, it is exemplified that the voice signal is sensed by the voice sensor (122) provided in the wearable device (100) and transmitted to the hub device (200).

[0100] However, the present disclosure is not limited thereto.

[0101] As shown in (5-1) of FIG. 5, since a voice sensor (122) is provided within the hub device (200), the hub device (200) can directly sense the voice signal on its own.

[0102] Alternatively, as illustrated in (5-2) of FIG. 5, the electronic device remote control system may have a voice sensor (122) separate from the wearable device (100), and the voice signal may be sensed by the separate voice sensor (122) and transmitted to the hub device (200).

[0103] Hereinafter, with reference to FIG. 6, the operation of the control command recognizer (280) will be examined in more detail. FIG. 6 is a flowchart of control command recognition according to one aspect of the present disclosure.

[0104] In the following description, it will be assumed that the biosignal is an electromyography (EMG) signal and the biosignal sensor is an EMG sensor. However, it should be understood that other biosignals besides the EMG signal can be used if the user's gestures can be distinguished.

[0105] The above control command recognizer (280) can receive the user's voice signal sensed through the voice sensor (122) in real time [S601].

[0106] The above control command recognizer (280) can perform voice recognition in real time from the received voice signal [S602]. Each recognized voice can be stored in a voice information DB (271) together with information on the time point at which the voice signal was sensed ((t-2), (t-1), (t), ...). The voice information DB (271) may be provided within the hub device (200) or may be provided in another external device or server.

[0107] The above control command recognizer (280) can detect a command (e.g., "Turn on the air conditioner") from the recognized voice [S603]. It will be assumed that the voice signal from which the command is detected is sensed at time t.

[0108] The above control command recognizer (280) can determine whether there is a control command (or control signal) (e.g., an air conditioner turn-on control command or control signal) pre-mapped to the command [S604]. The control command may include an operation control command (or operation control signal) (e.g., a power turn-on command) and information on an electronic device to be controlled (e.g., an air conditioner).

[0109] If there is no control command pre-mapped to the above command, the process may return to step S601.

[0110] However, if there is a control command pre-mapped to the above command, the control command recognizer (280) can determine whether there was a trigger word (e.g., “high LG”) within a predetermined past time interval from the time point t [S605].

[0111] If the above-mentioned trigger word did not exist within the above-mentioned predetermined past time interval, the process may return to the above-mentioned step S601.

[0112] However, if the trigger word existed at time t-1 within the predetermined past time interval, the control command recognizer (280) can check the operation control command of the control command and the information on the electronic device to be controlled [S606]. That is, the relationship between the operation control command and the information on the electronic device to be controlled can be confirmed. This is to check whether the operation control command is valid for the electronic device to be controlled. For example, the operation control command "changing broadcast channels" may be valid for a TV, but not for an air conditioner or a washing machine.

[0113] If the above-mentioned operation control command is valid for the electronic device to be controlled, the control command recognizer (280) can transmit the control command to the management server (300) [S607]. The management server (300) can transmit the operation control command of the control command to the electronic device to be controlled (400).

[0114] If the above-mentioned motion control command is not valid for the electronic device to be controlled, the control command recognizer (280) may ignore the control command.

[0115] The above step S606 has been described as being performed in the control command recognizer (280) of the hub device (200). However, the present disclosure is not limited thereto. The above step S606 may also be performed in the management server (300).

[0116] The above control command recognizer (280) can receive the user's electromyography signal sensed through the electromyography sensor (143) in real time [S608].

[0117] The above control command recognizer (280) can determine whether the EMG signal is valid [S609]. The validity of the EMG signal can be determined based on whether a meaningful user gesture can be recognized from the EMG signal based on at least one of the magnitude and waveform of the EMG signal.

[0118] If the above EMG signal is invalid, the process may return to step S608.

[0119] If the above EMG signal is valid, the control command recognizer (280) can perform gesture recognition in real time from the received EMG signal [S610]. Each recognized gesture can be stored in the gesture information DB (272) together with the time point information ((t-2), (t-1), (t), ...) at which the corresponding EMG signal was sensed. The gesture information DB (272) may be provided in the hub device (200) or may be provided in another external device or server. In addition, each recognized gesture can be mapped to a voice signal, voice, and / or command sensed, recognized, or detected at the time point at which the corresponding EMG signal was sensed. The mapped voice signal, voice, and / or command can be utilized as label information for artificial intelligence learning. This will be described later.

[0120] The above control command recognizer (280) can determine whether the recognized gesture is a pre-registered gesture by referring to the pre-registered gesture DB (273) [S611]. Here, the pre-registered gestures may refer to gestures that are clearly distinguished from each other and are pre-registered at the time of device release. The pre-registered gesture DB (273) is for storing the pre-registered gestures and newly registered gestures to be described later. The pre-registered gesture DB (273) may be provided within the hub device (200) or may be provided in another external device or server.

[0121] If the recognized gesture is the pre-registered gesture, the control command recognizer (280) can use the recognized gesture determined to be registered to tune or fine-tune a gesture judgment model (not shown) that determines which recognized gesture corresponds to a pre-registered gesture [S615]. Even if the same gesture is made for each user, the sensed electromyography signal may differ within a certain range. The tuning or fine-tuning may mean a task of correcting the value of the electromyography signal corresponding to the pre-registered gesture to better suit the physical characteristics of each user. Of course, the step S615 may be omitted.

[0122] The above control command recognizer (280) can determine whether there is a control command (or control signal) (e.g., an air conditioner turn-on control command) pre-mapped to the recognized gesture [S612]. The control command may include a motion control command (or motion control signal) (e.g., a power turn-on command) and information on an electronic device to be controlled (e.g., an air conditioner).

[0123] If there is a control command pre-mapped to the recognized gesture, the control command recognizer (280) can check the operation control command and the information of the electronic device to be controlled of the control command, and if the operation control command is valid for the electronic device to be controlled, the control command can be transmitted to the management server (300), as described above with respect to steps S606 and S607.

[0124] If there is no control command pre-mapped to the recognized gesture, the control command recognizer (280) may proceed with learning to map a control command corresponding to the mapped voice signal, voice, and / or command to the recognized gesture [S613]. The step S613 is described again with reference to FIG. 7.

[0125] Meanwhile, if the recognized gesture is not the pre-registered gesture as a result of the determination in step S611, the control command recognizer (280) accumulates information about the recognized gesture, and if it is determined that a gesture identical to the recognized gesture or similar within a certain range has been sensed a predetermined number of times in the past, gesture learning is performed using the accumulated information about the gestures sensed a predetermined number of times in the past, so that the recognized gesture can be newly registered [S614]. Few-shot learning can be utilized for the gesture learning. The newly registered gesture can be mapped to correspond to the corresponding control command through step S613. Step 614 is described again with reference to FIG. 9.

[0126] Hereinafter, step S612 will be examined in more detail with reference to FIG. 7. FIG. 7 is a detailed flowchart of step S613 of FIG. 6.

[0127] If the judgment result of the step S612 above shows that there is no control command pre-mapped to the recognized gesture, the control command recognizer (280) can label the recognized gesture with a command corresponding to the voice signal sensed at the sensing time of the electromyography signal of the recognized gesture [S71]. That is, the command can be used as label information for the gesture. The gesture labeled with the command can be cumulatively stored in the pre-registered gesture DB (273).

[0128] The above control command recognizer (280) can learn a plurality of gestures labeled with the cumulatively stored commands in order to map the corresponding control commands to the recognized gestures [S73].

[0129] If the above artificial intelligence learning is successful, the control command recognizer (280) can map the corresponding control command to the recognized gesture and store it in the gesture-control command mapping DB (274) [S74, S75]. The gesture-control command mapping DB (274) can be utilized in the determination in step S612. The gesture-control command mapping DB (274) may be provided within the hub device (200) or may be provided in another external device or server.

[0130] The above description describes how to correlate the EMG signal and the voice signal based on the sensed point in time. However, the correlation between the EMG signal and the voice signal can also be performed in other ways. This will be further described with reference to FIG. 8. FIG. 8 illustrates an example of the correlation between the EMG signal and the voice signal according to one aspect of the present disclosure.

[0131] The above control command recognizer (280) can identify a voice signal section corresponding to a command and identify an electromyography signal section corresponding to the voice signal section [S81]. That is, the voice signal section can be used as a labeling section to determine an electromyography signal section that is the target of labeling of the command.

[0132] Next, the control command recognizer (280) can split the electromyography signal corresponding to the electromyography signal section [S82].

[0133] Next, the control command recognizer (280) can label the command to the gesture corresponding to the split electromyography signal [S71].

[0134] Hereinafter, step S614 will be examined in more detail with reference to FIG. 9. FIG. 9 is a detailed flowchart of step S614 of FIG. 6.

[0135] If the recognized gesture is not a pre-registered gesture as a result of the determination in step S611, the control command recognizer (280) can cumulatively store information about the recognized gesture in an unregistered gesture DB (275) [S91]. The unregistered gesture DB (274) may be provided within the hub device (200) or may be provided in another external device or server.

[0136] The above control command recognizer (280) can determine whether a predetermined number (n) or more of identical or similar gestures are accumulated in the unregistered gesture DB (275) [S92].

[0137] If the judgment result shows that a predetermined number (n) or more of identical or similar gestures have accumulated, the control command recognizer (280) can learn the identical or similar gestures to define them as a new gesture [S93]. Few-shot learning can be utilized for the gesture learning.

[0138] If the above artificial intelligence learning is successful, the control command recognizer (280) can store the newly defined gesture in the pre-registered gesture DB (273) [S95]. That is, the newly defined gesture can be managed as a newly registered gesture. The newly registered gesture can be mapped to correspond to the corresponding control command through step S613.

[0139] Hereinafter, examples of gestures, corresponding control target electronic devices, and motion control commands will be described with reference to FIG. 10. FIG. 10 illustrates examples of gestures, corresponding control target electronic devices, and motion control commands according to one aspect of the present disclosure.

[0140] As illustrated in Fig. 10, each gesture can be mapped to a corresponding control target electronic device and motion control command.

[0141] The present invention described above can be implemented as computer-readable code on a medium in which a program is recorded. The computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of the computer-readable medium include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc., and also includes media implemented in the form of a carrier wave (e.g., transmission via the Internet). In addition, the computer may include a control unit (180) of a wearable device and a control command recognizer (280) of a hub device.

[0142] Accordingly, the above detailed description should not be construed as limiting in all respects, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included within the scope of the present invention.

Claims

1. A voice signal filter for filtering a voice signal from a signal received from a voice signal sensor; A biosignal filter for filtering biosignals from signals received from a biosignal sensor; and Recognize a first user gesture corresponding to the above biosignal and a first command corresponding to the above voice signal, Map the first control signal corresponding to the first command to the first user gesture, A hub device including a control command recognizer that controls to transmit a first control signal to an external device corresponding to the first control signal when a second user gesture corresponding to a subsequent bio-signal corresponds to the first user gesture.

2. In the first paragraph, the control command recognizer If the first user gesture is a pre-registered gesture, determine whether the corresponding control signal is mapped to the first user gesture, A hub device characterized in that it controls to map a first control signal to a first user gesture if the corresponding control signal is not pre-mapped to the first user gesture.

3. In the second paragraph, the control command recognizer, A hub device characterized in that, when a second control signal is mapped to a first user gesture, the second control signal is controlled to be transmitted to an external device corresponding to the second control signal.

4. In the second paragraph, the control command recognizer, A hub device characterized by controlling tuning of a gesture judgment model for judging whether the first user gesture is a pre-registered gesture by using the first user gesture.

5. In the second paragraph, the control command recognizer, A hub device characterized by controlling labeling of a first user gesture using a first command.

6. In the fifth paragraph, the control command recognizer When the first user gesture labeled with the first command is input multiple times, the first user gesture input multiple times is learned, A hub device characterized by controlling mapping of a first control signal to a first user gesture according to a learning result.

7. In the first paragraph, the control command recognizer A hub device characterized by controlling to register a first user gesture if the first user gesture is an unregistered gesture.

8. In paragraph 7, the control command recognizer, Save the first user gesture, When the same or similar first user gestures are input more than a predetermined number of times, the first user gestures inputted the predetermined number of times are learned, A hub device characterized by controlling the first user gesture to be newly registered based on the learning result.

9. In paragraph 8, the control command recognizer A hub device characterized by using few-shot learning to learn a first user gesture inputted in the above multiple times.

10. In the 8th paragraph, the control command recognizer, A hub device characterized in that, when the newly registered first user gesture is recognized again, a first control signal is controlled to be mapped to the newly registered first user gesture.

11. In the first paragraph, the control command recognizer Determine whether the trigger word existed prior to recognition of the first command, A hub device characterized in that, when the above-mentioned trigger word exists, the first control signal is controlled to be transmitted to the external device corresponding to the first control signal.

12. In paragraph 1, The first control signal includes an operation control signal and information on an electronic device to be controlled, and the control command recognizer, A hub device characterized by controlling the above operation control signal to be transmitted to the external device corresponding to the control target electronic device information.

13. In the 12th paragraph, the control command recognizer, Determining whether the above operation control signal is valid for the above controlled electronic device, A hub device characterized in that, when determined to be valid, the operation control signal is controlled to be transmitted to the external device corresponding to the information on the electronic device to be controlled.

14. In paragraph 1, A hub device further comprising a communication unit for short-range communication with a wearable device including the voice signal sensor and the bio-signal sensor.

15. In paragraph 1, A hub device characterized in that the above biosignal is an electromyography signal.

16. A step of filtering a voice signal from a signal received from a voice signal sensor; A step of filtering a biosignal from a signal received from a biosignal sensor; A step of recognizing a first user gesture corresponding to the above biosignal and a first command corresponding to the above voice signal; A step of mapping a first control signal corresponding to a first command to a first user gesture; An electronic device remote control method, comprising: a step of transmitting a first control signal to an external device corresponding to the first control signal when a second user gesture corresponding to a subsequent bio-signal corresponds to the first user gesture; 17. In paragraph 16, If the first user gesture is a pre-registered gesture, a step of determining whether a corresponding control signal is mapped to the first user gesture; and An electronic device remote control method, comprising: a step of mapping a first control signal to a first user gesture, if the corresponding control signal is not pre-mapped to the first user gesture; 18. In paragraph 17, An electronic device remote control method, comprising: a step of transmitting a second control signal to an external device corresponding to the second control signal when a second control signal is mapped to a first user gesture; 19. In paragraph 17, A method for remotely controlling an electronic device, characterized by comprising the step of labeling a first user gesture using a first command.

20. In paragraph 19, A step of learning a first user gesture input multiple times when a first user gesture labeled with a first command is input multiple times; and A method for remotely controlling an electronic device, characterized in that it controls to map a first control signal to a first user gesture according to a learning result.

Images