Method and apparatus for removing vibration signal for gesture recognition

Abstract

An electronic device according to an embodiment may remove a vibration signal from a sensor signal including a gesture signal and the vibration signal. Particularly, the electronic device comprises: an actuator for generating vibration; a plurality of sensors including a first sensor and a second sensor for recognizing a gesture of a user of the electronic device; a memory that stores instructions; and one or more processors operatively connected to the actuator, the plurality of sensors, and the memory and including processing circuitry, wherein the instructions, when executed individually or collectively by the one or more processors, may cause the electronic device to: perform processing for suppressing or reducing a component of a first frequency in a first signal on the basis that the first frequency at which vibration occurs in the first signal acquired from the first sensor matches a vibration frequency range corresponding to the actuator; and recognize a first gesture on the basis of at least the processed first signal and a second signal acquired from the second sensor, from among a plurality of signals acquired from the plurality of sensors.

Description

Method and device for removing vibration signals for gesture recognition

[0001] Various embodiments of the present invention relate to a method for removing a vibration signal to improve gesture recognition performance within an electronic device.

[0002] Among user interface technologies, gesture recognition technology can be broadly categorized into technology that recognizes gestures through images using image sensors and technology that recognizes gestures using sensors other than image sensors (for example, an inertial measurement unit (IMU) of a linear acceleration sensor or a gyro sensor). Technologies that recognize gestures using sensors other than image sensors have the advantage of greater freedom than technologies that recognize gestures using image sensors because they can be used anytime and anywhere.

[0003] According to one embodiment, an electronic device includes an actuator for generating vibration, a plurality of sensors including a first sensor and a second sensor for recognizing a gesture of a user of the electronic device, a memory for storing instructions, and one or more processors operatively connected to the actuator, the plurality of sensors, and the memory, wherein when the instructions are executed by the one or more processors, the electronic device performs processing to suppress or reduce a component of a first frequency in a first signal obtained from the first sensor based on a first frequency at which vibration is generated matching a vibration frequency range corresponding to the actuator, and recognizes a first gesture based on at least the processed first signal and the second signal obtained from the second sensor among a plurality of signals obtained from the plurality of sensors.

[0004] A method performed by an electronic device according to one embodiment may include an operation of performing a process to suppress or reduce a first signal obtained from a first sensor based on a first frequency at which vibration occurs matching a vibration frequency range corresponding to an actuator, and an operation of recognizing a first gesture based on at least the processed first signal and a second signal obtained from the second sensor among a plurality of signals obtained from a plurality of sensors including the first sensor and the second sensor.

[0005] Electronic devices can classify gesture signals based on user gestures. However, since vibration signals generated by the electronic device's vibration and gesture signals generated by the user's gestures can overlap, problems may arise where the electronic device fails to recognize the gesture signals or incorrectly classifies the gesture signals. This decline in the recognition rate of user gestures can cause significant inconvenience in the usability of the electronic device. Conventional methods have been used to prevent this decline in user gesture recognition rates by setting vibrations to be short and weak or requiring the user to perform clear and large gestures. However, these methods clearly have the disadvantage of limiting the usability of electronic devices.

[0006] In contrast, an electronic device according to one embodiment can acquire a vibration frequency by monitoring a sensor signal acquired through a sensor in the frequency domain, and downscale a signal adjacent to the acquired vibration frequency. This method can prevent a decline in the recognition rate of user gestures while enhancing the usability of the electronic device.

[0007] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to various embodiments.

[0008] FIGS. 2A and 2B are perspective views of an electronic device according to one embodiment.

[0009] Figure 3 is an exploded perspective view of an electronic device according to one embodiment.

[0010] FIG. 4 is a diagram illustrating actuators that generate vibrations of different vibration frequency ranges included in an electronic device according to one embodiment and signals recognized by a sensing module by vibrations generated from different actuators.

[0011] FIG. 5 is a flowchart schematically illustrating a method for processing a vibration signal by monitoring a signal acquired through a sensor in the frequency domain by an electronic device according to one embodiment.

[0012] FIG. 6 is a diagram schematically illustrating a method for an electronic device according to one embodiment to monitor a user's gesture signal and vibration signal in the frequency domain and suppress the vibration signal.

[0013] FIG. 7 is a flowchart specifically explaining a method for an electronic device to monitor a user's gesture signal and vibration signal in the frequency domain and suppress the vibration signal according to one embodiment.

[0014] FIG. 8 is a diagram illustrating a method for an electronic device according to one embodiment to suppress or reduce a vibration signal based on a modification of a sensor signal that fails to detect the vibration signal.

[0015] FIGS. 9 to 11 are exemplary diagrams of a gesture recognition module included in an electronic device that suppresses or reduces a vibration signal based on an envelope signal.

[0016] FIG. 12 is a diagram schematically illustrating a gesture recognition module of an electronic device according to one embodiment of the present invention that performs a vibration frequency monitoring method and a method of suppressing or reducing a vibration signal based on an envelope signal.

[0017] FIG. 13 is a flowchart schematically illustrating a method for an electronic device according to one embodiment to process a vibration signal included in a vibration section.

[0018] FIG. 14 is a diagram schematically illustrating a process by which an electronic device recognizes a user's gesture according to one embodiment.

[0019] FIGS. 15A and 15B illustrate a scenario in which a gesture signal and a vibration signal overlap in a sensor signal detected in an electronic device according to one embodiment.

[0020] FIG. 16 is a diagram illustrating a process in which an electronic device according to one embodiment calculates a vibration section based on vibration information according to vibration.

[0021] Figure 17 illustrates sensor signals that appear when vibrations in an electronic device and a user's gesture occur in close proximity to each other.

[0022] FIG. 18 is a flowchart illustrating a process in which an electronic device according to one embodiment sets an expected vibration section based on vibration information and calculates a vibration section within the expected vibration section.

[0023] FIG. 19 is a diagram illustrating a process for changing the time required from the time an electronic device recognizes a gesture based on a user input to the time it generates vibration according to the recognition of the gesture, according to one embodiment.

[0024] FIGS. 20 and 21 are diagrams illustrating a process in which an electronic device according to one embodiment calculates a boundary point of a vibration signal and a gesture signal within an expected vibration range.

[0025] FIG. 22 is a drawing schematically illustrating the structure of a gesture recognition module of an electronic device according to one embodiment.

[0026] FIG. 23 is a diagram illustrating a process in which an electronic device passively processes a vibration signal to prevent misclassification according to one embodiment.

[0027] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0028] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0029] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0030] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0031] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0032] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0033] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0034] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0035] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0036] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0037] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0038] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0039] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0040] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, an actuator, or an electrical stimulation device.

[0041] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0042] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0043] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0044] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0045] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0046] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0047] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0048] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0049] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0050]

[0051] Referring to FIGS. 2A and 2B , an electronic device (200) according to one embodiment (e.g., the electronic device (101) of FIG. 1 ) may include a housing (210) including a first side (or front side) (201A), a second side (or back side) (210B), and a side surface (210C) surrounding a space between the first side (210A) and the second side (210B), and a fastening member (250, 260) connected to at least a portion of the housing (210) and configured to detachably fasten the electronic device (200) to a part of a user's body (e.g., a wrist, an ankle, etc.). In another embodiment (not shown), the housing may also refer to a structure forming a portion of the first side (210A), the second side (210B), and the side surface (210C) of FIGS. 2A and 2B . In one embodiment, the first side (210A) may be formed by a front plate (201) that is at least partially substantially transparent (e.g., a glass plate or a polymer plate comprising various coating layers). The second side (210B) may be formed by a substantially opaque back plate (207). The back plate (207) may be formed of, for example, coated or colored glass, ceramic, polymer, metal (e.g., aluminum, stainless steel (STS), or magnesium), or a combination of at least two of the foregoing materials. The side surface (210C) may be formed by a side bezel structure (or “side member”) (206) that is coupled to the front plate (201) and the back plate (207) and comprises a metal and / or a polymer. In some embodiments, the back plate (207) and the side bezel structure (206) may be formed integrally and comprise the same material (e.g., a metal material such as aluminum). The above-mentioned fastening member (250, 260) may be formed of various materials and shapes. The integral and multiple unit links may be formed to be mutually movable by a combination of at least two of the above-mentioned materials, such as woven fabric, leather, rubber, urethane, metal, ceramic, or a combination of the above-mentioned materials.

[0052] According to one embodiment, the electronic device (200) may include at least one of a display (220, see FIG. 3), an audio module (205, 208), a sensor module (211), a key input device (202, 203, 204), and a connector hole (209). In some embodiments, the electronic device (200) may omit at least one of the components (e.g., the key input device (202, 203, 204), the connector hole (209), or the sensor module (211)) or may additionally include other components.

[0053] The display (220) may be visually exposed, for example, through a significant portion of the front plate (201). The shape of the display (220) may correspond to the shape of the front plate (201), and may have various shapes such as a circle, an oval, or a polygon. The display (220) may be combined with or disposed adjacent to a touch detection circuit, a pressure sensor capable of measuring the intensity (pressure) of a touch, and / or a fingerprint sensor.

[0054] The audio module (205, 208) may include a microphone hole (205) and a speaker hole (208). The microphone hole (205) may have a microphone positioned therein for acquiring external sounds, and in some embodiments, multiple microphones may be positioned therein to detect the direction of sounds. The speaker hole (208) may be used as an external speaker and a receiver for calls. In some embodiments, the speaker hole (208) and the microphone hole (205) may be implemented as a single hole, or a speaker may be included without the speaker hole (208) (e.g., a piezo speaker).

[0055] The sensor module (211) can generate an electric signal or data value corresponding to an internal operating state of the electronic device (200) or an external environmental state. The sensor module (211) can include, for example, a biometric sensor module (211) (e.g., an HRM sensor) disposed on the second surface (210B) of the housing (210). The electronic device (200) can further include at least one of a sensor module not shown, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0056] The sensor module (211) may include electrode areas (213, 214) forming a portion of the surface of the electronic device (200) and a biosignal detection circuit (not shown) electrically connected to the electrode areas (213, 214). For example, the electrode areas (213, 214) may include a first electrode area (213) and a second electrode area (214) arranged on a second surface (210B) of the housing (210). The sensor module (211) may be configured such that the electrode areas (213, 214) obtain an electrical signal from a portion of the user's body, and the biosignal detection circuit detects the user's bioinformation based on the electrical signal.

[0057] The key input devices (202, 203, 204) may include a wheel key (202) disposed on a first side (210A) of the housing (210) and rotatable in at least one direction, and / or a side key button (203, 204) disposed on a side surface (210C) of the housing (210). The wheel key may have a shape corresponding to the shape of the front plate (201). In other embodiments, the electronic device (200) may not include some or all of the above-mentioned key input devices (202, 203, 204), and the key input devices (202, 203, 204) that are not included may be implemented in another form, such as a soft key, on the display (220). The connector hole (209) may accommodate a connector (e.g., a USB connector) for transmitting and receiving power and / or data with an external electronic device, and may include another connector hole (not shown) for receiving a connector for transmitting and receiving audio signals with the external electronic device. The electronic device (200) may further include, for example, a connector cover (not shown) that covers at least a portion of the connector hole (209) and blocks the inflow of external foreign substances into the connector hole.

[0058] The fastening member (250, 260) can be detachably fastened to at least a portion of the housing (210) using a locking member (251, 261). The fastening member (250, 260) can include one or more of a fixing member (252), a fixing member fastening hole (253), a band guide member (254), and a band fastening ring (255).

[0059] The fixing member (252) can be configured to fix the housing (210) and the fastening members (250, 260) to a part of the user's body (e.g., wrist, ankle, etc.). The fastening member fastening hole (253) can fix the housing (210) and the fastening members (250, 260) to a part of the user's body in response to the fastening member (252). The band guide member (254) is configured to limit the range of movement of the fastening member (252) when the fastening member (252) is fastened to the fastening member fastening hole (253), thereby allowing the fastening members (250, 260) to be fastened in close contact with a part of the user's body. The band fixing ring (255) can limit the range of movement of the fastening members (250, 260) when the fastening member (252) and the fastening member fastening hole (253) are fastened.

[0060] Referring to FIG. 3, an electronic device (300) (e.g., the electronic device (101) of FIG. 1 or the electronic device (200) of FIG. 2A) may include a side bezel structure (310), a wheel key (320), a front plate (201), a display (220), a first antenna (350), a second antenna (355), a support member (360) (e.g., a bracket), a battery (370), a printed circuit board (380), a sealing member (390), a rear plate (393), and fastening members (395, 397). At least one of the components of the electronic device (300) may be the same as or similar to at least one of the components of the electronic device (101) of FIG. 1 or the electronic device (200) of FIG. 2A, and a redundant description thereof will be omitted below. The support member (360) may be disposed inside the electronic device (300) and connected to the side bezel structure (310), or may be formed integrally with the side bezel structure (310). The support member (360) may be formed of, for example, a metallic material and / or a non-metallic (e.g., polymer) material. The support member (360) may have a display (220) coupled to one surface and a printed circuit board (380) coupled to the other surface. A processor, a memory, and / or an interface may be mounted on the printed circuit board (380). The processor may include, for example, one or more of a central processing unit (CPU), a graphic processing unit (GPU), an application processor, a sensor processor, or a communication processor. The memory may include, for example, a volatile memory or a non-volatile memory. The interface may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, and / or an audio interface.The interface may, for example, electrically or physically connect the electronic device (300) to an external electronic device and may include a USB connector, an SD card / MMC connector, or an audio connector.

[0061] The battery (370) is a device for supplying power to at least one component of the electronic device (300), and may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell. At least a portion of the battery (370) may be disposed substantially on the same plane as, for example, the printed circuit board (380). The battery (370) may be disposed integrally within the electronic device (200), or may be disposed detachably from the electronic device (200).

[0062] The first antenna (350) may be positioned between the display (220) and the support member (360). The first antenna (350) may include, for example, a near field communication (NFC) antenna, a wireless charging antenna, and / or a magnetic secure transmission (MST) antenna. The first antenna (350) may, for example, perform short-range communication with an external device, wirelessly transmit and receive power required for charging, and transmit a magnetic-based signal including a short-range communication signal or payment data. In another embodiment, the antenna structure may be formed by a portion or a combination of the side bezel structure (310) and / or the support member (360).

[0063] The second antenna (355) may be disposed between the printed circuit board (380) and the back plate (393). The second antenna (355) may include, for example, a near field communication (NFC) antenna, a wireless charging antenna, and / or a magnetic secure transmission (MST) antenna. The second antenna (355) may, for example, perform short-range communication with an external device, wirelessly transmit and receive power required for charging, and transmit a magnetic-based signal including a short-range communication signal or payment data. In another embodiment, the antenna structure may be formed by a portion or a combination of the side bezel structure (310) and / or the back plate (393).

[0064] A sealing member (390) may be positioned between the side bezel structure (310) and the rear plate (393). The sealing member (390) may be configured to block moisture and foreign substances from entering the space surrounded by the side bezel structure (310) and the rear plate (393) from the outside.

[0065]

[0066] FIG. 4 is a diagram illustrating signals obtained by a sensing module by different actuators and vibrations generated by different actuators according to one embodiment.

[0067] In one embodiment, an electronic device (e.g., electronic device 101 of FIG. 1, electronic device 200 of FIG. 2A, or electronic device 300 of FIG. 3) may be a wearable device detachably attached to a part of a user's body (e.g., a wrist). When an event occurs, the electronic device may provide haptic feedback corresponding to the event to the user. For example, the electronic device may generate a vibration corresponding to the event. For example, the electronic device may generate a vibration in response to an alarm for an event that occurs (e.g., an incoming call, an incoming text message, or an incoming instant message). As another example, the electronic device may generate a vibration feedback corresponding to a user's gesture. For example, when a call comes in to the electronic device, a user wearing the electronic device may perform a gesture (e.g., clenching and unclenching a fist) to answer the call. When the electronic device successfully recognizes the user's gesture, the electronic device may generate a vibration feedback corresponding to the successful recognition of the gesture.

[0068] In one embodiment, the electronic device may include an actuator (410). As another example, the electronic device may include an actuator (420). For example, the actuators (410) and (420) may each represent different vibration motors. For example, the actuator (410) may physically generate a linear vibration. As another example, the actuator (420) may physically generate a circular vibration. In other words, the vibration generated by the actuator (410) and the vibration generated by the actuator (420) may each have a physical shaking direction. The vibration frequency range of the vibration generated by the actuator (410) and the vibration frequency range of the vibration generated by the actuator (420) may be different from each other. For example, the vibration frequency of the vibration generated by the actuator (410) may be 180 Hz. For another example, the vibration frequency of the vibration generated by the actuator (420) may be 110 Hz. In other words, the signal sensed by the sensor module (176) in the first case (401) where the actuator (410) is mounted on the electronic device may be different from the signal sensed by the sensor module (176) in the second case (402) where the actuator (420) is mounted on the electronic device.

[0069] Hereinafter, a first case (401) in which an actuator (410) is mounted on an electronic device and a second case (402) in which an actuator (420) is mounted on an electronic device will be described separately. In addition, the description will focus on a case in which the vibration frequency of a vibration signal generated by the actuator (410) is higher than the vibration frequency generated by the actuator (420).

[0070] The sensor module (176) can sense signals (e.g., acceleration signals and / or angular velocity signals) generated due to the movement of the electronic device. The movement of the electronic device can be generated by elements within the electronic device (e.g., actuator (410) or actuator (420)) or can be generated according to a user's gesture. In this specification, a signal generated due to a user's gesture can be referred to as a gesture signal. For example, the gesture signal can include a signal corresponding to acceleration applied to the electronic device due to the gesture (e.g., acceleration signal) and / or a signal corresponding to angular velocity (e.g., angular velocity signal). There are various applications in which gesture recognition is used. For example, when a call comes in, a user wearing the electronic device can perform a gesture of clenching and unclenching his or her fist to answer the call. When the user performs the gesture of clenching and unclenching his or her fist, a subtle movement can occur in the electronic device, and the sensor module (176) can detect a gesture signal corresponding to the movement. For example, the sensor module (176) can detect changes in acceleration and / or angular velocity that occur in the electronic device as the user performs a gesture of clenching and unclenching a fist.

[0071] In the first case (401), the sensor module (176) can detect a vibration signal generated by the actuator (410). However, if the gesture signal generated based on the user's gesture overlaps with the vibration signal generated by the actuator (410), the electronic device may fail to recognize the gesture. Similarly, in the second case (402), the sensor module (176) can detect a vibration signal generated by the actuator (420). However, if the user's gesture signal overlaps with the vibration signal generated by the actuator (420), the electronic device may fail to recognize the gesture. Therefore, it is important to remove the vibration signal from the signal detected by the sensor module (176) to obtain a clearer gesture signal.

[0072] As described in FIG. 1, the sensor module (176) may include different sensors. For example, the sensor module (176) may include a first sensor and a second sensor. The different sensors included in the sensor module (176) may detect vibration signals of different physical forms. In other words, the first sensor and the second sensor may detect vibration signals of different vibration axes. For example, when the first sensor is an acceleration sensor, the first sensor may detect vibration based on a linear acceleration axis. For example, the first sensor may detect vibration that causes a linear acceleration change of the electronic device based on three axes (e.g., x-axis, y-axis, z-axis) of the acceleration sensor. For example, when the second sensor is an angular velocity sensor, the second sensor may detect vibration based on a rotation axis. For example, the second sensor may detect vibrations that cause rotational changes in the electronic device about the three axes of the angular velocity sensor (e.g., pitch, roll, yaw).

[0073] Due to the characteristics of the actuators (410, 420) (e.g., the actuator (410) generates linear vibration and the actuator (420) generates circular vibration), the vibration signal may only be generated in a specific sensor. In this case, if the frequency of the vibration is higher than the sensing frequency of the sensor, the vibration signal may lose its periodicity and appear spread across the entire frequency band. In other words, there may not be a frequency that can distinguish the vibration signal in the signal acquired through the sensor module (176). Furthermore, the vibration signal may also be overlapped in the frequency band of the gesture signal in the signal acquired through the sensor module (176).

[0074] For example, the actuator (410) can generate a physically linear high-frequency band (e.g., 180 Hz) vibration signal, and the actuator (420) can generate a physically circular low-frequency band (e.g., 110 Hz) vibration signal. In this case, the frequency band of the vibration signal detectable by the first sensor and the second sensor can be close to 100 Hz. The vibration frequency (e.g., 180 Hz) of the vibration signal generated by the actuator (410) can be higher than the vibration frequency (e.g., 100 Hz) detectable by the first sensor and the second sensor. In addition, the vibration frequency (e.g., 110 Hz) of the vibration signal generated by the actuator (420) can be close to the vibration frequency (e.g., 100 Hz) detectable by the first sensor and the second sensor. Accordingly, even if the vibration signal generated by the actuator (410) is detected by the first sensor, it may not be distinguished from the gesture signal in the frequency domain. However, the vibration frequency of the vibration signal generated by the actuator (420) may be close to the vibration frequency detectable by the second sensor. Accordingly, when the electronic device analyzes (or monitors) the signal sensed through the second sensor in the frequency domain, the vibration signal generated by the actuator (420) may be distinguished from the gesture signal.

[0075] To aid understanding, the result of the vibration signal generated by the actuator (410) being detected by the sensor module (176) may be illustrated as a graph (430). The graph (430) represents a graph in which a vibration signal is sensed by a first sensor (e.g., an acceleration sensor) of an electronic device, but a vibration signal is not sensed by a second sensor (e.g., an angular velocity sensor). For example, in a section (415) of the graph (430), a linear acceleration value is greater than a predetermined threshold value, but an angular velocity value is less than a specific threshold value. That is, it can be confirmed that in a section (415) of the graph (430), a vibration signal is detected by the acceleration sensor, but a vibration signal is not detected by the angular velocity sensor. Therefore, the graph (430) represents a case in which the first sensor detects a vibration signal, and the second sensor does not detect a vibration signal. In the first case (401), the electronic device can improve the accuracy of user gesture recognition by removing a vibration signal from the signal detected by the first sensor based on monitoring the signal detected by the first sensor in the time domain.

[0076] For another example, the result of a vibration signal generated by an actuator (420) being detected by a sensor module (176) may be illustrated as a graph (440). The graph (440) represents a graph in which a vibration signal is not measured by a first sensor (e.g., an acceleration sensor) of the electronic device, but a vibration signal is measured by a second sensor (e.g., an angular velocity sensor). For example, in a section (425) of the graph (440), it can be confirmed that the linear acceleration value is below a specific threshold value, but the angular velocity value is above a specific threshold value. Accordingly, the graph (440) may represent a case in which the second sensor detects the vibration signal, but the first sensor does not detect the vibration signal. In the second case (402), the electronic device may improve the accuracy of user gesture recognition by removing the vibration signal from the signal detected by the second sensor based on monitoring the signal detected by the second sensor in the frequency domain.

[0077] Hereinafter, FIGS. 5 to 12 describe a technique for recognizing a gesture of an electronic device user by reducing or eliminating a vibration signal among sensor signals, even if the electronic device does not calculate a vibration section from a sensor signal measured on a time axis. In addition, FIGS. 13 to 23 describe a technique for recognizing a gesture of an electronic device user by calculating a vibration section from a sensor signal measured on a time axis and reducing or eliminating a vibration signal in the corresponding vibration section.

[0078]

[0079] FIG. 5 is a flowchart schematically illustrating a method for processing a vibration signal by monitoring a signal acquired through a sensor in the frequency domain by an electronic device according to one embodiment.

[0080] In operation (510), an electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (200) of FIG. 2A, or an electronic device (300) of FIG. 3) can suppress or reduce a first signal based on the first frequency of the vibration generated from the first signal acquired from the first sensor matching a vibration frequency range corresponding to the actuator.

[0081] An electronic device according to one embodiment can detect vibration information regarding vibrations that have already occurred or are about to occur in the electronic device. The vibration information is information representing vibrations generated and / or sensed by the electronic device, and for example, the vibration information may include at least one of vibration intensity, vibration pattern, vibration period, vibration length, and vibration time. The vibration pattern may indicate that vibrations of a specific intensity are repeated at predetermined intervals. In this case, the vibration may be generated by an actuator of the electronic device. As described in FIG. 4, the actuator may, for example, provide vibration feedback in response to the identification of a user's gesture when the electronic device identifies the gesture. As another example, the actuator may generate vibrations in response to an alarm for an event occurring in the electronic device (e.g., an incoming call or a text message). The vibration frequency of a vibration signal generated by the actuator may be predetermined depending on the type of actuator. For example, depending on the type of vibrator included in the actuator and the type of vibration motor, the vibration frequency of the actuator can be predetermined as 110 Hz or 180 Hz.

[0082] As described in FIG. 4, the electronic device can convert the first signal acquired from the first sensor into a frequency band to distinguish between a gesture signal and a vibration signal. The electronic device can detect a first frequency at which vibration occurs (e.g., having the largest frequency component) based on monitoring the first signal acquired from the first sensor in the frequency domain. If the detected first frequency matches the vibration frequency of the actuator, the electronic device can suppress the vibration signal in the first signal by suppressing or reducing the frequency components of the first frequency domain. In other words, the electronic device can suppress or reduce the vibration signal even if it does not calculate the vibration section in the first signal in the time domain. A specific method for the electronic device to monitor the first signal in the frequency domain and remove the vibration signal will be described below with reference to FIG. 6.

[0083] In operation (520), an electronic device according to one embodiment can recognize a first gesture based on at least a processed first signal and a second signal obtained from a second sensor among multiple signals obtained from multiple sensors. In other words, the electronic device can remove a vibration signal from the first signal and more accurately recognize the user's gesture together with the second signal.

[0084]

[0085] FIG. 6 is a diagram schematically illustrating a method for an electronic device according to one embodiment to monitor a user's gesture signal and vibration signal in the frequency domain and suppress the vibration signal.

[0086] An electronic device according to an embodiment (e.g., the electronic device 101 of FIG. 1, the electronic device 200 of FIG. 2A, or the electronic device 300 of FIG. 3) may perform a conversion operation (621) of converting a signal acquired through a first sensor and / or a second sensor into a signal in a frequency domain. FIG. 6 illustrates an example in which the first sensor is a 3-axis angular velocity sensor and the second sensor is a 3-axis acceleration sensor. In addition, the following illustrates an example in which an actuator included in the electronic device generates a vibration signal based on a characteristic of the actuator (e.g., the generated vibration signal is physically either linear or circular) and the vibration signal is detected only by the first sensor included in the electronic device. The first signal acquired through the first sensor may include a periodicity of the vibration signal in the frequency domain. The periodicity of the vibration signal may indicate that the vibration signal is not evenly distributed across the entire frequency domain, but is detected in a specific frequency range. In general, the vibration frequency may be higher than the frequency of a user's gesture signal. In other words, the electronic device can monitor the vibration signal in the frequency domain based on a predetermined frequency value that can distinguish the vibration signal regardless of the presence or absence of the user's gesture signal. Accordingly, the electronic device can monitor by performing a conversion operation (621) of converting the first signal acquired from the first sensor into a signal in the frequency domain. The electronic device can suppress or reduce a portion of the first signal corresponding to the vibration signal based on the result of the frequency conversion of the first signal monitored. However, the electronic device must first determine whether vibration has occurred in the electronic device.

[0087] For convenience of understanding, when the first sensor corresponds to a 3-axis angular velocity sensor, the results of the first sensor detecting the gesture signal and the vibration signal may be illustrated as graphs (610, 611, 612, 613). Graph (611) is a graph for a first-axis angular velocity signal corresponding to the result of the first-axis angular velocity sensor included in the first sensor detecting the gesture signal and the vibration signal. Graph (612) is a graph for a second-axis angular velocity signal corresponding to the result of the second-axis angular velocity sensor included in the first sensor detecting the gesture signal and the vibration signal. In addition, graph (613) is a graph for a third-axis angular velocity signal corresponding to the result of the third-axis angular velocity sensor included in the first sensor detecting the gesture signal and the vibration signal. Graph (610) is a graph for a first signal corresponding to the result of combining the first-axis angular velocity signal, the second-axis angular velocity signal, and the third-axis angular velocity signal. The electronic device can determine that vibration has occurred in the electronic device based on the first axis signal in section (620) of the graph (611), the second axis signal in section (620) of the graph (612), and the third axis signal in section (620) of the graph (613). Hereinafter, a case where the electronic device determines that vibration has occurred in the electronic device will be described.

[0088] In order to determine that vibration has occurred in the electronic device, the electronic device according to one embodiment may first perform a conversion operation (621) of converting a first signal detected in the time axis into a signal in the frequency domain. For example, the electronic device may perform a conversion operation (621) of converting the first signal into a signal in the frequency domain by performing a fast Fourier transform (FFT) on the first signal. For example, when the first sensor corresponds to a three-axis angular velocity sensor, the electronic device may perform a conversion operation (621) of converting a first-axis angular velocity signal acquired from a first-axis angular velocity sensor among the three-axis angular velocity sensors into a signal in the frequency domain through FFT. In other words, the electronic device may perform a conversion operation (621) of converting a graph (611) corresponding to the first-axis angular velocity signal into a graph (630) in the frequency domain. Similarly, the electronic device can perform a transformation operation (621) to transform a graph (612) into a graph (631) in the frequency domain, and can perform a transformation operation (621) to transform a graph (613) into a graph (632) in the frequency domain.

[0089] The electronic device may determine that vibration has occurred in the electronic device based on the frequency components of the first-axis angular velocity signal in the frequency domain, the frequency components of the second-axis angular velocity signal, and the frequency components of the third-axis angular velocity signal. For example, the electronic device may determine that vibration has occurred in the electronic device when the frequency component corresponding to the first frequency (e.g., frequency 640) in the first-axis angular velocity signal in the frequency domain (e.g., the first-axis angular velocity signal in the frequency domain corresponding to graph 630) is less than a predetermined threshold value, the frequency component corresponding to the first frequency (e.g., frequency 640) in the second-axis angular velocity signal in the frequency domain (e.g., the second-axis angular velocity signal in the frequency domain corresponding to graph 631) is greater than or equal to the predetermined threshold value, and the frequency component corresponding to the first frequency (e.g., frequency 640) in the third-axis angular velocity signal in the frequency domain (e.g., the third-axis angular velocity signal corresponding to graph 632) is greater than or equal to the predetermined threshold value. When the electronic device determines that a vibration has occurred in the electronic device, the electronic device can suppress or reduce the vibration signal in the first signal by suppressing and reducing the frequency components of the frequency corresponding to the vibration signal (e.g., the first frequency and frequency (640)). For example, the electronic device can suppress or reduce the vibration signal in the first signal based on filtering in the frequency components of a frequency range lower than the frequency corresponding to the vibration signal in the first signal (e.g., the first frequency, frequency (640)) and filtering out the frequency components of a frequency range higher than the frequency (e.g., the first frequency, frequency (640)).

[0090]

[0091] FIG. 7 is a flowchart specifically explaining a method for an electronic device to monitor a user's gesture signal and vibration signal in the frequency domain and suppress the vibration signal according to one embodiment.

[0092] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2A, or electronic device (300) of FIG. 3) may convert signals detected by a plurality of sensors, including a first sensor and a second sensor for recognizing a user's gesture, into a frequency domain. The electronic device may suppress or reduce a frequency component of a frequency at which vibration is determined to have occurred based on monitoring the signals converted into the frequency domain. However, in the frequency domain, a gesture signal and a vibration signal may overlap in some of the same frequency ranges. The electronic device may suppress or reduce a frequency component of some frequency ranges where the gesture signal and the vibration signal overlap. As a result, the electronic device may experience a loss of information about the gesture signal in some frequency ranges where the gesture signal and the vibration signal overlap. Therefore, the electronic device needs to accurately determine whether a specific frequency component in the frequency domain is a vibration signal that does not overlap with the gesture signal. Below, a method is specifically described in which an electronic device monitors a first signal acquired from a first sensor in the frequency domain to determine the frequency and frequency components at which vibration is determined to have occurred, and then suppresses or reduces the corresponding frequency components after vibration is determined to have occurred. The operation (510) of FIG. 5 is described in more detail below.

[0093] In operation (710), the electronic device can store a first signal acquired from a first sensor for a predetermined time period.

[0094] In one embodiment, the electronic device may include a three-axis angular velocity sensor as the first sensor. Although the type of the first sensor is not limited thereto, the following description focuses on a case where the first sensor is a three-axis angular velocity sensor. The three-axis angular velocity sensor may include a first-axis angular velocity sensor (e.g., an X-axis angular velocity sensor (GYR X)), a second-axis angular velocity sensor (e.g., a Y-axis angular velocity sensor (GYR Y)), and a third-axis angular velocity sensor (e.g., a Z-axis angular velocity sensor (GYR Z)). The first signal detected by the first sensor may represent a signal in which a first-axis angular velocity signal detected by the first-axis angular velocity sensor, a second-axis angular velocity signal detected by the second-axis angular velocity sensor, and a third-axis angular velocity signal detected by the third-axis angular velocity sensor are combined. Accordingly, the electronic device may store the first angular velocity signal, the second angular velocity signal, and the third angular velocity signal, respectively, for a predetermined time interval. Additionally, the electronic device can store a first signal that combines the first to third axis angular velocity signals. For example, if the predetermined time interval is 320 ms and the operating frequencies of the first angular velocity sensor, the second angular velocity sensor, and the third angular velocity sensor included in the first sensor are 100 Hz, the electronic device can sample and store 32 of each of the first angular velocity signal, the second angular velocity signal, and the third angular velocity signal.

[0095] In operation (720), the electronic device can convert the stored first signal into a signal in the frequency domain and store a candidate vibration frequency based on the frequency component.

[0096] In one embodiment, the electronic device can perform a fast Fourier transform (FFT) of a predetermined size (e.g., an N-size FFT) on each of the stored first angular velocity signal, the second angular velocity signal, and the third angular velocity signal to extract Fourier coefficients. The Fourier coefficients can correspond to frequency components. For example, the electronic device can perform a 32-size FFT on each of the 32 sampled first angular velocity signals, the 32 sampled second angular velocity signals, and the 32 sampled third angular velocity signals to extract Fourier coefficients corresponding to frequency components. The electronic device can store the extracted Fourier coefficients, that is, the frequency corresponding to the maximum value among the frequency components, as a candidate vibration frequency. For example, the electronic device can store the frequency corresponding to the maximum value among the frequency components in the frequency domain of the first-axis angular velocity signal as the first-axis candidate vibration frequency. For example, the electronic device may store a frequency corresponding to a maximum value among frequency components in the frequency domain of the second-axis angular velocity signal as a second-axis candidate vibration frequency. For example, the electronic device may store a frequency corresponding to a maximum value among frequency components in the frequency domain of the third-axis angular velocity signal as a third-axis candidate vibration frequency.

[0097] In operation (730), the electronic device may determine whether the stored candidate vibration frequency matches a vibration frequency range corresponding to the actuator, and may determine that vibration has occurred for the electronic device based on a comparison result of a frequency component corresponding to the candidate vibration frequency and a predetermined threshold value.

[0098] In one embodiment, the electronic device can determine whether the candidate vibration frequency matches the vibration frequency range corresponding to the actuator. As described in FIG. 4, the actuator can generate a vibration signal in a specific vibration frequency range depending on its hardware characteristics. Alternatively, the actuator can be controlled to generate a vibration signal in a vibration frequency range desired by the user based on an internal algorithm or a control signal. In other words, the range of vibration frequencies generated by the electronic device can be predetermined depending on the type of actuator. For example, when N is 32, the 5th or 6th Fourier coefficient among the Fourier coefficients corresponding to the 32-size FFT for the first signal can fall within the vibration frequency range corresponding to the actuator. The electronic device may determine that the candidate vibration frequency and the vibration frequency range corresponding to the actuator match when the first axis candidate vibration frequency is separated from the vibration frequency range corresponding to the actuator by a predetermined threshold value or more, and the second axis candidate vibration frequency and the third axis candidate vibration frequency are close to the vibration frequency range corresponding to the actuator by a predetermined threshold value or less. At the same time, the electronic device may determine that vibration has occurred for the electronic device when the frequency component of the first axis candidate vibration frequency is less than the predetermined threshold value, and the frequency component of the second axis candidate vibration frequency and the frequency component of the third axis candidate vibration frequency are equal to or greater than the predetermined threshold value.

[0099] In operation (740), when it is determined that vibration has occurred, the electronic device may determine a candidate vibration frequency from the first signal as a first frequency, and suppress or reduce a vibration signal included in the first signal based on the first frequency.

[0100] In one embodiment, the electronic device can suppress or reduce a vibration signal included in a first signal for a preset period of time. For example, the electronic device can set a maximum vibration time of a vibration signal generated by an actuator to 4.8 seconds. When the electronic device sets the maximum vibration time of a vibration signal generated by an actuator to 4.8 seconds, the electronic device can suppress or reduce the vibration signal included in the first signal for 4.8 seconds. The electronic device can determine a candidate vibration frequency as a first frequency (e.g., frequency 640 of FIG. 6) in the first signal. The electronic device can suppress or reduce the first signal by filtering in frequency components in a frequency range lower than the first frequency and filtering out frequency components in a frequency range higher than the first frequency for a preset period of time.

[0101] In operation (750), the electronic device may determine whether a preset period of time has elapsed. For example, the electronic device may determine whether a preset period of time has elapsed at regular time intervals. For example, the electronic device may determine whether a preset period of time corresponding to the maximum vibration time has elapsed at one-second intervals.

[0102] In operation (760), the electronic device may determine that suppression or reduction of the frequency component is completed when it is determined that a preset time has elapsed.

[0103]

[0104] FIG. 8 is a diagram illustrating a method for an electronic device according to one embodiment to suppress or reduce a vibration signal based on a modification of a sensor signal that does not include the influence of the vibration signal in the time domain.

[0105] An electronic device according to one embodiment (e.g., the electronic device 101 of FIG. 1, the electronic device 200 of FIG. 2A, or the electronic device 300 of FIG. 3) may include a first sensor and a second sensor. For example, the electronic device may use the first sensor and the second sensor to detect vibration feedback generated in the electronic device in response to a user's gesture, or a vibration signal generated based on an event (e.g., a phone call, a text message, etc.) that occurs in the electronic device. The vibration frequency range of the vibration detected by the first sensor and the physical properties of the vibration detected by the second sensor may be different from each other. For example, the vibration detected by the first sensor may correspond to vibration generated based on a rotational movement. Additionally, the vibration detected by the second sensor may correspond to vibration generated based on a linear movement. In other words, based on the physical properties of the vibration signal generated in the electronic device, the first sensor may detect vibration, and the second sensor may not detect vibration. Below, a method is described for removing a vibration signal by monitoring a first signal along a time axis based on a second signal obtained from a second sensor that is not affected by vibration.

[0106] FIG. 8 illustrates graphs (811, 812, 813, 814) for a first signal (810) acquired from a first sensor of an electronic device and graphs (821, 822, 823, 824) for a second signal (820) acquired from a second sensor. The graphs (811, 812, 813, 814) may include a vibration signal in a section (815) and a gesture signal in a section (816). In contrast, the graphs (821, 822, 823, 824) may not include a vibration signal in a section (815) and may include a gesture signal in a section (816). In other words, the first signal (810) acquired from the first sensor may include a vibration signal and a gesture signal, and the second signal (820) acquired from the second sensor may include only a gesture signal. The electronic device can extract an envelope signal (830) based on the shape of the second signal (820). For example, the electronic device can extract the envelope signal (830) from a signal included in at least one of the graphs (821, 822, 823, 824) corresponding to the second signal (820) that does not include a vibration signal. When the second sensor corresponds to a three-axis acceleration sensor, the graph (821) (e.g., LACC M) can correspond to a graph that is a sum of a graph (822) (e.g., LACC X) of a signal acquired by a first-axis acceleration sensor among the three-axis acceleration sensors, a graph (823) (e.g., LACC Y) of a signal acquired by a second-axis acceleration sensor, and a graph (824) (e.g., LACC Z) of a signal acquired by a third-axis acceleration sensor. Although FIG. 8 illustrates that the electronic device extracts an envelope signal (830) based on a signal included in a graph (821) of a second signal (820), the electronic device may also extract an envelope signal from a graph (822), a graph (823), or a graph (824).The extracted envelope signal (830) may exhibit characteristics that do not include a vibration signal in the section (815) and include a user's gesture signal in the section (816). For reference, the envelope signal (830) may refer to a signal that represents a change in the magnitude of a complex vibration signal. The envelope signal (830) may include information about the amplitude of a signal that changes over time. For example, the electronic device may extract the envelope signal (830) from the second signal (820) based on a Hilbert transform or an upper and lower envelope extraction method. However, the above examples are only examples of methods by which the electronic device extracts the envelope signal (830), and the method by which the electronic device extracts the envelope signal (830) is not limited thereto. The electronic device may smooth the envelope signal (830). The electronic device can reduce distortion of the second signal (820) that occurs when extracting the envelope signal (830) from the second signal (820) by flattening the envelope signal (830). A method by which the electronic device flattens the envelope signal (830) is described in detail below with reference to FIGS. 9 to 11.

[0107] The electronic device may perform processing to suppress or reduce the first signal (810) based on the extracted envelope signal (830). For example, the electronic device may apply the extracted envelope signal (830) to the first signal (810) as a filter to suppress or reduce the first signal (810). For example, the electronic device may multiply the extracted envelope signal (830) by the first signal (810) to strengthen the signal in the section (816) including the gesture signal in the first signal and weaken the signal in the section (815) including the vibration signal. For example, if the first sensor corresponds to a three-axis angular velocity sensor, the graph (811) (e.g., GYRO M) may correspond to a graph that sums all of the graphs (812) (e.g., GYRO X) of the signal acquired from the first-axis angular velocity sensor, the graph (813) (e.g., GYRO Y) of the signal acquired from the second-axis angular velocity sensor, and the graph (813) (e.g., GYRO Z) of the signal acquired from the third-axis angular velocity sensor. The electronic device may suppress or reduce the vibration signal in the first signal (810) by multiplying the extracted envelope signal (830) by at least one of the signals shown in the graphs (811, 812, 813, 814). The envelope signal (830) may have a value corresponding to '0' in the section (815). Accordingly, the electronic device can suppress or reduce the signal in the section (815) corresponding to the vibration signal by multiplying the envelope signal (830) by the signal shown in the graphs (811, 812, 813, 814).

[0108]

[0109] FIGS. 9 to 11 are exemplary diagrams of a gesture recognition module included in an electronic device that suppresses or reduces a vibration signal based on an envelope signal.

[0110] In one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2A, or electronic device (300) of FIG. 3) can recognize a user's gesture through a gesture recognition module (900). The electronic device may include a sensor module (910) and a gesture recognition module (900).

[0111] The sensor module (910) can transmit sensor signals generated by a plurality of sensors included in the sensor module (910) to the gesture recognition module (900). For example, an angular velocity signal can be generated from a gyro sensor, and a linear acceleration signal can be generated from a linear acceleration sensor.

[0112] The gesture recognition module (900) can receive a sensor signal from a sensor module (910) (e.g., the sensor module (176) of FIG. 1). For example, the sensor module (910) can include a first sensor (e.g., an angular velocity sensor (GYRO)) and a second sensor (e.g., an acceleration sensor (LACC)). Accordingly, the gesture recognition module (900) can receive a first signal from the first sensor included in the sensor module (910) and a second signal from the second sensor. At this time, the first signal can be a signal including a signal corresponding to a user's gesture and a vibration generated in an electronic device, and the second signal can be a signal including a signal corresponding to the user's gesture. In other words, the first signal can mean a signal including a vibration signal, and the second signal can mean a signal not including a vibration signal. The gesture recognition module (900) can process a vibration signal included in the received sensor signal. For example, the gesture recognition module (900) can perform a preprocessing operation (920), a segmentation operation (930), And by performing a filtering operation (950), the vibration signal included in the received sensor signal can be processed. The gesture recognition module (900) can recognize the gesture of the electronic device user by performing a feature extraction operation (970) and a classification operation (980) on the sensor signal from which the vibration signal has been processed.

[0113] For example, as illustrated in FIG. 9, the gesture recognition module (900) of the electronic device may perform a preprocessing operation (920). For example, in the preprocessing operation (920), the gesture recognition module (900) may remove signals corresponding to user movements other than the user's gestures through a high-pass filter.

[0114] The gesture recognition module (900) may perform a segmentation operation (930) that monitors a preprocessed sensor signal and crops a portion where the signal jumps. For example, in the segmentation operation (930), the gesture recognition module (900) may obtain a valid signal section by cropping a valid signal section within the preprocessed sensor signal. For example, a valid signal section may indicate a section that includes a sensor signal greater than a predetermined threshold value among sensor signals. For example, if a user of an electronic device leaves the electronic device still, the sensor signal may indicate 0. If the user moves the electronic device, the sensor signal of the electronic device may have a value other than 0 depending on the user's movement. In this case, a section that includes a signal generated according to the movement of the electronic device may be designated as a valid signal section.

[0115] The gesture recognition module (900) can perform a filtering operation (950) to suppress or reduce a vibration signal in a sensor signal cropped through a segmentation operation (930). For example, FIG. 9 assumes that a first signal (e.g., GYRO M) received by the gesture recognition module (900) from a first sensor included in the sensor module (910) includes a vibration signal. At the same time, FIG. 9 assumes that a second signal (e.g., LACC M) received by the gesture recognition module (900) from a second sensor included in the sensor module (910) does not include a vibration signal. The gesture recognition module (900) can extract an envelope signal (e.g., the envelope signal (830) of FIG. 8) corresponding to the second signal (e.g., LACC M) that does not include a vibration signal. The gesture recognition module (900) can perform a smoothing operation (960) on the extracted envelope signal. For example, the gesture recognition module (900) can perform a 5-tap smoothing operation on the amplitude of the envelope signal. For reference, the 5-tap smoothing operation can refer to an operation of smoothing the envelope signal by calculating an average using five data included in the envelope signal. As another example, the gesture recognition module (900) can perform a smoothing operation (960) that simplifies the amplitude of the envelope signal into a rectangular function. In other words, the gesture recognition module (900) can select a specific smoothing operation (960) from among the examples above based on the distortion of the signal. For example, the gesture recognition module (900) can amplify the gesture signal and simultaneously remove the vibration signal even if the gesture signal is distorted by smoothing the envelope signal into a rectangular function. For another example, the gesture recognition module (900) can remove only the vibration signal without distortion of the gesture signal by flattening the envelope signal based on a 5-tap flattening operation.The 5-tap flattening operation may require a higher computational load than flattening using a square function. Therefore, the gesture recognition module (900) can reduce power consumption by selecting an appropriate flattening operation (960).

[0116]

[0117] The gesture recognition module (900) can perform a filtering operation (950) by multiplying a signal obtained by flattening the envelope signal of the second signal by the first signal. Through this, the gesture recognition module (900) can suppress or reduce the vibration signal included in the first signal and amplify the gesture signal included in the first signal.

[0118] The gesture recognition module (900) can perform a feature extraction operation (970) to extract various features from a suppressed or reduced vibration signal obtained by performing a filtering operation (950).

[0119] Finally, the gesture recognition module (900) may perform a classification operation (980) to classify a gesture signal from a signal in which a vibration signal is suppressed or reduced by using various features acquired through the feature extraction operation (970). Classifying the gesture signal may refer to determining the type of gesture corresponding to the gesture signal. For example, the type of gesture determined by the gesture recognition module (900) of the electronic device may include gestures such as an (OCO: open-clench-open) gesture, a fist clenched and unclenched twice, a pinch, or a thumb-index-finger double-click.

[0120] For example, in the classification operation (980), the gesture recognition module (900) can classify the gesture signal based on inputting various features of the signal extracted through the feature extraction operation (970) into a classifier. For example, the classifier can include a machine learning model, a deep learning model, and a neural network model capable of classifying gestures. The classifier can be trained based on machine learning or deep learning to classify the gesture signal with high accuracy. For example, in FIG. 9, the gesture recognition module (900) can perform a classification operation (980) based on a classifier trained with a signal obtained by performing a filtering operation (950) through an envelope signal corresponding to a second signal (e.g., LACC M).

[0121] In Fig. 10, unlike in Fig. 9, it is assumed that the first signal (e.g., GYRO M) received by the gesture recognition module (1000) of the electronic device from the sensor module (910) does not include a vibration signal, and the second signal (e.g., LACC M) includes a vibration signal. The preprocessing operation (920), segmentation operation (930), and feature extraction operation (970) performed by the gesture recognition module (1000) in Fig. 10 are the same as the operations performed by the gesture recognition module (900) in Fig. 9, and therefore will not be described again.

[0122] In FIG. 10, the gesture recognition module (1000) can extract an envelope signal from a first signal (e.g., GYRO M) that does not include a vibration signal. The gesture recognition module (1000) can perform a smoothing operation (1060) on the extracted envelope signal. For example, the gesture recognition module (1000) can perform a 5-tap smoothing operation on the extracted envelope signal. The gesture recognition module (1000) can perform a filtering operation (1050) by multiplying a signal obtained by smoothing the envelope signal of the first signal by a second signal. Through this, the gesture recognition module (1000) can suppress a vibration signal included in the second signal and amplify the gesture signal.

[0123] The gesture recognition module (1000) can perform a classification operation (1080) in the same manner as the classification operation (980) performed in FIG. 9. However, when the gesture recognition module (1000) performs the classification operation (1080) through a classifier, the classifier for performing the classification operation (1080) may be a classifier trained based on machine learning or deep learning using a signal obtained by performing a filtering operation (1050) through an envelope signal corresponding to the first signal (e.g., GYRO M).

[0124] In FIG. 11, the gesture recognition module (1100) extracts an envelope signal corresponding to a first signal (e.g., GYRO M) received from a sensor module (910), and further extracts another envelope signal corresponding to a second signal (e.g., LACC M), and then performs a combining operation (1170) that combines the envelope signal and the other envelope signal to perform a filtering operation (1150) on the combined envelope signal generated.

[0125] The preprocessing operation (920), segmentation operation (930), and feature extraction operation (970) performed by the gesture recognition module (1100) in FIG. 11 are the same as the operations performed by the gesture recognition module (900) in FIG. 9 and the gesture recognition module (1000) in FIG. 10, and therefore will not be described again.

[0126] The gesture recognition module (1100) can extract an envelope signal corresponding to a first signal (e.g., GYRO M) of a valid signal section obtained through a segmentation operation (930), and can extract another envelope signal corresponding to a second signal (e.g., LACC M) of the valid signal section. The gesture recognition module (1100) can perform a flattening operation (1160) on the envelope signal corresponding to the first signal and another envelope signal corresponding to the second signal. The gesture recognition module (1100) can perform a combining operation (1170) that combines the envelope signal on which the flattening operation (1160) has been performed and another envelope signal on which the flattening operation (1160) has been performed, to generate a combined envelope signal. In other words, the gesture recognition module (1100) can extract envelope signals from all signals and perform a combining operation (1170) to combine the extracted envelope signals to generate a combined envelope signal, without having to check which of the first signal and the second signal received from the sensor module (910) includes the vibration signal. For example, if the envelope signal corresponding to the first signal has a value of 0 in a section corresponding to vibration, when the gesture recognition module (1100) performs a combining operation (1170) (e.g., multiplication) to combine the envelope signal corresponding to the first signal and another envelope signal corresponding to the second signal, the value of the section corresponding to vibration in the combined envelope signal may correspond to 0. The gesture recognition module (1100) can suppress or reduce the first signal (e.g., GYRO M) and the second signal (e.g., LACC M), respectively, based on the combined envelope signal. For example, the gesture recognition module (1100) can suppress or reduce a vibration signal in the first signal (e.g., GYRO M) by multiplying the combined envelope signal, and can simultaneously suppress or reduce a vibration signal in the second signal (e.g., LACC M) by multiplying the combined envelope signal.

[0127]

[0128] FIG. 12 is a diagram schematically illustrating a method for performing a vibration frequency monitoring method and a method for suppressing or reducing a vibration signal based on an envelope signal by a gesture recognition module of an electronic device according to one embodiment.

[0129] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2A, or the electronic device (300) of FIG. 3) can recognize a user's gesture through a gesture recognition module (1200). The electronic device may include a sensor module (1210) and a gesture recognition module (1200). The sensor module (1210) included in the electronic device is the same as the sensor module (910) described in FIG. 9, and therefore, a duplicate description will be omitted.

[0130] The gesture recognition module (1200) can suppress or reduce a vibration signal from a sensor signal received from the sensor module (1210) in order to recognize a user's gesture. The gesture recognition module (1200) can perform at least one of a frequency-axis vibration monitoring operation (1212) and a filtering operation (1230) based on an envelope signal in order to remove the vibration signal from the sensor signal. In FIG. 12, the gesture recognition module (1200) is illustrated as performing the frequency-axis vibration monitoring operation (1212) before the filtering operation (1230) using an envelope signal extracted based on time-axis vibration monitoring. However, the order of the operations performed in the gesture recognition module (1200) is not limited thereto. For example, unlike FIG. 12, the gesture recognition module (1200) may first perform the filtering operation (1230) based on the envelope signal and then perform the frequency-axis vibration monitoring operation (1212). Furthermore, the gesture recognition module (1200) may perform the segmentation operation (1230) before removing the vibration signal, or may perform the segmentation operation (1230) after removing the vibration signal. In other words, the order of the operations (1210, 1211, 1212, 1220, 1230, 1240, 1250, and 1260) arranged in FIG. 12 may be changed.

[0131] The gesture recognition module (1200) can perform a preprocessing operation (1211), a frequency axis vibration signal monitoring operation (1212), a vibration signal processing operation (1220), a filtering operation (1230), a segmentation operation (1240), a feature extraction operation (1250), and a classification operation (1260) on a sensor signal received from a sensor module (1210). The preprocessing operation (1211), the segmentation operation (1240), the feature extraction operation (1250), and the classification operation (1260) performed by the gesture recognition module (1200) are the same as the operations in FIGS. 9 to 11, and therefore, a duplicate description thereof will be omitted.

[0132] The gesture recognition module (1200) can perform a frequency axis vibration signal monitoring operation (1212) and a vibration signal processing operation (1220). For example, in the frequency axis vibration signal monitoring operation (1212), the gesture recognition module (1200) can convert the sensor signal generated by the preprocessing operation (1211) into a frequency domain signal to determine whether vibration has occurred. If the gesture recognition module (1200) determines that vibration has occurred in the sensor signal through the frequency monitoring operation (1212), it can perform a vibration signal processing operation (1220) to suppress or reduce a frequency component corresponding to the vibration frequency. However, since the frequency monitoring operation (1212) is a method for removing a vibration signal from a sensor signal when the vibration frequencies of the vibration signal and the gesture signal do not overlap with each other, the frequency axis vibration signal monitoring operation (1212) and the vibration signal processing operation (1220) alone may not be able to completely remove the vibration signal included in the sensor signal.

[0133] The gesture recognition module (1200) may perform a filtering operation (1230) corresponding to one of the filtering operations (950, 1050, 1150) described in FIGS. 9 to 11 on a sensor signal generated in a frequency-axis vibration signal monitoring operation (1212) or a vibration signal processing operation (1220). For example, the gesture recognition module (1200) may perform a filtering operation (1230) on a sensor signal from which a vibration signal that is distinguishable from a gesture signal in the frequency domain has been removed by the vibration signal processing operation (1220). As another example, when the vibration signal and the gesture signal are not distinguishable in the frequency domain, the gesture recognition module (1200) may perform a filtering operation (1240) on a sensor signal after the frequency-axis vibration signal monitoring operation (1212) without the vibration signal processing operation (1220).

[0134] In other words, the gesture recognition module (1200) can secondarily remove the vibration signal by monitoring the vibration signal on the time axis that was not removed in the frequency axis vibration signal monitoring operation (1212) through the filtering operation (1230).

[0135] The gesture recognition module (1200) can recognize a user's gesture for an electronic device by performing a segmentation operation (1240), a feature extraction operation (1250), and a classification operation (1260) on a sensor signal generated after a filtering operation (1230).

[0136]

[0137]

[0138] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0139] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0140] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0141] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0142] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0143] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0144]

[0145] In the above, a technology has been described for recognizing the gestures of an electronic device user by reducing or eliminating vibration signals among sensor signals, even if the electronic device does not calculate the vibration interval from the sensor signal measured along the time axis. Below, a technology is specifically described for reducing or eliminating vibration signals in the vibration interval by calculating the vibration interval from the sensor signal measured along the time axis.

[0146] FIG. 13 is a flowchart schematically illustrating a method for an electronic device according to one embodiment to process a vibration signal included in a vibration section.

[0147] In operation (1310), an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2A, or electronic device (300) of FIG. 3) can identify vibration information according to vibration.

[0148] In operation (1320), the electronic device may perform processing to suppress or reduce the signal size of a vibration section determined based on vibration information in a first signal obtained from a first sensor.

[0149] In one embodiment, an electronic device may include a plurality of sensors, including a first sensor and a second sensor, for recognizing a user's gesture. The first sensor may represent a sensor capable of detecting vibration. The first sensor may include a linear acceleration sensor. The second sensor may represent a sensor other than the first sensor. The second sensor may be a sensor that detects vibrations of different physical properties from the first sensor. For example, the vibration detected by the first sensor may be a linear vibration. In another example, the vibration detected by the second sensor may be a circular vibration. For example, the second sensor may include at least one of a gyro sensor, a PPG sensor, or an EMG sensor different from the first sensor. In FIG. 13 and below, the description is based on the case where the first sensor is a linear acceleration sensor and the second sensor is a gyro sensor, but the present invention is not limited thereto. In addition, in this document, the first sensor or the second sensor is merely a method for expressing sensors that are distinct from each other, and is not intended to be limiting.

[0150] In operation (1330), the electronic device can recognize a first gesture based on at least a processed first signal and a second signal obtained from a second sensor among a plurality of signals obtained from a plurality of sensors.

[0151]

[0152] FIG. 14 is a diagram schematically illustrating a process by which an electronic device recognizes a user's gesture according to one embodiment.

[0153] Gesture recognition is used in various applications. For example, when a call comes in, a user wearing an electronic device (1401) (e.g., the electronic device (101) of FIG. 1 , the electronic device (200) of FIG. 2A , or the electronic device (300) of FIG. 3 ) may perform an open-clench-open (OCO) gesture (1430) to answer the call, and the electronic device (1401) may perform a function of answering the incoming call in response to the recognition of the user's OCO gesture (1430). The electronic device (1401) may recognize the user's gesture through the gesture recognition module (1410). The following describes a general process by which the electronic device (1401) recognizes the user's gesture.

[0154] The gesture recognition module (1410) can communicate with a sensor module (1420) including a plurality of sensors including a first sensor and a second sensor. The first sensor can include a linear acceleration sensor. The second sensor can be a sensor other than a linear acceleration sensor, for example, a gyro sensor and a PPG sensor. Hereinafter, the description will be based on the case where the first sensor is a linear acceleration sensor and the second sensor is a gyro sensor, but the present invention is not limited thereto. For example, the first sensor can be a gyro sensor and the second sensor can be a linear acceleration sensor. The gesture recognition module (1410) can be in the form of software that is stored in a memory (e.g., the memory (130) of FIG. 1) and executed by or included in an application (e.g., the application (146) of FIG. 1), or can be in the form of separate hardware.

[0155] The sensor module (1420) can transmit sensor signals generated by a plurality of sensors included in the sensor module (1420) to the gesture recognition module (1410). For example, an angular velocity signal can be generated from a gyro sensor, and a linear acceleration signal can be generated from a linear acceleration sensor.

[0156] The gesture recognition module (1410) can determine a gesture performed by a user based on sensor signals (e.g., a PPG signal (1441), an angular velocity signal (1442), and / or a linear acceleration signal (1443)) received from the sensor module (1420). The gesture recognition module (1410) can synchronize the sampling periods of the sensors included in the sensor module (1420).

[0157] First, the gesture recognition module (1410) can perform a preprocessing operation (1411) on the sensor signal. For example, in the preprocessing operation (1411), the gesture recognition module (1410) can perform a function of removing signals corresponding to user movements other than the user's gestures through a high-pass filter.

[0158] The gesture recognition module (1410) may observe the preprocessed sensor signal and then perform a segmentation operation (1412) to crop out a portion where the signal jumps. For example, in the segmentation operation (1412), the gesture recognition module (1410) may obtain a valid signal section by cropping a valid signal section within the preprocessed sensor signal.

[0159] The gesture recognition module (1410) can perform a feature extraction operation (1413) that extracts multiple features from a signal section obtained by performing a segmentation operation (1412).

[0160] Finally, the gesture recognition module (1410) may perform a classification operation (1414) to classify a gesture signal appearing in a valid signal section using features of the valid signal section acquired through the feature extraction operation (1413). Classifying the gesture signal may refer to determining the type of gesture corresponding to the gesture signal. The type of gesture determined by the electronic device (1401) may include, for example, a fist-clench-open (OCO) gesture, a fist-clenching and opening twice, a pinch, or a thumb-index-finger-pinch twice.

[0161] For example, in the classification operation (1414), the gesture recognition module (1410) can classify a gesture signal appearing in a valid signal section based on inputting features of the valid signal section into a classifier. For example, the classifier can include a network capable of classifying gestures. The network can have a random forest structure based on a boosting tree. The network can be trained based on machine learning or deep learning to classify gesture signals with high accuracy.

[0162] When a user wearing an electronic device (1401) performs a gesture, the muscles from the back of the hand to the elbow contract and relax slightly due to the movement of the fingers, causing the electronic device (1401) to shake. Since the electronic device (1401) detects minute signals of the user's muscles through a sensor, the patterns of the sensor signals (e.g., PPG signal (1441), angular velocity signal (1442), and / or linear acceleration signal (1443)) may vary depending on the user or the state in which the electronic device (1401) is worn. Since numerous signal patterns may appear for each gesture, it is very difficult to classify gesture signals using rules created by a person. Therefore, it is necessary to mechanically learn the features of each gesture to generate numerous rules, and to accurately classify gesture signals based on the generated rules. A network with a random forest structure based on boosting trees embedded in the classifier generates accurate criteria for classifying gesture signals by comparing the characteristics of various signal patterns. Using these criteria, the network increases the classification accuracy of gesture signals while avoiding bias toward a single criterion by generating multiple criteria. In other words, the network can prevent overfitting to specific signal patterns.

[0163] FIGS. 15A and 15B illustrate a scenario in which a gesture signal and a vibration signal overlap in a sensor signal detected in an electronic device according to one embodiment.

[0164] According to one embodiment, a signal (hereinafter, “gesture signal”) detected by a sensor (e.g., a first sensor or a second sensor) of an electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (200) of FIG. 2A, an electronic device (300) of FIG. 3, or an electronic device (1401) of FIG. 14) when a user performs a gesture and a signal (hereinafter, “vibration signal”) detected by vibration generation in the electronic device may overlap with each other. For example, the electronic device may generate vibration according to vibration feedback corresponding to a gesture or may generate vibration according to an alarm for an event occurring in the electronic device. The electronic device may detect the gesture signal within a section in which the vibration signal appears. In this case, a problem may arise in that the vibration signal and the gesture signal overlap in a specific section, making it difficult to accurately extract the gesture signal.

[0165] Figure 15a illustrates a scenario in which a vibration signal detected by vibration according to vibration feedback and a gesture signal detected by a user's gesture performance overlap.

[0166] Vibration feedback may indicate vibrations to notify the user whether the user input was successfully received when the electronic device receives user input. The electronic device may generate vibration feedback after a preset time from the time the user input was received. The user input may vary, such as input based on the user's gesture performance and user touch input on a touchscreen interface. A scenario in which vibration feedback may affect the performance of gesture recognition is illustrated in FIG. 15a.

[0167] Referring to FIG. 15A, a user of an electronic device may perform a gesture to control the electronic device. A sensor of the electronic device may detect a gesture signal (1511) generated by the user's gesture. The electronic device may recognize the gesture performed by the user, execute a function corresponding to the recognized user's gesture, and generate vibration as vibration feedback to notify the user that the gesture has been recognized normally and that the function corresponding to the recognized gesture has been executed. The sensor of the electronic device may detect a vibration signal (1512) generated by the vibration feedback. Then, the user of the electronic device may recognize (1513) the vibration generated by the electronic device. The user who recognizes the vibration may recognize that the desired function has been executed and perform the next gesture for the next control of the electronic device. In this case, the user typically performs the next gesture after the vibration signal (1512) ends. However, as shown in FIG. 15A, a user who desires quick control may perform the next gesture while the vibration signal (1512) is being detected. In this case, a part of the section where the vibration signal (1512) is detected and a part of the section where the gesture signal (1514) is detected according to the user's next gesture may overlap. Furthermore, the user may perform the next gesture in advance before the vibration signal (1512) is detected. Even in this case, a part of the section where the vibration signal (1512) is detected and a part of the section where the gesture signal (1514) generated by the user's next gesture is detected may overlap. In this case, when classifying the gesture signal (1514) according to the next gesture, there is a problem that the gesture signal (1514) and the vibration signal (1512) may overlap, and thus the pattern and characteristics of the gesture signal (1514) may be distorted and derived, resulting in an incorrect recognition result for the next gesture.

[0168] Figure 15b illustrates a scenario where a vibration signal detected by vibration due to an alarm and a gesture signal detected by a user performing a gesture overlap.

[0169] An alarm can represent a vibration that occurs at a regular interval. Events that trigger an alarm include, for example, a phone call from an external device, an incoming message, or an application alarm. A scenario in which an alarm can affect gesture recognition performance is described in Figure 15b.

[0170] Referring to FIG. 15B, an electronic device can detect an event corresponding to a vibration alarm (e.g., a call from an external device) and generate an alarm in response to the event detection. For example, a sensor of the electronic device can detect vibration signals (1521, 1522) generated by vibrations generated by an alarm. A user of the electronic device can recognize (1524) the vibration generated by the electronic device. The user who recognizes the vibration recognizes that an alarm has occurred and can perform a gesture to control the electronic device. The sensor of the electronic device can detect a gesture signal (1523) generated by the user's gesture. The electronic device can recognize the gesture performed by the user and execute a function corresponding to the recognized user's gesture (1525). As illustrated in FIG. 15B, the user can perform a gesture while a vibration signal (1522) is detected. In this case, a portion of the section where the vibration signal (1522) is detected and a portion of the section where the gesture signal (1523) is detected according to the user's gesture performance may overlap. In this case, there is a problem that the gesture signal (1523) and the vibration signal (1522) overlap, causing the pattern and characteristics of the gesture signal (1523) to be distorted and derived, resulting in incorrect recognition results for the gesture.

[0171] An electronic device according to one embodiment may provide a technique for minimizing the influence of a vibration signal in a situation where vibration signals and gesture signals may overlap. An electronic device according to one embodiment proposes a method for maintaining a high recognition rate for a gesture signal even when the gesture signal is influenced by the vibration signal.

[0172]

[0173] FIG. 16 is a diagram illustrating a process in which an electronic device according to one embodiment calculates a vibration section based on vibration information according to vibration.

[0174] In one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2A, the electronic device (300) of FIG. 3, or the electronic device (1401) of FIG. 14) can identify vibration information based on vibration. The electronic device can suppress or reduce the signal size of a vibration section determined based on the vibration information in a first signal acquired from a first sensor. The electronic device can recognize a first gesture based on a processed first signal (e.g., a signal whose signal size of a vibration section is suppressed or reduced) among a plurality of signals acquired from a plurality of sensors and a second signal acquired from a second sensor. Hereinafter, vibration information based on vibration will be described in detail.

[0175] In one embodiment, the electronic device may generate vibrations in response to vibration feedback or in response to an alarm for an event occurring in the electronic device.

[0176] In one embodiment, when an electronic device generates vibrations in response to vibration feedback, the vibration information identified by the electronic device may be information related to vibrations corresponding to a second gesture that is recognized temporally earlier than the first gesture. When an electronic device generates vibrations in response to an alarm for an event, the vibration information identified by the electronic device may be information related to the alarm for the event.

[0177] In one embodiment, an electronic device may include an actuator for generating vibration. The vibration information may be a control signal applied to the actuator. The electronic device may apply the control signal to the actuator to generate vibration, and may estimate the vibration period by checking the applied control signal. For example, the control signal may include at least one of a vibration pattern, a vibration time, a vibration length, a vibration period, and a vibration magnitude.

[0178] In one embodiment, the vibration information may be information identified by the processor for generating vibration. For example, if the electronic device generates vibration in response to vibration feedback, information regarding the vibration to be generated in response to recognition of a second gesture may be loaded from memory or an application. For another example, if the electronic device generates vibration in response to an alarm, information regarding the vibration to be generated in response to the alarm may be loaded from memory or an application. For example, the loaded information may include at least one of a vibration pattern, a vibration time, a vibration length, a vibration period, and a vibration magnitude.

[0179] In one embodiment, the electronic device can calculate a vibration interval based on the identified vibration information. For example, the electronic device can calculate the vibration interval from at least one of the vibration pattern, vibration period, vibration duration, vibration length, and vibration time included in the identified vibration information. Below, the operation of the electronic device calculating the vibration interval based on the identified vibration information is described in more detail.

[0180] FIG. 16 illustrates sensor signals that appear when vibration in an electronic device and a user's gesture occur at intervals from each other. Graph (1601) is a graph representing a first signal value (e.g., linear acceleration value) at each point in time measured by a first sensor (e.g., linear acceleration sensor) of the electronic device. Graph (1602) is a graph representing a second signal value (e.g., angular velocity value) at each point in time measured by a second sensor (e.g., gyro sensor) of the electronic device. Section (1610) is a section in which a gesture signal caused by a user's gesture performance mainly appears. Section (1611) is a section in which a vibration signal caused by a vibration generated in the electronic device mainly appears.

[0181] In one embodiment, the electronic device can calculate a vibration range based on a first signal obtained (before processing) from a first sensor and a second signal obtained from a second sensor.

[0182] More specifically, the electronic device can calculate the time period during which the second signal is less than the second threshold value within the time period during which the first signal exceeds the first threshold value as the vibration period. Referring to FIG. 16, when vibration occurs in the electronic device, in the period (1611) during which the vibration signal mainly appears, the first signal value (e.g., linear acceleration value) at each point in time may appear to exceed the first threshold value, but the second signal value (e.g., angular velocity value) at each point in time may appear to be less than the second threshold value. This is because the vibration frequency of the vibration that the first sensor can detect and the vibration frequency of the vibration that the second sensor can detect are different. For example, when vibration occurs in the electronic device, a change in linear acceleration due to the vibration may be detected, but a change in angular velocity may not be detected. For example, the first threshold value may be 60 cm / s. 2 The second threshold may be, but is not limited to, 10 rad / sec.

[0183] In summary, in cases where the vibration of the electronic device and the user's gesture occur at intervals, as in FIG. 16, the time interval during which the second signal is below the second threshold within the interval during which the first signal exceeds the first threshold can be calculated as the vibration interval. However, in cases where the vibration of the electronic device and the user's gesture occur adjacent to each other, the vibration signal and the gesture signal overlap at least partially, making the boundary between the vibration signal and the gesture signal unclear, so it is necessary to calculate the vibration interval more precisely.

[0184]

[0185] FIG. 17 illustrates sensor signals that appear when vibration in an electronic device and a user's gesture occur adjacent to each other. Graph (1701) is a graph representing a first signal value (e.g., a linear acceleration value) at each point in time measured by a first sensor (e.g., a linear acceleration sensor) of the electronic device. Graph (1702) is a graph representing a second signal value (e.g., an angular velocity value) at each point in time measured by a second sensor (e.g., a gyro sensor) of the electronic device. Section (1711) is a section in which a gesture signal caused by a user's gesture performance and a vibration signal caused by vibration generation in the electronic device appear together. That is, in section (1711), a point in time or time section in which the gesture signal and the vibration signal overlap may exist. The electronic device needs to detect the vibration section more precisely because the boundary between the vibration signal and the gesture signal is unclear within section (1711). This is because the more gesture signals are included in the vibration section, the more the gesture signal may be distorted when processing the vibration signal. Hereinafter, FIGS. 20 and 21 describe in more detail the process by which an electronic device calculates a boundary point between a vibration signal and a gesture signal when the vibration signal and the gesture signal overlap.

[0186]

[0187] FIG. 18 is a flowchart illustrating a process in which an electronic device according to one embodiment sets an expected vibration section based on vibration information and calculates a vibration section within the expected vibration section.

[0188] In one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2A, electronic device (300) of FIG. 3, or electronic device (1401) of FIG. 14) may calculate a vibration section within the entire time interval, but calculating a vibration section within the entire time interval may be inefficient in terms of data processing. Therefore, the electronic device may first calculate an expected vibration section in which vibration is expected to occur within the entire time interval, and may calculate a vibration section within the calculated expected vibration section. In this case, the electronic device may be efficient in terms of data processing because it calculates a vibration section within the expected vibration section rather than calculating a vibration section within the entire time interval. Hereinafter, a method of calculating an expected vibration section and calculating a vibration section from the expected vibration section will be described. The following operations may be operations that specify operation (1320) of FIG. 13.

[0189] In operation (1821), the electronic device can calculate an expected vibration range based on the identified vibration information.

[0190] In one embodiment, the electronic device may recognize the user's second gesture and generate vibration based on the vibration feedback. In this case, the electronic device may identify information related to the vibration corresponding to the recognized second gesture as vibration information.

[0191] At this time, the vibration information identified by the electronic device may include information about the time required from the time the electronic device recognizes the second gesture to the time the electronic device generates vibration according to the vibration feedback generated in response to the recognition of the second gesture. In addition, the vibration information may include information about the length of the vibration (e.g., the time required for one vibration) according to the vibration feedback generated in response to the recognition of the second gesture. For example, the vibration information may include information that it takes 1 second from the time the user's second gesture is recognized to the time the vibration according to the vibration feedback is generated, and may include information that the length of the vibration according to the vibration feedback generated in response to the recognition of the second gesture is 0.5 seconds.

[0192] In one embodiment, if the identified vibration information is information related to vibration corresponding to a recognized second gesture, the electronic device may calculate a preset period of time from the time the second gesture is recognized as an expected vibration period. Here, the electronic device may set the preset period of time as a time (e.g., 1.5 seconds) obtained by adding the vibration length (e.g., 0.5 seconds) to the time (e.g., 1 second) required from the time the second gesture is recognized to the time at which vibration is generated according to the vibration feedback.

[0193] In one embodiment, the electronic device may generate vibrations in response to an alarm about an event occurring on the electronic device. In such cases, the electronic device may identify information related to the alarm about the event as vibration information.

[0194] At this time, the vibration information identified by the electronic device may include information regarding the vibration period and vibration length for the alarm. For example, the vibration information may include information indicating that the vibration period for the alarm is 3 seconds and the vibration length is 0.4 seconds.

[0195] In one embodiment, if the identified vibration information is information related to an alarm for an event, the electronic device may calculate an expected vibration interval using the vibration period and vibration length extracted from the identified vibration information. For example, the vibration period of a vibration alarm may represent the time from the start of a previous vibration to the start of the next vibration. Accordingly, the electronic device may set the interval during the vibration period extracted from the end point of the previous vibration interval as the expected vibration interval. For example, the previous vibration interval may be the interval from point A to point B, and the vibration period may be 3 seconds. The electronic device may set the interval from point B, which is the end point of the previous vibration interval, to the vibration period of 3 seconds as the expected vibration interval. As another example, the electronic device may identify vibration information that is changed by the user in real time. The electronic device identifying vibration information based on a user input will be described in detail below with reference to FIG. 19.

[0196] In operation (1822), the electronic device can calculate a vibration range using sensor signals within a set expected vibration range. As described above, directly calculating the vibration range within the entire signal range can be inefficient in terms of data processing due to the relatively large amount of data to be processed. Therefore, the electronic device can efficiently process data by first calculating the expected vibration range within the entire signal range, and then calculating the vibration range within the expected vibration range.

[0197] Referring to graphs (1601) and (1602) of FIG. 16, vibrations in an electronic device and a user's gesture may occur at intervals. The electronic device may first calculate an expected vibration period and then calculate a vibration period from the expected vibration period. The electronic device may calculate a time period within the expected vibration period in which a first signal acquired from a first sensor exceeds a first threshold value, and may calculate a time period within the calculated time period in which a second signal acquired from a second sensor is less than a second threshold value as a vibration period.

[0198] The following Figures 20 and 21 describe a method in which an electronic device first calculates an expected vibration section and then calculates a vibration section from the expected vibration section when a vibration in an electronic device and a user's gesture occur adjacent to each other.

[0199] In operation (1823), the electronic device may perform processing to suppress or reduce the signal size of the detected vibration section.

[0200]

[0201] FIG. 19 is a diagram illustrating a process for changing the time required from the time an electronic device recognizes a gesture based on a user input to the time it generates vibration according to the recognition of the gesture, according to one embodiment.

[0202] In one embodiment, an electronic device (1901) (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2A, the electronic device (300) of FIG. 3, or the electronic device (1401) of FIG. 14) may provide information (1910) for setting vibration information to a user. For example, the electronic device (1901) may output information (1910) such as 'Set the vibration generation time for the gesture recognition enhancement algorithm to operate when vibration occurs during gesture recognition' on a display (e.g., the display (220) of FIG. 3). After outputting the information (1910), the electronic device (1901) may provide the user with a user interface (UI) (hereinafter, UI) (1920) through which the user can change the vibration information setting. For example, the electronic device (1901) may provide a plurality of items (1921) that can change the vibration generation time through the UI (1920) to change the start time of the haptic vibration that occurs to notify the user that the electronic device (1901) has recognized the user's gesture. For example, the user may set the time for vibration generation from the gesture recognition time to one of 0.5 seconds, 1 second, 1.5 seconds, or 2 seconds among the plurality of items (1921) provided by the electronic device (1901). The electronic device (1901) may identify the vibration information based on the change in vibration information input by the user (e.g., changing the vibration generation time to 1 second after gesture recognition). At this time, the electronic device (1901) may provide the user with information (1930) about the vibration information changed by the user. If the user determines that the electronic device (1901) does not recognize the gesture well based on the changed vibration information, the user can reset the vibration information based on the setting change UI (1931).

[0203]

[0204] FIGS. 20 and 21 are diagrams illustrating a process in which an electronic device according to one embodiment calculates a boundary point of a vibration signal and a gesture signal within an expected vibration range.

[0205] Figures 20 and 21 are drawings explaining a process in which an electronic device calculates boundary points of a vibration signal and a gesture signal within an expected vibration range.

[0206] In one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2A, electronic device (300) of FIG. 3, electronic device (1401) of FIG. 14, or electronic device (1901) of FIG. 19) can calculate a vibration section within an expected vibration section. To calculate the vibration section, the electronic device can calculate a boundary point between a vibration signal and a gesture signal within the expected vibration section. Hereinafter, an operation (e.g., operation (1822) of FIG. 18) in which the electronic device calculates the vibration section using a sensor signal within the expected vibration section will be described in more detail.

[0207] First, in operation (2021) of FIG. 20, the electronic device can search for a reference point in time at which the first signal value (e.g., linear acceleration value) is at its maximum within the expected vibration section (2131).

[0208] Referring to FIG. 21, a graph (2101) is a graph representing a first signal value (e.g., a linear acceleration value) at each point in time measured by a first sensor (e.g., a linear acceleration sensor) of an electronic device. A graph (2102) is a graph representing a second signal value (e.g., an angular velocity value) at each point in time measured by a second sensor (e.g., a gyro sensor) of an electronic device. Within the expected vibration section (2131), a gesture signal caused by a user's gesture performance and a vibration signal caused by vibration generation in the electronic device may overlap and appear.

[0209] In one embodiment, the electronic device can calculate a detailed vibration search section (2132) within the expected vibration section (2131). The electronic device can calculate a section within the expected vibration section (2131) in which a first signal value (e.g., a linear acceleration value) measured by a first sensor (e.g., a linear acceleration sensor) exceeds a first threshold value and a second signal value (e.g., an angular velocity value) measured by a second sensor (e.g., a gyro sensor) is less than a second threshold value as the detailed vibration search section (2132). The electronic device can search for a reference time point (2141) in which the first signal value (e.g., a linear acceleration value) is at a maximum within the detailed vibration search section (2132).

[0210] In operation (2022), the electronic device may determine one of the time points at which the first signal value appears below the first ratio with respect to the first signal value at the searched reference time point (2141), as the interval minimum time point.

[0211] Referring to FIG. 21, since the first signal value (e.g., linear acceleration value) at the reference point (2141) is maximum within the expected vibration section (2131), the reference point (2141) may represent a peak point of the vibration signal. The electronic device may determine the section minimum point, which is a boundary point between the vibration signal and the gesture signal, through unit tracking based on the reference point (2141).

[0212] In one embodiment, the electronic device is configured to: (1) measure a first ratio (e.g., a linear acceleration value) to a first signal value (e.g., a linear acceleration value) at a reference point (2141); ) can be determined as a candidate minimum point in time at which a first signal value (e.g., linear acceleration value) less than 1 appears. The first ratio ( ) can have a value greater than 0 and less than 1. For example, the first ratio can be 0.5, but is not limited thereto.

[0213] In one embodiment, the electronic device can determine the interval minimum point by repeatedly updating the candidate minimum point. In other words, the electronic device can determine the last updated candidate minimum point as the interval minimum point.

[0214] In one embodiment, the electronic device can determine whether there is another point in time after the first point in time (2161), which is the candidate minimum point in time, at which a first signal value (e.g., a linear acceleration value) at the first point in time (2161) appears that is less than the first signal value. The electronic device can determine whether there is another point in time in a chronological order.

[0215] In one embodiment, the electronic device determines whether there is another point in time at which a first signal value appears that is less than a first signal value of a first point in time (2161), which is a candidate minimum point in time, and searches for a second point in time (2162) at which a first signal value appears that is less than a first signal value of the first point in time (2161), and before the second point in time (2162) is searched, determines a second ratio ( ) If the first signal value above does not appear, the second point in time (2162) can be updated to the candidate minimum point in time. Here, the second ratio ( ) may have a value exceeding 1. For example, the second ratio may be 2, but is not limited thereto. In the same manner as above, the electronic device may repeatedly update the candidate minimum point in time. For example, the electronic device may determine whether there is another point in time after the second point in time (2162), which is the candidate minimum point in time, at which a first signal value smaller than the first signal value of the second point in time (2162) appears. During the operation of determining whether there is another point in time at which a first signal value smaller than the first signal value of the second point in time (2162), which is the updated candidate minimum point in time, the electronic device searches for a third point in time (2163) at which a first signal value smaller than the first signal value of the second point in time (2162) appears, and before the third point in time (2163) is searched, the second ratio ( ) If no linear acceleration value above appears, the third time point (2163) can be updated as the candidate minimum time point.

[0216] In one embodiment, the electronic device can determine whether there is another point in time after the third point in time (2163), which is the updated candidate minimum point in time, at which a first signal value appears that is less than the first signal value of the third point in time (2163). In one embodiment, the electronic device can determine whether there is another point in time at which a first signal value appears that is less than the first signal value of the third point in time (2163), which is the updated candidate minimum point in time, before the other point in time is searched, if the second ratio of the first signal value of the third point in time (2163), which is the candidate minimum point in time, is greater than or equal to the second ratio of the first signal value of the third point in time (2163), which is the candidate minimum point in time. ) If the first signal value above appears, the third time point (2163), which is the candidate minimum time point, can be determined as the interval minimum time point.

[0217] In the action (2023), the electronic device can calculate the determined interval minimum point as the boundary point of the vibration signal and the gesture signal.

[0218] For example, the electronic device can calculate the third time point (2163), which is the determined minimum time point of the interval, as the boundary time point between the vibration signal and the gesture signal. The electronic device can calculate the interval (2133) from the time point (2151) at which the first signal value exceeds the first threshold value within the expected vibration interval (2131) to the interval minimum time point (e.g., the third time point (2163)), which is the boundary time point between the vibration signal and the gesture signal, as the vibration interval.

[0219] The point in time (2151) when the first signal value exceeds the first threshold value may be the same as the starting point of the detailed vibration search section (2132). In other words, the electronic device may calculate the section (2133) from the starting point of the detailed vibration search section (2132) to the minimum point in time of the section (e.g., the third point in time (2163)) as the vibration section.

[0220] Since the boundary point between the vibration signal and the gesture signal determined in the above manner is calculated and the section (2133) is calculated as the vibration section, the section in which the gesture signal is determined to be valid can be preserved within the detailed vibration search section (2132). If the electronic device does not perform the operation of calculating the boundary point (2163) within the detailed vibration search section (2132), downscaling may be performed on the signal included in the detailed vibration search section (2132). In this case, since the section in which the gesture signal is valid is included within the detailed vibration search section (2132), a part of the gesture signal may also be downscaled, which may cause distortion of the gesture signal. In other words, the electronic device calculates the vibration section (e.g., section (2133)) through the boundary point (2163) between the vibration signal and the gesture signal, and downscales the signal included in the vibration section (e.g., section (2133)), thereby minimizing distortion of the gesture signal and effectively processing the vibration signal. Below, the process by which the electronic device downscales the signal included in the vibration section (2133) is described in more detail.

[0221] In one embodiment, the electronic device may downscale the magnitude of a signal contained in a generated vibration segment (e.g., segment (2133)) by a preset ratio or preset value. For example, the preset ratio may be 10%, and the preset value may be 3, but is not limited thereto.

[0222] In one embodiment, the electronic device may downscale only sensor signals in which a vibration signal appears among sensor signals generated from a plurality of sensors. More specifically, depending on the vibration frequency of the vibration, the vibration signal may not appear in a second signal acquired from a second sensor (e.g., an angular velocity signal of a gyro sensor or a PPG signal of a PPG sensor) but may appear in a first signal acquired from a first sensor (e.g., a linear acceleration signal of a linear acceleration sensor). Accordingly, the electronic device may apply downscaling only to the first signal (e.g., a linear acceleration signal) generated by the first sensor (e.g., a linear acceleration sensor).

[0223] In another embodiment, the electronic device may remove a signal included in a vibration section (e.g., section 2133). In other words, the electronic device may perform nulling, which changes the magnitude of a signal included in a vibration section (e.g., section 2133) to 0. The nulling method may be effective in processing vibration signals because it completely removes the signal included in a vibration section (e.g., section 2133) where vibration signals are mainly included. However, nulling may cause side effects because it may discontinuously make the signal and the nulled section itself may be viewed as a feature of the signal. More specifically, the electronic device may perform a fast Fourier transform (FFT) on the sensor signal to extract Fourier coefficients in order to extract features in the frequency domain. When extracting the Fourier coefficients, the nulled section may be determined as a discontinuous section, which may distort the features of the sensor signal. Additionally, zero padding due to nulled intervals can be judged as the same feature for different gesture signals, leading to high similarity between different gesture signals. Due to this side effect, downscaling methods that reduce the influence of sensor signals without discontinuities are preferred over nulling.

[0224]

[0225] FIG. 22 is a drawing schematically illustrating the structure of a gesture recognition module of an electronic device according to one embodiment.

[0226] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2A, the electronic device (300) of FIG. 3, the electronic device (1401) of FIG. 14, or the electronic device (1901) of FIG. 19) can recognize a user's gesture through a gesture recognition module (2210) (e.g., the gesture recognition module (1410) of FIG. 5). The gesture recognition module (2210) can receive a sensor signal from a sensor module (2220). The gesture recognition module (2210) can process a vibration signal included in the received sensor signal.

[0227] In one embodiment, the gesture recognition module (2210) of the electronic device may identify vibration information. For example, the electronic device may identify one of the following as vibration information: information related to a vibration corresponding to a second gesture recognized before the first gesture, or information related to an alarm for an event occurring in the electronic device.

[0228] The gesture recognition module (2210) can load vibration information from the vibration information storage (2250). The vibration information storage (2250) can receive vibration information from a service (2260) that stores vibration information about vibration to be generated in response to a gesture or vibration information about an alarm. For example, the vibration information storage (2250) can connect to communication with the service (2260) when a second gesture is recognized to receive vibration information about vibration to be generated in response to the second gesture, or can receive and store vibration information about vibration to be generated in response to the second gesture from the service (2260) in advance and then load vibration information about vibration to be generated in response to the second gesture. As another example, the vibration information storage (2250) can connect to communication with the service (2260) when an event that generates an alarm occurs to receive vibration information about vibration according to an alarm to be generated, or can receive and store vibration information about vibration according to an alarm to be generated from the service (2260) in advance and then load vibration information about vibration according to an alarm to be generated.

[0229] In one embodiment, the timing at which the gesture recognition module (2210) of the electronic device processes the vibration signal may vary.

[0230] For example, as illustrated in FIG. 22, the gesture recognition module (2210) of the electronic device may perform a vibration signal processing operation (2230) after performing a preprocessing operation (2211) and a segmentation operation (2212). In other words, the gesture recognition module (2210) of the electronic device may perform a preprocessing operation (2211) for preprocessing a sensor signal received from a sensor module (2220), and a segmentation operation (2212) for generating a cropped time interval for processing a vibration signal by cropping a valid signal interval from the preprocessed sensor signal. The gesture recognition module (2210) of the electronic device may calculate a vibration interval for the cropped time interval, and perform a vibration signal processing operation (2230) for downscaling a signal included in the calculated vibration interval. In addition, the electronic device can perform a feature extraction operation (2213) for extracting multiple features from a signal section in which a vibration signal is processed, and a classification operation (2214) for classifying a gesture signal using the multiple extracted features.

[0231] For another example, the gesture recognition module (2210) of the electronic device may perform a vibration signal processing operation (2230) before performing the preprocessing operation (2211). For another example, the gesture recognition module (2210) of the electronic device may perform the vibration signal processing operation (2230) between the preprocessing operation (2211) and the segmentation operation (2212).

[0232] Table 1 below describes the advantages and disadvantages of the vibration signal processing operation (2230) depending on the timing of execution. Since the preprocessing operation (2211) performs a function of removing signals corresponding to the user's movements other than the user's gestures, it may be effective to perform the vibration signal processing operation (2230) after the preprocessing operation (2211) in order to accurately perform the classification of gesture signals in a situation where the user's movements are present. In addition, since it is unnecessary to continuously process vibration signals in a section where no gesture signal appears, it may be more efficient to perform the vibration signal processing operation (2230) after extracting a valid signal section through the segmentation operation (2212).

[0233] When the vibration signal processing operation is performed after the segmentation operationWhen the vibration signal processing operation is performed between the segmentation operation and the preprocessing operationWhen the vibration signal processing operation is performed before the preprocessing operationAdvantagesHigh accuracy of processing vibration signals in situations where there is user movement other than user gesturesEfficient in terms of operation because the vibration signal processing operation is performed only when a valid signal section is extractedHigh accuracy of processing vibration signals in situations where there is user movement other than user gesturesProcessing of vibration signals included in existing sensor signals is possibleDisadvantagesWhen performing the preprocessing operation, the vibration signal is dispersed, making it difficult to completely remove the vibration signalWhen performing the preprocessing operation, the vibration signal is dispersed, making it difficult to completely remove the vibration signalThe accuracy of processing vibration signals may be lowered by user movement other than gestures

[0234]

[0235] FIG. 23 is a diagram illustrating a process in which an electronic device, according to one embodiment, passively processes a vibration signal to prevent misclassification.

[0236] Even if the vibration signal processing operation (2330) suppresses the signal in the section where the vibration signal is dominant, distortion of the gesture signal is unavoidable because the vibration signal is not completely separated and removed from the gesture signal. This distortion of the gesture signal may result in worse results than when the gesture signal is classified without processing the vibration signal. In other words, if the gesture signal was not processed, the characteristics of the gesture signal may be further distorted due to the processing of the vibration signal, resulting in incorrect classification of the gesture. To solve this problem, a passive vibration signal processing method is proposed.

[0237] In one embodiment, the electronic device may perform classification on a gesture signal included in a valid signal section without calculating a vibration section. Here, the valid signal section may represent a signal section obtained by performing a preprocessing operation (2211) and a segmentation operation (2212) on a sensor signal received from a sensor module (2220). The electronic device may not perform a vibration signal processing operation (2230) on the valid signal section. The electronic device may determine whether the first classification result for the gesture signal in the valid signal section is classified as a preset gesture. Here, the preset gesture may be, for example, an OCO gesture or a pinch gesture.

[0238] In one embodiment, if the first classification result for a gesture signal in a valid signal section for which the vibration signal processing operation (2330) has not been performed is not classified as a preset gesture, the electronic device may re-classify the gesture signal by performing the vibration signal processing operation (2330). More specifically, if the first classification result is not classified as a preset gesture, the electronic device may re-classify the gesture signal in the valid signal section based on performing the vibration signal processing operation (2330) of calculating a vibration section in the valid signal section and downscaling a signal included in the calculated vibration section. The electronic device may calculate the final classification result using the second classification result for the gesture signal in the valid signal section. For example, the electronic device may calculate the second classification result as the final classification result. As another example, the electronic device may combine the first classification result and the second classification result to calculate the final classification result.

[0239] The passive vibration signal processing method is to not process the vibration signal if the gesture signal is classified normally without processing the vibration signal, and to reclassify the gesture signal only if the gesture signal is not classified normally after processing the vibration signal. This passive vibration signal processing method can prevent misclassification problems caused by vibration signal processing.

[0240]

[0241] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0242] Software may include computer programs, codes, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, or computer storage medium or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0243] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0244] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Claims

1. In electronic devices (101; 200; 300; 1401; 1901), An actuator (410; 420) for generating vibration; A plurality of sensors including a first sensor and a second sensor for recognizing a gesture of a user of the electronic device; Memory (130) for storing instructions; and One or more processors (120) operatively connected to the actuator (410; 420), the plurality of sensors and the memory (130), and including processing circuitry Including, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 200; 300; 1401; 1901) causes: Based on the fact that the first frequency (640) at which vibration occurs in the first signal acquired from the first sensor matches the vibration frequency range corresponding to the actuator (410; 420), a process is performed to suppress or reduce the component of the first frequency (640) in the first signal, Recognizing a first gesture based on at least the processed first signal and the second signal acquired from the second sensor among the plurality of signals acquired from the plurality of sensors. Electronic devices (101; 200; 300; 1401; 1901).

2. In paragraph 1, If the above first sensor corresponds to a three-axis angular velocity sensor, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 200; 300; 1401; 1901) causes: In the first-axis angular velocity signal obtained from the first-axis angular velocity sensor among the three-axis angular velocity sensors, the frequency component corresponding to the first frequency (640) is less than a predetermined threshold value, If the frequency components corresponding to the first frequency (640) in each of the second-axis angular velocity signal acquired from the second-axis angular velocity sensor and the third-axis angular velocity signal acquired from the third-axis angular velocity sensor are greater than a predetermined threshold value, It is determined that vibration has occurred in the above electronic device (101; 200; 300; 1401; 1901). Electronic devices (101; 200; 300; 1401; 1901).

3. In paragraph 2, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 200; 300; 1401; 1901) causes: The frequency corresponding to the maximum value among the frequency components in the frequency domain of the first-axis angular velocity signal is stored as the first-axis candidate vibration frequency. The frequency corresponding to the maximum value among the frequency components in the frequency domain of the second-axis angular velocity signal is stored as the second-axis candidate vibration frequency. The frequency corresponding to the maximum value among the frequency components in the frequency domain of the third-axis angular velocity signal is stored as the third-axis candidate vibration frequency. Based on the comparison results between the first axis candidate vibration frequency, the second axis candidate vibration frequency and the third axis candidate vibration frequency respectively and the vibration frequency range corresponding to the actuator (410; 420), processing is performed to suppress or reduce the first signal. Electronic devices (101; 200; 300; 1401; 1901).

4. In any one of paragraphs 1 to 3, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 200; 300; 1401; 1901) causes: A process is performed to suppress or reduce the first signal by filtering out frequency components in a frequency range lower than the first frequency (640) among the frequency components of the first signal and filtering out frequency components in a frequency range higher than the first frequency (640). Electronic devices (101; 200; 300; 1401; 1901).

5. In any one of paragraphs 1 to 4, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 200; 300; 1401; 1901) causes: Extracting an envelope signal (830) corresponding to the second signal, and performing processing to suppress or reduce the first signal based on the extracted envelope signal (830). Electronic devices (101; 200; 300; 1401; 1901).

6. In paragraph 5, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 200; 300; 1401; 1901) causes: By applying the extracted envelope signal (830) to the first signal, a process of suppressing or reducing the first signal is performed. Electronic devices (101; 200; 300; 1401; 1901).

7. In any one of paragraphs 5 to 6, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 200; 300; 1401; 1901) causes: Smoothing the above extracted envelope signal (830), Electronic devices (101; 200; 300; 1401; 1901).

8. In any one of paragraphs 5 to 7, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 200; 300; 1401; 1901) causes: Extracting another envelope signal corresponding to the first signal, and generating a combined envelope signal by combining the extracted envelope signal (830) and the other envelope signal, Based on the generated combined envelope signal, processing is performed to suppress or reduce the first signal and the second signal, respectively. Electronic devices (101; 200; 300; 1401; 1901).

9. In paragraph 8, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 200; 300; 1401; 1901) causes: By applying the combined envelope signal to each of the first signal and the second signal, processing is performed to suppress or reduce each of the first signal and the second signal. Electronic devices (101; 200; 300; 1401; 1901).

10. In paragraph 1, When the above instructions are individually or collectively executed by the one or more processors (120), the electronic device (101; 200; 300; 1401; 1901) causes: Identifying vibration information related to vibration of the above electronic device (101; 200; 300; 1401; 1901), When the first sensor corresponds to a three-axis acceleration sensor, processing is performed to suppress or reduce the data size of the vibration section (2133) determined based on the vibration information in the first signal. Recognizing a first gesture based on the processed first signal and the second signal among the plurality of signals acquired from the plurality of sensors. Electronic devices (101; 200; 300; 1401; 1901).

11. In paragraph 10, The above vibration information is, One of the information related to vibration temporarily generated by a second gesture recognized before the first gesture recognized above or information related to vibration periodically generated by an event occurring in the electronic device, Electronic devices (101; 200; 300; 1401; 1901).

12. In any one of paragraphs 10 to 11, The above vibration information A control signal applied to the actuator (410; 420) or information identified by the one or more processors for generation of the vibration, Electronic devices (101; 200; 300; 1401; 1901).

13. In any one of paragraphs 10 to 12, The above vibration section (2133) is At least one of the vibration pattern, vibration period, vibration length, and vibration time included in the identified vibration information is derived. Electronic devices (101; 200; 300; 1401; 1901).

14. In any one of paragraphs 10 to 13, The above vibration section (2133) is A section calculated based on the first signal and the second signal, Electronic devices (101; 200; 300; 1401; 1901).

15. A method performed by a processor (120) of an electronic device (101; 200; 300; 1401; 1901), An operation of performing a process to suppress or reduce a component of the first frequency (640) in the first signal based on the first frequency (640) at which vibration occurs in the first signal acquired from the first sensor matching the vibration frequency range corresponding to the actuator (410; 420); and An operation of recognizing a first gesture based on at least the processed first signal and the second signal acquired from the second sensor among a plurality of signals acquired from a plurality of sensors including the first sensor and the second sensor. A method including:

Images