Electronic device, and method for biometric signal pattern recognition using artificial intelligence model in electronic device

Abstract

The electronic device of the present disclosure comprises: a biometric sensor; a memory for storing instructions; and at least one processor operatively connected to the biometric sensor and the memory, wherein, when executed by the at least one processor, the instructions allow the electronic device to: acquire a biometric signal frame of a designated frame size from a biometric signal received by the biometric sensor; identify support sets to be compared to the biometric signal frame from among a plurality of support sets including biometric signal frame samples corresponding to respective body reactions; use an artificial intelligence model to acquire a vector value including features of a plurality of domain-specific biometric signals for the biometric signal frame; identify a support set including a biometric signal frame sample having the vector value that is most similar to the vector value of the biometric signal frame from among the identified support sets; and identify, on the basis of the identified support set, the body reaction corresponding to the received biometric signal.

Description

Method for recognizing biosignal patterns using artificial intelligence models in electronic devices and electronic devices

[0001] Various embodiments of the present disclosure relate to electronic devices and methods for recognizing biosignal patterns using artificial intelligence models in electronic devices.

[0002] As interest in healthcare has increased recently, methods for recognizing specific patterns in biosignals measured through wearable devices (e.g., earbuds, watches) or other biosignal measuring devices (e.g., medical devices) are becoming increasingly important.

[0003] Typically, when pattern recognition of biosignals is performed, when time series biosignal data is input, it is determined whether pattern recognition of the input biosignal data is possible, and if pattern recognition is possible, the pattern of the biosignal can be recognized.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0005] Conventional biosignal pattern recognition methods can recognize biosignal patterns, but cannot provide information about what type of biosignal pattern the recognized pattern is.

[0006] It may be necessary to utilize various bio-signals collected in real time to extract various patterns of bio-signals according to the body's situation and / or changes, and to identify bio-signal patterns that are appropriate for each of the body's situations and / or changes from the extracted patterns.

[0007] According to an embodiment of the present disclosure, an electronic device may include a biosensor, a memory storing instructions, and at least one processor operatively connected to the biosensor and the memory. The instructions according to an embodiment, when executed by the at least one processor, may cause the electronic device to obtain a biosignal frame of a specified frame size from a biosignal received by the biosensor. The instructions according to an embodiment, when executed by the at least one processor, may cause the electronic device to identify support sets to be compared with the biosignal frame among a plurality of support sets including biosignal frame samples corresponding to respective responses of a body. The instructions according to an embodiment, when executed by the at least one processor, may cause the electronic device to obtain a vector value including features of a plurality of domain-specific biosignals for the biosignal frame using an artificial intelligence model. The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to identify a support set including a biosignal frame sample having a vector value most similar to the vector value of the biosignal frame among the identified support sets. The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to identify a bodily response corresponding to the received biosignal based on the identified support set.

[0008] In accordance with one embodiment of the present disclosure, a method for recognizing a biosignal pattern using an artificial intelligence model in an electronic device may include an operation of obtaining a biosignal frame of a specified frame size from a biosignal received by a biosensor. In accordance with one embodiment, the method may include an operation of identifying support sets to be compared with the biosignal frame from among a plurality of support sets including biosignal frame samples corresponding to respective bodily reactions. In accordance with one embodiment, the method may include an operation of obtaining a vector value including features of a plurality of domain-specific biosignals for the biosignal frame using the artificial intelligence model. In accordance with one embodiment, the method may include an operation of identifying a support set including a biosignal frame sample having a vector value most similar to the vector value of the biosignal frame from among the identified support sets. In accordance with one embodiment, the method may include an operation of identifying a bodily reaction corresponding to the received biosignal based on the identified support set.

[0009] In a non-transitory storage medium storing commands according to an embodiment of the present disclosure, the commands, when executed by an electronic device, are configured to cause the electronic device to perform at least one operation, wherein the at least one operation may include an operation of obtaining a biosignal frame of a specified frame size from a biosignal received by a biosensor. The at least one operation according to an embodiment may include an operation of identifying support sets to be compared with the biosignal frame from among a plurality of support sets including biosignal frame samples corresponding to respective bodily reactions. The at least one operation according to an embodiment may include an operation of obtaining a vector value including features of a plurality of domain-specific biosignals for the biosignal frame using the artificial intelligence model. The at least one operation according to an embodiment may include an operation of identifying a support set including a biosignal frame sample having a vector value most similar to the vector value of the biosignal frame from among the identified support sets. The at least one operation according to an embodiment may include an operation of identifying a bodily reaction corresponding to the received biosignal based on the identified support set.

[0010] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0011] Figure 2 is a block diagram of an electronic device according to one embodiment.

[0012] FIG. 3 is a diagram illustrating a processor according to one embodiment.

[0013] FIG. 4 is a flowchart illustrating a biosignal pattern recognition operation in an electronic device according to one embodiment.

[0014] FIG. 5 is a flowchart illustrating a biosignal pattern recognition operation capable of registering a support set and learning an artificial intelligence model in an electronic device according to one embodiment.

[0015] FIG. 6 is a diagram for explaining an operation of recognizing brainwave signals patterns according to one embodiment.

[0016] Figure 7 is a diagram showing examples of brain wave signals according to one embodiment.

[0017] FIG. 8a is a diagram showing a normalized signal of a time domain brainwave signal according to one embodiment.

[0018] FIG. 8b is a diagram showing a normalized signal of an EEG signal in the frequency domain according to one embodiment.

[0019] FIG. 8c is a diagram showing a normalized signal of an EEG signal in the time-frequency domain according to one embodiment.

[0020] FIG. 9 is a diagram illustrating an example of a biosensor and an electronic device according to one embodiment.

[0021] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to an embodiment. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) and the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to an embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0022] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0023] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0024] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0025] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0026] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0027] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0028] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0029] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0030] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0031] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0032] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0033] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0034] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0035] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0036] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0037] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0038] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0039] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0040] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0041] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0042] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0043] In the detailed description below, reference numerals in the drawings may be used interchangeably with or omitted for configurations that can be easily understood through the preceding embodiments, and their detailed descriptions may also be omitted. An electronic device according to an embodiment disclosed in this document may be implemented by selectively combining configurations of various embodiments, and a configuration of one embodiment may be replaced by a configuration of another embodiment. For example, it should be noted that the present disclosure is not limited to a specific drawing or embodiment.

[0044] Figure 2 is a block diagram of an electronic device according to one embodiment.

[0045] Referring to FIG. 2, an electronic device (201) according to an embodiment (e.g., the electronic device (101) of FIG. 1) includes a processor (220), a memory (230), a display (260), and may further include a biometric sensor (276) and a communication circuit (290). The electronic device (201) according to an embodiment is not limited thereto and may further include various components or may be configured by excluding some of the components. The electronic device (201) according to an embodiment may further include all or part of the electronic device (101) illustrated in FIG. 1.

[0046] A processor (220) according to an embodiment (e.g., the processor (120) of FIG. 1 or at least one processor) may include a circuit for processing. The processor (220) according to an embodiment may include some or all of a central processing unit (CPU), an application processor (AP), a microprocessor unit (MPU), a data processing unit (DPU), and a graphics processing unit (GPU). At least one processor according to an embodiment may include a hardware structure (e.g., an AI chip) specialized for processing an artificial intelligence (AI) model. The processor (220) according to an embodiment may perform an overall control operation of the electronic device (201) and may perform a biosignal recognition operation using an artificial intelligence model.

[0047] A processor (220) according to one embodiment may receive a biosignal sensed from a biosensor (e.g., a biosensor (276) of the electronic device (201) or an external biosensor). In the following description of the present disclosure, a case of receiving a biosignal from a biosensor (276) of the electronic device (201) is described as an example, but the biosignal may be received from an external biosensor of the electronic device (201) through communication via a communication circuit (290).

[0048] A processor (220) according to an embodiment may receive a biosignal sensed from a biosensor (276). A processor (220) according to an embodiment may receive a biosignal sensed from a biosensor (276) in real time. A biosignal according to an embodiment may include a time-series (or time-continuous) biosignal measured from a body. For example, the biosignal may include at least one of an electroencephalogram (EEG) signal, an electrocardiogram (ECG) signal, and an electromyography (EMG) signal, and other time-series biosignals measurable from the body may also be possible. A processor (220) according to an embodiment may receive a time-series biosignal, perform noise filtering on it, and obtain a noise-filtered time-series biosignal.

[0049] According to an embodiment, a processor (220) may obtain a biosignal frame of a specified frame size from a biosignal received (or received and noise-filtered) from a biosensor (276). The specified frame size according to an embodiment may correspond to an input frame size of an artificial intelligence model. According to an embodiment, a processor (220) may divide a biosignal received in time series from a biosensor (276) into a specified frame size corresponding to the input frame size of an artificial intelligence model (e.g., every 0.25 s), and obtain biosignal frames of the specified frame size.

[0050] A processor (220) according to an embodiment may identify support sets to be compared with a biosignal frame among a plurality of support sets corresponding to pre-registered biosignal patterns. The biosignal patterns according to an embodiment may refer to biosignal frame samples that are collected (or acquired) in advance corresponding to each of body reactions (or movements). A processor (220) according to an embodiment may identify a variance value of an amplitude of a biosignal frame, a threshold value of a maximum value of an amplitude of a biosignal frame, or a threshold value of a ratio of a specific frequency of a biosignal frame using a designated algorithm (e.g., a statistical or empirical algorithm), and may identify support sets that satisfy the variance value of an amplitude of a biosignal frame, the threshold value of a maximum value of an amplitude of a biosignal frame, or the threshold value of a ratio of a specific frequency of a biosignal frame among a plurality of support sets. The bodily reactions (or movements) according to one embodiment include stillness (default), chewing food (munch), clenching teeth (clenching), blinking, winking the left eye (wink_left), and / or winking the right eye (wink_right), and may further include other bodily reactions (or movements) of the person.According to an embodiment, a plurality of support sets corresponding to bio-signal patterns of pre-registered body reactions (or movements) may include a support set corresponding to an electroencephalogram (or brainwave) signal pattern when still, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when chewing food, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when clenching teeth, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when blinking eyes, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when winking a left eye, and / or a support set corresponding to an electroencephalogram (or brainwave) signal pattern when winking a right eye, and may further include a support set corresponding to another bio-signal pattern. Each of the plurality of support sets according to an embodiment may include k (e.g., a natural number, or 2 to 3) classes for few-shot training (or few-shot learning), and each class may include a plurality of sample bio-signal frames to be compared with a bio-signal frame. For example, a support set corresponding to an EEG (or brain wave) signal pattern during an eye blink may include k classes (e.g., a natural number, or 2 to 3), and each class may include sample biosignal frames corresponding to the EEG signal pattern during an eye blink to be compared with the biosignal frame.

[0051] According to an embodiment, a processor (220) may acquire a plurality of domain-specific biosignals for a biosignal frame and, using a learned artificial intelligence model, may acquire a vector value corresponding to a biosignal frame including features of a plurality of domain-specific biosignals for the biosignal frame. According to an embodiment, the processor (220) may acquire a first biosignal in a time domain corresponding to the biosignal frame and may acquire a normalized first biosignal obtained by normalizing the first biosignal in the time domain. According to an embodiment, the processor (220) may acquire a second biosignal in a frequency domain corresponding to the biosignal frame and may acquire a normalized second biosignal obtained by normalizing the second biosignal in the frequency domain. According to an embodiment, a processor (220) may obtain a third biosignal in a time-frequency domain including two-dimensional values ​​(2Dimension values) corresponding to the biosignal frame by applying a short time Fourier transform (STFT) to the biosignal frame, and may obtain third biosignals of a normalized third channel by normalizing the third biosignal in the time-frequency domain by row and column. According to an embodiment, the processor (220) may apply (or input) the normalized first biosignal, the normalized second biosignal, and the normalized three-channel third biosignals to a pre-trained artificial intelligence model, and may obtain a vector value corresponding to the biosignal frame through the artificial intelligence model. According to an embodiment, the artificial intelligence model may be an artificial intelligence model trained to interpret the normalized time-domain biosignal, the normalized frequency-domain biosignal, and the normalized three-channel biosignals of the time-frequency domain, respectively, and represent a vector value corresponding to the biosignal frame including features of biosignals of a plurality of domains.An artificial intelligence model according to an embodiment may include a convolution neutral network (CNN) model, a recurrent neutral network (RNN) model, or an attention mechanism model. A processor (220) according to an embodiment may identify whether or not the artificial intelligence model has been trained to obtain a vector value corresponding to a biosignal frame, and if the artificial intelligence model has not been trained to obtain a vector value corresponding to a biosignal frame, may perform a learning operation to obtain a vector value corresponding to the biosignal frame.

[0052] In one embodiment, the processor (220) may identify a support set including a sample biosignal frame having a vector value most similar to a vector value corresponding to a biosignal frame among support sets. In one embodiment, the processor (220) may use a function for calculating similarity to identify a support set including a biosignal frame sample having a vector value most similar to a vector value corresponding to a biosignal frame among support sets. For example, the function for calculating similarity may include cosine similarity.

[0053] According to one embodiment, the processor (220) can identify (or determine) a bodily response (or action) corresponding to a biosignal received from a biosensor (276) based on the identified support set.

[0054] According to an embodiment, a processor (220) may store support sets for each of bio-signal frames sequentially acquired from a time-series bio-signal through a predicted label storage (340). According to an embodiment, the processor (220) may identify the most frequently identified support set by using n consecutive support sets among the support sets identified for previous bio-signal frames stored in the predicted label storage (340) and the support sets identified for the current bio-signal frame, and may identify (or determine) a body response (or action) corresponding to a bio-signal received from a bio-sensor (276) based on the most frequently identified support set.

[0055] According to one embodiment, the processor (220) may identify a command corresponding to an identified physical reaction (or motion) and perform an operation corresponding to the command corresponding to the physical reaction (or motion). For example, the command corresponding to the physical reaction (or motion) may include a command to execute a function of an application. According to one embodiment, the application may be one of a music playback application, an Internet browser application, a game application, or a plurality of applications executable on other electronic devices. For example, when the processor (220) identifies a physical reaction (or motion) of eye blinking during music playback using a music playback application, the processor (220) may perform a music playback stop command corresponding to the physical reaction (or motion) of eye blinking.

[0056] A biosensor (276) according to one embodiment can measure an electrical signal generated from a living body. A biosensor (276) according to one embodiment can measure an electrical signal generated from a body and output at least one of an electroencephalogram (EEG) signal, an electrocardiogram (ECG) signal, or an electromyography (EMG) signal, and can further output other biosignals based on electrical signals generated from other bodies.

[0057] A memory (230) according to an embodiment (e.g., memory (130) of FIG. 1) may store instructions. The memory (230) according to an embodiment may include one or more storage media that store instructions. The memory (230) according to an embodiment may store various data used by at least one component (e.g., processor (220), sensor (276), display (260), and / or communication circuit (290)) of the electronic device (201). The data may include, for example, software (e.g., software module or program (140)) and input data or output data for commands related thereto. The memory (230) according to an embodiment may store an artificial intelligence model, and the artificial intelligence model may be an artificial intelligence model that is trained or can be trained to obtain a vector value corresponding to a biosignal frame including features of a plurality of domain-specific biosignals for the biosignal frame. A memory (230) according to an embodiment may store various data generated during execution of a program (e.g., software or a program (140) of FIG. 1) that identifies (or determines) a physical response (or action) corresponding to a biosignal using an artificial intelligence model. A memory (230) according to an embodiment may store commands (or instructions) that cause a processor (220) to perform a biosignal pattern recognition operation (or method) using the artificial intelligence model of the present disclosure.

[0058] A display (260) according to an embodiment (e.g., display (160) of FIG. 1) may display various information based on the control of the processor (220). The display (260) according to an embodiment may display a screen related to the performance of a biosignal pattern recognition operation (or method) using an artificial intelligence model using an AI model (232). According to an embodiment, the display (260) may be implemented in the form of a touch screen. When the display (260) is implemented together with an input module in the form of a touch screen, it may display various information generated according to a user's touch operation.

[0059] A communication circuit (290) according to an embodiment (e.g., a communication module (190) of FIG. 1) may include a wired communication module (e.g., a USB communication module) and / or a wireless communication module (e.g., a cellular module, a Wi-Fi (wireless-fidelity) module, a Bluetooth module, or a NFC (near field communication) module). The communication circuit (290) according to an embodiment may communicate with an external biosensor (e.g., a biosensor (900) of FIG. 9). The communication circuit (290) according to an embodiment may receive a time-series biosignal sensed from an external biosensor based on the control of the processor (220).

[0060] According to one embodiment, the electronic device (201) is not limited to the configuration illustrated in FIG. 2 and may further include various components. According to one embodiment, the electronic device (201) may further include an input module (not illustrated) (e.g., the input module (150) of FIG. 1) and receive various inputs related to performing a biosignal pattern recognition operation (or method) using an artificial intelligence model through the input module.

[0061] In one embodiment, the main components of the electronic device have been described through the electronic device (201) of FIG. 2. However, in various embodiments, not all of the components illustrated through FIG. 2 are essential components, and the connection relationship of the main components of the electronic device (201) described above through FIG. 2 may be changed according to various embodiments.

[0062] An electronic device (e.g., the electronic device 101 of FIG. 1 or the electronic device 202 of FIG. 2) according to an embodiment of the present disclosure may include a biosensor (e.g., the sensor module 176 of FIG. 1 or the biosensor 276 of FIG. 2), a memory (e.g., the memory 130 of FIG. 1 or the memory 230 of FIG. 2) for storing instructions, and at least one processor operatively connected to the biosensor and the memory. The instructions according to an embodiment, when executed by the at least one processor, may cause the electronic device to obtain a biosignal frame having a specified frame size from a biosignal received by the biosensor. The instructions according to an embodiment, when executed by the at least one processor, may cause the electronic device to identify support sets to be compared with the biosignal frame among a plurality of support sets including biosignal frame samples corresponding to respective responses of a body. The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to obtain a vector value including features of a plurality of domain-specific biosignals for the biosignal frame using an artificial intelligence model. The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to identify a support set including a biosignal frame sample having a vector value most similar to the vector value of the biosignal frame among the identified support sets. The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to identify a bodily response corresponding to the received biosignal based on the identified support set.

[0063] The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to identify a command corresponding to the bodily reaction and to perform the command.

[0064] In one embodiment, the plurality of domains include a time domain, a frequency domain, and a time-frequency domain, and the instructions in one embodiment, when executed by the at least one processor, cause the electronic device to obtain a first biosignal in a time domain corresponding to the biosignal frame and obtain a normalized first biosignal by normalizing the first biosignal in the time domain, obtain a second biosignal in a frequency domain corresponding to the biosignal frame and obtain a normalized second biosignal by normalizing the second biosignal in the frequency domain, obtain a third biosignal in a time-frequency domain including two-dimensional values ​​(2Dimension values) corresponding to the biosignal frame by applying a short time Fourier transform (STFT) to the biosignal frame, obtain third biosignals of a normalized third channel by normalizing the third biosignal in the time-frequency domain row-by-row, column-by-column, and obtain the normalized first biosignal, the normalized second biosignal, and the normalized second biosignal. The third bio-signals of the three channels can be applied to the learned artificial intelligence model, and the vector value corresponding to the bio-signal frame can be obtained through the artificial intelligence model.

[0065] The above-mentioned specified frame size according to one embodiment may correspond to the input frame size of the artificial intelligence model.

[0066] The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to identify a variance value of the amplitude of the biosignal frame, a threshold value of a maximum value of the amplitude of the biosignal frame, or a ratio threshold value of a specific frequency of the biosignal frame, and to identify support sets that satisfy a condition based on the variance value of the amplitude of the biosignal frame, the threshold value of the maximum value of the amplitude of the biosignal frame, or the ratio threshold value of a specific frequency of the biosignal frame among the plurality of support sets.

[0067] The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to identify a specified number of consecutive support sets from among the support sets identified for previous bio-signal frames and the support sets identified for the bio-signal frame, and to identify a bodily response corresponding to the specified number of consecutive support sets.

[0068] The instructions according to one embodiment, when executed by the at least one processor, may cause the electronic device to register the plurality of support sets including the biosignal frame samples corresponding to each of the body's responses.

[0069] According to one embodiment, the biosignal may include an electroencephalogram signal, an electrocardiogram signal, or an electromyogram signal.

[0070] The above physical responses according to one embodiment may include staying still (default), chewing food (munch), clenching teeth (clenching), blinking, winking the left eye (wink_left), and / or winking the right eye (wink_right).

[0071] According to one embodiment, the artificial intelligence model may include a convolution neutral network (CNN) model, a recurrent neutral network (RNN) model, or an attention mechanism model.

[0072] FIG. 3 is a diagram illustrating a processor according to one embodiment.

[0073] Referring to FIG. 3, a processor (220) according to an embodiment may include a processing circuit and / or a software module. The processing circuit according to an embodiment may include a physical circuit including electrical elements, and the software module may include a program, data, and / or an algorithm. The processor (220) according to an embodiment may include a signal receiver (310), a support set manager (320), a pattern recognition / classification model (330), and / or a predicted label storage (340) corresponding to the processing circuit (or software module).

[0074] A signal receiver (310) according to one embodiment can receive a real-time biological signal sensed from a biosensor (276). According to an embodiment, a biosignal may include a time-series (or time-continuous) biosignal measured from the body by a biosensor (276) (or an external biosensor (e.g., a biosensor (900) of FIG. 9)). For example, the biosignal may include at least one of an electroencephalogram (EEG) signal, an electrocardiogram (ECG) signal, and an electromyography (EMG) signal, and other time-series biosignals that can be measured from the body may also be possible. A signal receiver (310) according to an embodiment may receive a time-series biosignal, perform noise filtering on it, and obtain a noise-filtered time-series biosignal. A signal receiver (310) according to an embodiment may include a frame extractor (312), and obtain a biosignal frame of a specified frame size from the biosignal by using the frame extractor (312). The specified frame size according to an embodiment may correspond to an input frame size of an artificial intelligence model. There is. A frame extractor (312) according to one embodiment may divide a time-series biosignal received from a biosensor (276) into a designated frame size corresponding to an input frame size of an artificial intelligence model (e.g., every 0.25 s), and transmit biosignal frames of the designated frame size to a support set manager (320) and a pattern recognition / classification model (330). The biosignal frame of the designated frame size may be a target signal (e.g., query) of the support set manager (320).

[0075] A support set manager (320) according to an embodiment may register one or more support sets corresponding to one or more bio-signal patterns corresponding to one or more bodily reactions (or bodily movements). The bio-signal patterns according to an embodiment may refer to bio-signal frame samples collected (or acquired) in advance corresponding to each of the bodily reactions (or movements). A support set manager (320) according to an embodiment may store and manage a plurality of support sets corresponding to bio-signal patterns of pre-registered bodily reactions (or bodily movements). A support set manager (320) according to an embodiment may identify (or select or determine) support sets to be compared with a bio-signal frame among a plurality of support sets corresponding to bio-signal patterns of pre-registered bodily reactions (or bodily movements). In one embodiment, the support set manager (320) may use a designated algorithm (e.g., a statistical or empirical algorithm) to identify a variance of the amplitude of a biosignal frame, a threshold value of a maximum value of the amplitude of the biosignal frame, or a threshold value of a ratio of a specific frequency of the biosignal frame, and may identify support sets that satisfy the variance of the amplitude of the biosignal frame, the threshold value of a maximum value of the amplitude of the biosignal frame, or the threshold value of a ratio of a specific frequency of the biosignal frame among a plurality of support sets. The bodily reactions (or motions) according to one embodiment include staying still (default), chewing food (munch), clenching teeth (clenching), blinking eyes (blinking), winking the left eye (wink_left), and / or winking the right eye (wink_right), and may further include other bodily reactions (or motions) of a person.According to an embodiment, a plurality of support sets corresponding to bio-signal patterns of pre-registered body reactions (or movements) may include a support set corresponding to an electroencephalogram (or brainwave) signal pattern when still, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when chewing food, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when clenching teeth, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when blinking eyes, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when winking a left eye, and / or a support set corresponding to an electroencephalogram (or brainwave) signal pattern when winking a right eye, and may further include a support set corresponding to another bio-signal pattern. Each of the plurality of support sets according to an embodiment may include k (e.g., a natural number, or 2 to 3) classes for few-shot training (or few-shot learning), and each class may include a plurality of sample bio-signal frames to be compared with a bio-signal frame. For example, a support set corresponding to an EEG (or brain wave) signal pattern during an eye blink may include k classes (e.g., a natural number, or 2 to 3), and each class may include sample biosignal frames corresponding to the EEG signal pattern during an eye blink to be compared with the biosignal frame.

[0076] A pattern recognition / classification model (330) according to an embodiment may receive a biosignal frame acquired by a signal receiver (310) and support sets identified (or selected) by a support set manager (320). A pattern recognition / classification model (330) according to an embodiment may identify (or determine) a body reaction (or action) corresponding to a biosignal received from a biosensor (276) using the biosignal frame and the support sets. A pattern recognition / classification model (330) according to one embodiment includes a preprocessor (331), an encoder (335), and / or a similarity calculator (337), and can identify (or determine) a body response (or action) corresponding to a biosignal received from a biosensor (276) using a biosignal frame and support sets through the preprocessor (331), the encoder (335), and / or the similarity calculator (337).

[0077] A preprocessor (331) according to an embodiment can obtain a plurality of domain-specific biosignals for a biosignal frame. The preprocessor (331) according to an embodiment includes a time domain processing module (time domain) (332), a frequency domain processing module (frequency domain) (334), and a time-frequency processing module (time-frequency domain) (333), and can obtain a plurality of domain-specific biosignals for a biosignal frame through the time domain processing module (time domain) (332), the frequency domain processing module (frequency domain) (334), and the time-frequency processing module (time-frequency domain) (333). The preprocessor (331) according to an embodiment can obtain a first biosignal in the time domain corresponding to a biosignal frame through the time domain processing module (time domain) (332), and obtain a normalized first biosignal by normalizing the first biosignal in the time domain. According to an embodiment, a preprocessor (331) may obtain a second biosignal in a frequency domain corresponding to a biosignal frame through a frequency domain processing module (frequency domain) (334) and obtain a normalized second biosignal by normalizing the second biosignal in the frequency domain. According to an embodiment, a preprocessor (331) may obtain a third biosignal in a time-frequency domain including two-dimensional values ​​(2Dimension values) corresponding to the biosignal frame by applying a short time Fourier transform (STFT) to the biosignal frame through a time-frequency processing module (time-frequency domain) (333), and may obtain third biosignals of a normalized third channel by normalizing the third biosignal in the time-frequency domain by row and column.

[0078] An encoder (335) according to an embodiment includes an artificial intelligence model, and can obtain a vector value corresponding to a biosignal frame including features of a plurality of domain-specific biosignals for a biosignal frame by using the artificial intelligence model. An encoder (335) according to an embodiment can apply (or input) the normalized first biosignal, the normalized second biosignal, and the normalized third biosignals of three channels to the artificial intelligence model, and obtain a vector value corresponding to the biosignal frame through the artificial intelligence model. The artificial intelligence model according to an embodiment may be an artificial intelligence model that is pre-learned (or can be learned) to interpret the normalized time domain biosignal, the normalized frequency domain biosignal, and the normalized three-channel biosignals of the time-frequency domain, respectively, and represent a vector value corresponding to a biosignal frame including features of a plurality of domain-specific biosignals. The artificial intelligence model according to an embodiment may include a convolution neutral network (CNN) model, a recurrent neutral network (RNN) model, or an attention mechanism model. According to one embodiment, the processor (220) can identify whether or not the artificial intelligence model of the encoder (335) has been trained to obtain a vector value corresponding to a biosignal frame, and if the artificial intelligence model has not been trained to obtain a vector value corresponding to a biosignal frame, perform a learning operation to obtain a vector value corresponding to the biosignal frame.

[0079] A similarity calculator (337) according to an embodiment may identify a support set including a sample biosignal frame having a vector value most similar to a vector value corresponding to a biosignal frame among support sets. A similarity calculator (337) according to an embodiment may use a function for calculating similarity to identify a support set including a biosignal frame sample having a vector value most similar to a vector value corresponding to a biosignal frame among support sets. For example, the function for calculating similarity may include cosine similarity.

[0080] A similarity calculator (337) according to one embodiment can identify (or determine) a body response (or action) corresponding to a biosignal received from a biosensor (276) based on an identified support set.

[0081] Predicted label storage (340) according to one embodiment can store support sets for each of bio-signal frames sequentially acquired from time-series bio-signals. Processor (220) according to one embodiment can identify n consecutive (or a specified number) support sets among support sets identified for previous bio-signal frames stored in Predicted label storage (340) and support sets identified for a current bio-signal frame, and can identify (or determine) a body response (or action) corresponding to a bio-signal received from a bio-sensor (276) based on the specified number of consecutive support sets.

[0082] FIG. 4 is a flowchart illustrating a biosignal pattern recognition operation in an electronic device according to one embodiment.

[0083] Referring to FIG. 4, a processor (e.g., processor (120) of FIG. 1 or processor (220) of FIG. 2, hereinafter, processor (220) of FIG. 2 will be described as an example) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2) according to one embodiment may perform at least one operation from operations 410 to 460.

[0084] In operation 410, the processor (220) according to an embodiment may receive a biosignal sensed from a biosensor (276). The processor (220) according to an embodiment may receive the biosignal sensed from the biosensor (276) in real time. The biosignal according to an embodiment may include a time-series (or time-continuous) biosignal measured from the body. For example, the biosignal may include at least one of an electroencephalogram (EEG) signal, an electrocardiogram (ECG) signal, and an electromyography (EMG) signal, and other time-series biosignals measurable from the body may also be possible. The processor (220) according to an embodiment may receive a time-series biosignal, perform noise filtering on it, and obtain a noise-filtered time-series biosignal.

[0085] In operation 420, the processor (220) according to an embodiment may obtain a biosignal frame of a specified frame size from a biosignal received (or received and noise-filtered) from a biosensor (276). The specified frame size according to an embodiment may correspond to an input frame size of an artificial intelligence model. The processor (220) according to an embodiment may divide a biosignal received in time series from the biosensor (276) into a specified frame size corresponding to the input frame size of the artificial intelligence model (e.g., every 0.25 s), and obtain biosignal frames of the specified frame size.

[0086] In operation 430, the processor (220) according to an embodiment may identify support sets to be compared with the biosignal frame among a plurality of support sets corresponding to pre-registered biosignal patterns. The biosignal patterns according to an embodiment may refer to biosignal frame samples that are collected (or acquired) in advance corresponding to each of the body's reactions (or operations). The processor (220) according to an embodiment may identify a variance value of the amplitude of the biosignal frame, a threshold value of the maximum value of the amplitude of the biosignal frame, or a ratio threshold value of a specific frequency of the biosignal frame using a designated algorithm (e.g., a statistical or empirical algorithm), and may identify support sets that satisfy the variance value of the amplitude of the biosignal frame, the threshold value of the maximum value of the amplitude of the biosignal frame, or the ratio threshold value of the specific frequency of the biosignal frame among the plurality of support sets. The bodily reactions (or movements) according to one embodiment include stillness (default), chewing food (munch), clenching teeth (clenching), blinking, winking the left eye (wink_left), and / or winking the right eye (wink_right), and may further include other bodily reactions (or movements) of the person.According to an embodiment, a plurality of support sets corresponding to bio-signal patterns of pre-registered body reactions (or movements) may include a support set corresponding to an electroencephalogram (or brainwave) signal pattern when still, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when chewing food, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when clenching teeth, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when blinking eyes, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when winking a left eye, and / or a support set corresponding to an electroencephalogram (or brainwave) signal pattern when winking a right eye, and may further include a support set corresponding to another bio-signal pattern. Each of the plurality of support sets according to an embodiment may include k (e.g., a natural number, or 2 to 3) classes for few-shot training (or few-shot learning), and each class may include a plurality of sample bio-signal frames to be compared with a bio-signal frame. For example, a support set corresponding to an EEG (or brain wave) signal pattern during an eye blink may include k classes (e.g., a natural number, or 2 to 3), and each class may include sample biosignal frames corresponding to the EEG signal pattern during an eye blink to be compared with the biosignal frame.

[0087] In operation 440, a processor (220) according to an embodiment may obtain a plurality of domain-specific biosignals for a biosignal frame and obtain a vector value including features of the plurality of domain-specific biosignals using an artificial intelligence model.

[0088] According to an embodiment, a processor (220) may acquire a first biosignal in a time domain corresponding to a biosignal frame and may acquire a normalized first biosignal by normalizing the first biosignal in the time domain. According to an embodiment, a processor (220) may acquire a second biosignal in a frequency domain corresponding to a biosignal frame and may acquire a normalized second biosignal by normalizing the second biosignal in the frequency domain. According to an embodiment, a processor (220) may apply a short time Fourier transform (STFT) to a biosignal frame to acquire a third biosignal in a time-frequency domain including two-dimensional values ​​(2Dimension values) corresponding to the biosignal frame, and may acquire third biosignals of a normalized third channel by normalizing the third biosignal in the time-frequency domain by row and column. According to one embodiment, a processor (220) may apply (or input) normalized first biosignals, normalized second biosignals, and normalized third biosignals of three channels to a pre-learned artificial intelligence model, and obtain vector values ​​corresponding to biosignal frames through the artificial intelligence model.

[0089] An artificial intelligence model according to an embodiment may be an artificial intelligence model trained to interpret a normalized time domain biosignal, a normalized frequency domain biosignal, and a three-channel biosignal of a normalized time-frequency domain, respectively, and represent a vector value corresponding to a biosignal frame including features of biosignals for each of a plurality of domains. The artificial intelligence model according to an embodiment may include a convolution neutral network (CNN) model, a recurrent neutral network (RNN) model, or an attention mechanism model. The processor (220) according to an embodiment may identify whether or not the artificial intelligence model is trained to obtain a vector value corresponding to a biosignal frame, and if the artificial intelligence model is not trained to obtain a vector value corresponding to a biosignal frame, perform a learning operation to obtain a vector value corresponding to a biosignal frame.

[0090] In operation 450, the processor (220) according to an embodiment may identify a support set including a biosignal frame sample having a vector value most similar to a vector value corresponding to a biosignal frame among support sets. The processor (220) according to an embodiment may use a function for calculating similarity to identify a support set including a biosignal frame sample having a vector value most similar to a vector value corresponding to a biosignal frame among support sets. For example, the function for calculating similarity may include cosine similarity.

[0091] In operation 460, the processor (220) according to an embodiment may identify (or determine) a physical response (or action) corresponding to a biosignal received from the biosensor (276) based on the identified support set. The processor (220) according to an embodiment may identify n consecutive support sets (or a specified number) among the support sets identified for previous biosignal frames stored in the predicted label storage (340) and the support sets identified for the current biosignal frame, and may identify (or determine) a physical response (or action) corresponding to the biosignal received from the biosensor (276) based on the specified number of consecutive support sets. The processor (220) according to an embodiment may identify a command corresponding to the identified physical response (or action) and further perform an action corresponding to the command corresponding to the physical response (or action). For example, the command corresponding to the physical response (or action) may include a command to execute a function of an application. According to one embodiment, the application may be a music playback application, an Internet browser application, a game application, or one of a plurality of applications executable on an electronic device. For example, the processor (220) may use a music playback application to identify a physical reaction (or motion) of eye blinking during music playback, and may perform a music playback stop command corresponding to the physical reaction (or motion) of eye blinking.

[0092] According to an embodiment of the present disclosure, a method for recognizing a biosignal pattern using an artificial intelligence model in an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (202) of FIG. 2) may include an operation of obtaining a biosignal frame of a specified frame size from a biosignal received by a biosensor (e.g., the sensor module (176) of FIG. 1 or the biosensor (276) of FIG. 2). The method according to an embodiment may include an operation of identifying support sets to be compared with the biosignal frame from among a plurality of support sets including biosignal frame samples corresponding to respective body reactions. The method according to an embodiment may include an operation of obtaining a vector value including features of a plurality of domain-specific biosignals for the biosignal frame using the artificial intelligence model. The method according to an embodiment may include an operation of identifying a support set including a biosignal frame sample having a vector value most similar to the vector value of the biosignal frame from among the identified support sets. The method according to one embodiment may include an operation of identifying a bodily response corresponding to the received biosignal based on the identified support set.

[0093] The method according to one embodiment may further include identifying a command corresponding to the bodily reaction and performing the command.

[0094] According to one embodiment, the plurality of domains may include a time domain, a frequency domain, and a time-frequency domain.

[0095] According to an embodiment, the method may include an operation of acquiring a first biosignal in a time domain corresponding to the biosignal frame and acquiring a normalized first biosignal by normalizing the first biosignal in the time domain. According to an embodiment, the method may include an operation of acquiring a second biosignal in a frequency domain corresponding to the biosignal frame and acquiring a normalized second biosignal by normalizing the second biosignal in the frequency domain. According to an embodiment, the method may include an operation of applying a short time Fourier transform (STFT) to the biosignal frame to acquire a third biosignal in a time-frequency domain including two-dimensional values ​​(2Dimension values) corresponding to the biosignal frame, and acquiring third biosignals of a normalized third channel by normalizing the third biosignal in the time-frequency domain row-by-row and column-by-column. According to one embodiment, the method may include applying the normalized first biosignal, the normalized second biosignal, and the normalized third biosignals of three channels to the learned artificial intelligence model, and obtaining the vector value corresponding to the biosignal frame through the artificial intelligence model.

[0096] According to one embodiment, the specified frame size may correspond to the input frame size of the artificial intelligence model.

[0097] According to an embodiment, the method may include an operation of identifying a variance value of the amplitude of the biosignal frame, a threshold value of a maximum value of the amplitude of the biosignal frame, or a ratio threshold value of a specific frequency of the biosignal frame. According to an embodiment, the method may include an operation of identifying support sets that satisfy a condition based on a variance value of the amplitude of the biosignal frame, a threshold value of a maximum value of the amplitude of the biosignal frame, or a ratio threshold value of a specific frequency of the biosignal frame, among the plurality of support sets.

[0098] The method according to one embodiment may include an operation of identifying a specified number of consecutive support sets among the support sets identified for previous bio-signal frames and the support sets identified for the bio-signal frame, and identifying a bodily response corresponding to the specified number of consecutive support sets.

[0099] The method according to one embodiment may include registering a plurality of support sets including biosignal frame samples corresponding to each of the body's responses.

[0100] In the method according to one embodiment, the biosignal may include an electroencephalogram signal, an electrocardiogram signal, or an electromyogram signal.

[0101] In one embodiment, the bodily responses may include staying still (default), chewing food (munch), clenching teeth (clenching), blinking, winking the left eye (wink_left), and / or winking the right eye (wink_right).

[0102] FIG. 5 is a flowchart illustrating a biosignal pattern recognition operation capable of registering a support set and learning an artificial intelligence model in an electronic device according to one embodiment.

[0103] Referring to FIG. 5, a processor (e.g., processor (120) of FIG. 1 or processor (220) of FIG. 2, hereinafter, processor (220) of FIG. 2 will be described as an example) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2) according to one embodiment may perform at least one operation from operations 510 to 560.

[0104] In operation 510, a processor (220) according to an embodiment may receive a biological signal sensed from a biological sensor (276). The processor (220) according to an embodiment may receive a biological signal sensed from the biological sensor (276) in real time. The biological signal according to an embodiment may include a time-series (or time-continuous) biological signal measured from the body. For example, the biological signal may include at least one of an electroencephalogram (EEG) signal, an electrocardiogram (ECG) signal, and an electromyography (EMG) signal, and other time-series biological signals measurable from the body may also be possible. The processor (220) according to an embodiment may receive a time-series biological signal, perform noise filtering on the time-series biological signal, and obtain a noise-filtered time-series biological signal.

[0105] In operation 520, the processor (220) according to an embodiment may obtain biological signal frames of a specified frame size from the received biological signals (Extract frames from biological signals). The specified frame size according to an embodiment may correspond to the input frame size of the artificial intelligence model. The processor (220) according to an embodiment may divide the biological signals received in time series from the biological sensor (276) into specified frame sizes corresponding to the input frame size of the artificial intelligence model (e.g., every 0.25 s), and obtain biological signal frames of the specified frame size.

[0106] In operation 522, the processor (220) according to one embodiment can identify whether there are multiple support sets corresponding to pre-registered bio-signal patterns (Is there any registered support set for comparison?).

[0107] In operation 524, the processor (220) according to an embodiment may register a plurality of support sets corresponding to the bio-signal patterns if there are no plurality of support sets corresponding to the pre-registered bio-signal patterns (Register comparison signals (support sets)). The processor (220) according to an embodiment may obtain bio-signal frame samples for each body response using bio-signals sensed in response to each of the body responses (or actions) and register a plurality of support sets corresponding to the bio-signal samples for each body response. For example, the processor (220) may obtain EEG bio-signal frame samples for each body response using EEG signals sensed in response to each of the body responses (or actions) for a brain-computer interface (e.g., staying still, chewing food, clenching teeth, blinking, left eye wink, and / or right eye wink) and register a plurality of support sets corresponding to the EEG bio-signal samples for each body response. According to one embodiment, a processor (220) can perform operation 530 after registering a plurality of support sets corresponding to biosignal patterns.

[0108] In operation 530, the processor (220) according to an embodiment may identify (or select) support sets to be compared with the biosignal frame from among a plurality of support sets corresponding to the biosignal patterns (Select support set in simple ways). The biosignal patterns according to an embodiment may refer to biosignal frame samples that are collected (or acquired) in advance corresponding to each of the body's reactions (or operations). The processor (220) according to an embodiment may identify a variance value of the amplitude of the biosignal frame, a threshold value of the maximum value of the amplitude of the biosignal frame, or a threshold value of the ratio of a specific frequency of the biosignal frame using a designated algorithm (e.g., a statistical or empirical algorithm), and may identify support sets that satisfy the variance value of the amplitude of the biosignal frame, the threshold value of the maximum value of the amplitude of the biosignal frame, or the threshold value of the ratio of a specific frequency of the biosignal frame from among the plurality of support sets. The bodily reactions (or movements) according to one embodiment include stillness (default), chewing food (munch), clenching teeth (clenching), blinking, winking the left eye (wink_left), and / or winking the right eye (wink_right), and may further include other bodily reactions (or movements) of the person.According to an embodiment, a plurality of support sets corresponding to bio-signal patterns of pre-registered body reactions (or movements) may include a support set corresponding to an electroencephalogram (or brainwave) signal pattern when still, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when chewing food, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when clenching teeth, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when blinking eyes, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when winking a left eye, and / or a support set corresponding to an electroencephalogram (or brainwave) signal pattern when winking a right eye, and may further include a support set corresponding to another bio-signal pattern. Each of the plurality of support sets according to an embodiment may include k (e.g., a natural number, or 2 to 3) classes for few-shot training (or few-shot learning), and each class may include a plurality of sample bio-signal frames to be compared with a bio-signal frame. For example, a support set corresponding to an EEG (or brain wave) signal pattern during an eye blink may include k classes (e.g., a natural number, or 2 to 3), and each class may include sample biosignal frames corresponding to the EEG signal pattern during an eye blink to be compared with the biosignal frame.

[0109] In operation 540, a processor (220) according to an embodiment may obtain a plurality of domain-specific biosignals for a biosignal frame (Decompose query and support set into three domain features). The processor (220) according to an embodiment may obtain a first biosignal in a time domain corresponding to the biosignal frame and obtain a normalized first biosignal by normalizing the first biosignal in the time domain. The processor (220) according to an embodiment may obtain a second biosignal in a frequency domain corresponding to the biosignal frame and obtain a normalized second biosignal by normalizing the second biosignal in the frequency domain. According to an embodiment, a processor (220) may obtain a third biosignal in a time-frequency domain including two-dimensional values ​​(2Dimension values) corresponding to the biosignal frame by applying a short time Fourier transform (STFT) to the biosignal frame, and may obtain third biosignals in a normalized third channel by normalizing the third biosignal in the time-frequency domain by row and column. According to an embodiment, a processor (220) may obtain a plurality of domain-specific biosignal samples for each of the biosignal frame samples included in each of the support sets in the same manner as the biosignal frame.

[0110] In operation 542, the processor (220) according to an embodiment may identify whether an artificial intelligence model to which multiple domain-specific biosignals are to be applied to a biosignal frame has been trained (Is the encoder trained?). If the artificial intelligence model to which multiple domain-specific biosignals are to be applied to a biosignal frame has not been trained, the processor (220) according to an embodiment may perform operation 550 after performing operation 544. If the artificial intelligence model to which multiple domain-specific biosignals are to be applied to a biosignal frame has been trained, the processor (220) according to an embodiment may perform operation 550.

[0111] In operation 544, a processor (220) according to an embodiment may train an artificial intelligence model (train the encoder to compress information for a plurality of domains of biological signals). The processor (220) according to an embodiment may perform training so that the artificial intelligence model interprets normalized time domain biological signals, normalized frequency domain biological signals, and three-channel normalized time-frequency domain biological signals, respectively, to obtain vector values ​​corresponding to biological signal frames including features of a plurality of domain-specific biological signals. The artificial intelligence model according to an embodiment may include a convolution neutral network (CNN) model, a recurrent neutral network (RNN) model, or an attention mechanism model.

[0112] In operation 550, the processor (220) according to an embodiment may obtain a vector value including features of a plurality of domain-specific biosignals corresponding to a biosignal frame using an artificial intelligence model. The processor (220) according to an embodiment may apply (or input) a normalized first biosignal, a normalized second biosignal, and a normalized three-channel third biosignal to a pre-trained artificial intelligence model, and obtain a vector value corresponding to the biosignal frame through the artificial intelligence model. The artificial intelligence model according to an embodiment may be an artificial intelligence model trained to interpret a normalized time domain biosignal, a normalized frequency domain biosignal, and a normalized three-channel time-frequency domain biosignal, respectively, and represent a vector value corresponding to a biosignal frame including features of a plurality of domain-specific biosignals.

[0113] In operation 560, the processor (220) according to an embodiment may identify a support set including a biosignal frame sample having a vector value most similar to a vector value corresponding to a biosignal frame among support sets (Find the highest similarity among the support sets). The processor (220) according to an embodiment may use a function for calculating similarity to identify a support set including a biosignal frame sample having a vector value most similar to a vector value corresponding to a biosignal frame among support sets. For example, the function for calculating similarity may include cosine similarity.

[0114] In operation 570, the processor (220) according to an embodiment may identify (or determine) a physical response (or action) corresponding to a biosignal received from the biosensor (276) based on the identified support set. The processor (220) according to an embodiment may identify n consecutive support sets (or a specified number) from among the support sets identified for previous biosignal frames stored in the Predicted label storage (340) and the support sets identified for the current biosignal frame, and may identify (or determine) a physical response (or action) corresponding to the biosignal received from the biosensor (276) based on the consecutive specified number of support sets (Derive final result by referring to past result). The processor (220) according to an embodiment may identify a command corresponding to the identified physical response (or action) and further perform an action corresponding to the command corresponding to the physical response (or action). For example, a command corresponding to a physical reaction (or motion) may include a command for controlling the electronic device (201). For example, a command corresponding to a physical reaction (or motion) may include a command for executing a function of an application for controlling the electronic device (201). According to one embodiment, the application may be one of a music playback application, an Internet browser application, a game application, or a plurality of applications executable on the electronic device. For example, when the processor (220) identifies a physical reaction (or motion) of eye blinking during music playback using a music playback application, the processor (220) may perform a music playback stop command corresponding to the physical reaction (or motion) of eye blinking.

[0115] FIG. 6 is a diagram for explaining an operation of recognizing brainwave signals patterns according to one embodiment.

[0116] Referring to FIG. 6, a processor (e.g., processor (120) of FIG. 1 or processor (220) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2) according to an embodiment may identify a command corresponding to a physical reaction (or action) from an electroencephalogram (or brainwave) signal received in real time from a biosensor (e.g., biosensor (276) of FIG. 2) through operations 610 to 660. The processor (220) according to an embodiment may use (or recognize or identify) a command corresponding to a physical reaction identified from the brainwave signal as a command for controlling the electronic device (201).

[0117] In operation 610, a processor according to an embodiment may receive a brain wave signal sensed from a biosensor (276) (e.g., an EEG sensor). The processor (220) according to an embodiment may receive the brain wave signal in real time. The brain wave signal according to an embodiment may include a time-series (or continuous over time) brain wave signal measured from the brain among the body.

[0118] In operation 620, the processor (220) according to an embodiment may obtain an EEG signal frame of a specified frame size from the received EEG signal. The specified frame size according to an embodiment may correspond to the input frame size of the artificial intelligence model. The processor (220) according to an embodiment may divide the EEG signal received in time series into specified frame sizes corresponding to the input frame size of the artificial intelligence model (e.g., every 0.25 s), and obtain EEG signal frames of the specified frame size.

[0119] In operation 630, the processor (220) according to an embodiment may identify support sets (632, 634, 636, 638) to be compared with the EEG signal frame among a plurality of support sets corresponding to pre-registered EEG signal patterns. The EEG signal patterns according to an embodiment may refer to EEG signal frame samples that are pre-collected (or acquired) corresponding to each of body reactions (or movements). The processor (220) according to an embodiment may identify a variance value of the amplitude of the EEG signal frame, a threshold value of the maximum value of the amplitude of the EEG signal frame, or a threshold value of the ratio of a specific frequency of the EEG signal frame using a designated algorithm (e.g., a statistical or empirical algorithm), and may identify support sets that satisfy the variance value of the amplitude of the EEG signal frame, the threshold value of the maximum value of the amplitude of the EEG signal frame, or the threshold value of the ratio of a specific frequency of the EEG signal frame among the plurality of support sets. The bodily reactions (or movements) according to one embodiment include stillness (default), chewing food (munch), clenching teeth (clenching), blinking, winking the left eye (wink_left), and / or winking the right eye (wink_right), and may further include other bodily reactions (or movements) of the person.According to an embodiment, a plurality of support sets corresponding to bio-signal patterns of pre-registered body reactions (or movements) may include a support set corresponding to an electroencephalogram (or brainwave) signal pattern when still, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when chewing food, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when clenching teeth, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when blinking eyes, a support set corresponding to an electroencephalogram (or brainwave) signal pattern when winking a left eye, and / or a support set corresponding to an electroencephalogram (or brainwave) signal pattern when winking a right eye, and may further include a support set corresponding to another bio-signal pattern. Each of the plurality of support sets according to an embodiment may include k (e.g., a natural number, or 2 to 3) classes for few-shot training (or few-shot learning), and each class may include a plurality of sample brainwave signal frames to be compared with an brainwave signal frame. For example, a support set corresponding to an EEG signal pattern during eye blinking may include k (e.g., a natural number, or 2 to 3) classes, and each class may include sample EEG signal frames corresponding to the EEG signal pattern during eye blinking to be compared with the EEG signal frame. For example, support sets (632, 634, 636, 638) to be compared with the EEG signal frame may include a support set (632) corresponding to an EEG signal pattern during clenching, a support set (634) corresponding to an EEG signal pattern during chewing, a support set (636) corresponding to an EEG signal pattern during stillness, and a support set (638) corresponding to an EEG signal pattern during eye blinking, which are some of a plurality of support sets.

[0120] In operation 640, the processor (220) according to an embodiment may obtain a plurality of domain-specific brainwave signals for a brainwave signal frame (query) and obtain a vector value including features of the plurality of domain-specific brainwave signals using an artificial intelligence model (CNN model). The processor (220) according to an embodiment may obtain a first brainwave signal in a time domain corresponding to the brainwave signal frame and obtain a normalized first brainwave signal by normalizing the first brainwave signal in the time domain. The processor (220) according to an embodiment may obtain a second brainwave signal in a frequency domain corresponding to the brainwave signal frame and obtain a normalized second brainwave signal by normalizing the second biosignal in the frequency domain. According to an embodiment, a processor (220) may obtain a third brain wave signal in a time-frequency domain including two-dimensional values ​​(2Dimension values) corresponding to the brain wave signal frame by applying a short time Fourier transform (STFT) to the brain wave signal frame, and may obtain third brain wave signals (or values ​​of signals) of a normalized third channel by normalizing the third brain wave signal in the time-frequency domain by row and column. According to an embodiment, the processor (220) may apply (or input) the normalized first brain wave signal, the normalized second brain wave signal, and the normalized three-channel third brain wave signals to a pre-learned artificial intelligence model, and may obtain a vector value corresponding to the brain wave signal frame through the artificial intelligence model. An artificial intelligence model according to one embodiment may be an artificial intelligence model trained to interpret brainwave signals in a normalized time domain, brainwave signals in a normalized frequency domain, and three-channel brainwave signals in a normalized time-frequency domain, respectively, and represent vector values ​​corresponding to brainwave signal frames including features of brainwave signals for each of multiple domains.An artificial intelligence model according to one embodiment may include a convolution neutral network (CNN) model, a recurrent neutral network (RNN) model, or an attention mechanism model.

[0121] In operation 650, a processor (220) according to an embodiment may obtain a plurality of domain-specific EEG signal samples for each of the EEG signal frame samples included in the support set for each of the support sets in the same manner as the EEG signal frame, and obtain vector values ​​including features of the plurality of domain-specific EEG signal samples.

[0122] In operation 660, the processor (220) according to an embodiment may identify a support set (e.g., a support set corresponding to an EEG signal during eye blinking) including an EEG signal frame sample having a vector value most similar to a vector value corresponding to an EEG signal frame among support sets (e.g., a vector value having a similarity of about 89%). The processor (220) according to an embodiment may use a function for calculating similarity to identify a support set including an EEG signal frame sample having a vector value most similar to a vector value corresponding to an EEG signal frame among support sets. For example, the function for calculating similarity may include cosine similarity.

[0123] In operation 670, the processor (220) according to an embodiment may identify (or determine) an eye blink response (or action) corresponding to a received brainwave signal based on the identified support set. The processor (220) according to an embodiment may identify a most frequently identified support set using n consecutive support sets (e.g., a natural number or 3) among the support sets identified for previous brainwave signal frames stored in the predicted label storage (340) and the support sets identified for the current brainwave signal frame, and may identify (or determine) an eye blink response (or action) corresponding to the received brainwave signal based on the most frequently identified support set. The processor (220) according to an embodiment may identify a command corresponding to the identified eye blink response (or action) and perform an action corresponding to the command corresponding to the eye blink response (or action). For example, the processor (220) may use a music playback application to identify an eye blink response (or motion) during music playback and stop music playback according to a playback stop command corresponding to the eye blink response (or motion).

[0124] Figure 7 is a diagram showing examples of brain wave signals according to one embodiment.

[0125] Referring to FIG. 7, a processor (e.g., processor (120) of FIG. 1 or processor (220) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2) according to an embodiment may receive a time-series (or time-continuous) brain wave signal measured from a biosensor (276) (e.g., EGG sensor). The brain wave signal according to an embodiment may be a different brain wave signal pattern depending on a body reaction.

[0126] Referring to the first graph (710) according to one embodiment, the x-axis may represent time and the y-axis may represent amplitude. According to one embodiment, the processor (220) may receive an EEG signal (712) of a first EEG signal pattern from a biosensor (276) when the processor (220) is at rest (default).

[0127] Referring to the second graph (720) according to one embodiment, the x-axis may represent time and the y-axis may represent amplitude. According to one embodiment, the processor (220) may receive an EEG signal (722) of a second EEG signal pattern from a biosensor (276) when chewing food (munch).

[0128] Referring to the third graph (730) according to one embodiment, the x-axis may represent time and the y-axis may represent amplitude. According to one embodiment, the processor (220) may receive an EEG signal (732) of a third EEG signal pattern from the biosensor (276) during clenching.

[0129] Referring to the fourth graph (740) according to one embodiment, the x-axis may represent time and the y-axis may represent amplitude. According to one embodiment, the processor (220) may receive an EEG signal (742) of the fourth EEG signal pattern from the biosensor (276) when the eye blinks.

[0130] FIG. 8a is a diagram showing a normalized signal of a time domain brainwave signal according to one embodiment.

[0131] Referring to FIG. 8A, a processor (e.g., processor (120) of FIG. 1 or processor (220) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2) according to an embodiment may obtain a first brainwave signal (852) of a normalized time domain as in the fifth graph (850). Referring to the fifth graph (850) according to an embodiment, the x-axis may be time and the y-axis may be amplitude, and the first brainwave signal (852) of a normalized time domain may be obtained by scaling the amplitude of the brainwave signal of the time domain to train an artificial intelligence model.

[0132] FIG. 8b is a diagram showing a normalized signal of an EEG signal in the frequency domain according to one embodiment.

[0133] Referring to FIG. 8B, a processor (e.g., processor (120) of FIG. 1 or processor (220) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2) according to an embodiment may obtain a second brainwave signal (862) of a normalized frequency domain as in the sixth graph (860). Referring to the sixth graph (860) according to an embodiment, the x-axis may be frequency and the y-axis may be amplitude, and the second brainwave signal (862) of a normalized frequency domain may be obtained by scaling the amplitude of the brainwave signal of the frequency domain to train an artificial intelligence model.

[0134] FIG. 8c is a diagram showing a normalized signal of an EEG signal in the time-frequency domain according to one embodiment.

[0135] Referring to FIG. 8C, a processor (e.g., the processor (120) of FIG. 1 or the processor (220) of FIG. 2) of an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) according to an embodiment may obtain third brainwave signals (or values ​​of signals) of a normalized three-channel time-frequency domain, such as the seventh to ninth graphs (872, 874, 876). For example, the processor (220) may obtain a third biosignal of a time-frequency domain including two-dimensional values ​​(2Dimension values) corresponding to the biosignal frame by applying a short time Fourier transform (STFT) to the biosignal frame, and may obtain third biosignals of a normalized third channel obtained by normalizing the third biosignal of the time-frequency domain by row and column.

[0136] FIG. 9 is a diagram illustrating an example of a biosensor and an electronic device according to one embodiment.

[0137] Referring to FIG. 9, a biometric sensor (900) according to one embodiment (e.g., a sensor module (176) of FIG. 1 or a biometric sensor (276) of FIG. 2) is a device that exists outside of an electronic device (901) (e.g., an electronic device (101) of FIG. 1 or an electronic device (202) of FIG. 2) and can communicate with the electronic device (901).

[0138] A biosensor (900) according to an embodiment may include a first electrode (910), a second electrode (920), a third electrode (930), a sensing circuit (not shown), and a communication circuit (not shown). The first electrode (910) according to an embodiment may be a reference electrode, may include a conductive material (e.g., 24k gold), and may be a portion attached to the body. The second electrode (920) according to an embodiment may be an active electrode, may include a conductive material (e.g., 24k gold), and may be a portion attached to the body. The third electrode (930) according to an embodiment may be a ground electrode, may include silicon, and may be a portion attached to the body. A sensing circuit (not shown) according to an embodiment may sense brainwave signals using the first to third electrodes (910, 920, 930). A communication circuit (not shown) according to one embodiment can transmit brain wave signals sensed in real time to an electronic device (901).

[0139] An electronic device (901) according to an embodiment may be a smart phone. The electronic device (901) according to an embodiment may receive a biometric signal from a biometric sensor (900) and identify a physical reaction (or motion) of a user wearing the biometric sensor (900) by using the received biometric signal through the methods (or components) disclosed in the present disclosure. The electronic device (901) according to an embodiment may identify a command corresponding to the identified physical reaction and control the electronic device (901) or a function of the electronic device (901) based on the identified command. For example, the electronic device (901) may identify a command corresponding to a specific physical reaction (e.g., a command corresponding to the physical reaction of eye blinking) by using the biometric sensor received from the biometric sensor (900) while playing music through a music application, and may stop playing the music if the identified command is a command to stop playing the music.

[0140] The biosignal pattern recognition method using biosignals from multiple domains according to the present disclosure can accurately and efficiently identify a registered biosignal pattern corresponding to a biosignal received in real time. According to the present disclosure, by accurately identifying a registered biosignal pattern corresponding to a received biosignal, the registered biosignal pattern can be utilized as a command to an electronic device.

[0141] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0142] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0143] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0144] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more commands stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0145] According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0146] In a non-transitory storage medium storing commands according to one embodiment of the present disclosure, the commands are configured to cause the electronic device to perform at least one operation when executed by the electronic device, wherein the at least one operation may include: obtaining a biosignal frame of a specified frame size from a biosignal received by a biosensor; identifying support sets to be compared with the biosignal frame among a plurality of support sets including biosignal frame samples corresponding to respective body reactions; obtaining a vector value including characteristics of a plurality of domain-specific biosignals for the biosignal frame using the artificial intelligence model; identifying a support set including a biosignal frame sample having a vector value most similar to the vector value of the biosignal frame among the identified support sets; and identifying a body reaction corresponding to the received biosignal based on the identified support set.

[0147] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0148] As used herein, the term “if” will be understood to mean “when, upon,” “in response to deciding,” or “in response to detecting,” depending on the context. Similarly, “if it is decided to do,” or “if [the stated condition or event] is detected,” will optionally be understood to mean “upon deciding,” or “in response to deciding,” “upon detecting [the stated condition or event],” or “in response to detecting [the stated condition or event].”

[0149] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. A processing device (or processing circuit) may execute an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0150] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0151] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0152] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0153] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In an electronic device (101, 201), Biometric sensors (176, 276); Memory (130, 230) for storing instructions; and At least one processor (120, 220) operatively connected to the biometric sensor and the memory, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Obtaining a biosignal frame of a specified frame size from a biosignal received by the above biosensor, Identifying support sets to be compared with the bio-signal frames among a plurality of support sets including bio-signal frame samples corresponding to each of the body's responses, Using an artificial intelligence model, a vector value including the features of multiple domain-specific biosignals for the biosignal frame is obtained, Identifying a support set including a biosignal frame sample having a vector value most similar to the vector value of the biosignal frame among the identified support sets, and An electronic device that identifies a body response corresponding to the received bio-signal based on the identified support set.

2. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: An electronic device that identifies a command corresponding to the above physical reaction and executes the command.

3. In paragraph 1 or 2, The above multiple domains include a time domain, a frequency domain, and a time-frequency domain, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Obtaining a first biosignal in the time domain corresponding to the above biosignal frame and obtaining a normalized first biosignal by normalizing the first biosignal in the time domain, Obtaining a second biosignal in a frequency domain corresponding to the above biosignal frame and obtaining a normalized second biosignal by normalizing the second biosignal in the frequency domain, Applying STFT (short time Fourier transform) to the above biosignal frame, obtaining a third biosignal in a time-frequency domain including two-dimensional values ​​(2Dimension values) corresponding to the above biosignal frame, obtaining third biosignals of a normalized third channel by normalizing the third biosignal in the time-frequency domain by row and column, and An electronic device that applies the normalized first bio-signal, the normalized second bio-signal, and the normalized third bio-signals of three channels to the learned artificial intelligence model, and obtains the vector value corresponding to the bio-signal frame through the artificial intelligence model.

4. In any one of paragraphs 1 to 3, An electronic device wherein the above-mentioned specified frame size corresponds to the input frame size of the artificial intelligence model.

5. In any one of paragraphs 1 to 4, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Identifying a variance value of the amplitude of the biosignal frame, a threshold value of the maximum value of the amplitude of the biosignal frame, or a ratio threshold value of a specific frequency of the biosignal frame, An electronic device that identifies support sets that satisfy a condition based on a variance value of the amplitude of the biosignal frame, a threshold value of the maximum value of the amplitude of the biosignal frame, or a threshold value of a ratio of a specific frequency of the biosignal frame among the plurality of support sets.

6. In any one of paragraphs 1 to 5, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: An electronic device that identifies a specified number of consecutive support sets among the support sets identified for the previous bio-signal frames and the support sets identified for the bio-signal frames, and identifies a bodily response corresponding to the specified number of consecutive support sets.

7. In any one of paragraphs 1 to 6, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: An electronic device that registers a plurality of support sets including the bio-signal frame samples corresponding to each of the body's reactions.

8. In any one of paragraphs 1 to 7, The above bio-signal is an electronic device including an electroencephalogram signal, an electrocardiogram signal, or an electromyogram signal.

9. In any one of paragraphs 1 to 8, The above physical reactions include electronic devices including staying still (default), chewing food (munch), clenching teeth (clenching), blinking, winking the left eye (wink_left), and / or winking the right eye (wink_right).

10. In any one of paragraphs 1 to 9, The above artificial intelligence model is an electronic device including a CNN (convolution neutral network) model, an RNN (recurrent neutral network) model, or an Attention mechanism model.

11. A method for recognizing a biosignal pattern using an artificial intelligence model in an electronic device (101, 201), An operation of acquiring a biosignal frame of a specified frame size from a biosignal received by a biosensor (176, 276); An operation of identifying support sets to be compared with the bio-signal frame among a plurality of support sets including bio-signal frame samples corresponding to each of the body's reactions; An operation of obtaining a vector value including features of multiple domain-specific bio-signals for the bio-signal frame using the artificial intelligence model; An operation of identifying a support set including a bio-signal frame sample having a vector value most similar to the vector value of the bio-signal frame among the identified support sets; and A method comprising an action of identifying a bodily response corresponding to the received biosignal based on the identified support set.

12. In paragraph 11, A method further comprising identifying a command corresponding to the above physical reaction and performing an action of the command.

13. In paragraph 11 or 12, The above multiple domains include a time domain, a frequency domain, and a time-frequency domain, An operation of acquiring a first biosignal in a time domain corresponding to the biosignal frame and acquiring a normalized first biosignal by normalizing the first biosignal in the time domain; An operation of acquiring a second biosignal in a frequency domain corresponding to the biosignal frame and acquiring a normalized second biosignal by normalizing the second biosignal in the frequency domain; An operation of applying a short time Fourier transform (STFT) to the above biosignal frame to obtain a third biosignal in a time-frequency domain including two-dimensional values ​​(2Dimension values) corresponding to the above biosignal frame, and obtaining third biosignals of a normalized third channel by normalizing the third biosignal in the time-frequency domain by row and column; and A method including an operation of applying the normalized first bio-signal, the normalized second bio-signal, and the normalized third bio-signals of three channels to the learned artificial intelligence model, and obtaining the vector value corresponding to the bio-signal frame through the artificial intelligence model.

14. In any one of paragraphs 11 to 13, A method in which the above-mentioned specified frame size corresponds to the input frame size of the artificial intelligence model.

15. In a non-transitory storage medium storing commands, the commands are set to cause the electronic device to perform at least one operation when executed by the electronic device, the at least one operation being: An operation of acquiring a biosignal frame of a specified frame size from a biosignal received by a biosensor; An operation of identifying support sets to be compared with the bio-signal frame among a plurality of support sets including bio-signal frame samples corresponding to each of the body's reactions; An operation of obtaining a vector value including features of multiple domain-specific bio-signals for the bio-signal frame using the artificial intelligence model; An operation of identifying a support set including a bio-signal frame sample having a vector value most similar to the vector value of the bio-signal frame among the identified support sets; and A storage medium including an operation for identifying a bodily response corresponding to the received biosignal based on the identified support set.

Images