Electronic device and method for identifying input, and non-transitory computer-readable storage medium

Abstract

This electronic device may comprise at least one processor. The at least one processor is configured to: acquire, from a wearable device connected to the electronic device, sensor data for a body part of a user wearing the wearable device; identify a sensor dataset for a pencil mode according to state information of the user, identified on the basis of the sensor data; identify, on the basis of the sensor dataset and the sensor data acquired from the wearable device, locations of dots formed on the screen of the display by means of a hovering input according to movement of the body part; and change, on the basis of the locations of the dots, the screen displayed on the display.

Description

Electronic device, method, and non-transitory computer-readable storage medium for identifying input

[0001] The following descriptions relate to electronic devices, methods, and non-transitory computer-readable storage media for identifying input.

[0002] An electronic device can obtain user input to change the screen displayed on the display of the electronic device using an input means. However, the user of the electronic device must carry the input means to obtain user input to change the screen displayed on the display of the electronic device.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] An electronic device is provided. The electronic device may include a communication circuit. The electronic device may include a display. The electronic device may include a memory storing instructions and including one or more storage media. The electronic device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain sensor data about a body part of a user wearing a wearable device connected to the electronic device from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a sensor data set for a pencil mode based on state information of the user identified based on the sensor data. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify positions of dots on a screen of the display caused by a hovering input according to movement of the body part, based on the sensor data set and the sensor data obtained from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to change the screen displayed through the display based on the positions of the dots.

[0005] A method performed by an electronic device is provided. The method may include an operation of acquiring sensor data regarding a body part of a user wearing a wearable device connected to the electronic device from the wearable device. The method may include an operation of identifying a sensor data set for a pencil mode based on status information of the user identified based on the sensor data. The method may include an operation of identifying positions of dots on a screen of the display caused by a hovering input according to a movement of the body part based on the sensor data set and the sensor data acquired from the wearable device. The method may include an operation of changing the screen displayed through the display based on the positions of the dots.

[0006] A non-transitory computer-readable storage medium storing one or more programs is provided. The one or more programs may include instructions that, when executed by an electronic device including at least one processor, cause the electronic device to obtain sensor data about a body part of a user wearing a wearable device connected to the electronic device from the wearable device. The one or more programs may include instructions that, when executed by the electronic device including at least one processor, cause the electronic device to identify a sensor data set for a pencil mode based on state information of the user identified based on the sensor data. The one or more programs may include instructions that, when executed by the electronic device including at least one processor, cause the electronic device to identify locations of dots on a screen of the display caused by a hovering input according to a movement of the body part based on the sensor data set and the sensor data acquired from the wearable device. The one or more programs may include instructions that, when executed by an electronic device including at least one processor, cause the electronic device to change the screen displayed through the display based on the positions of the dots.

[0007] An electronic device is provided. The electronic device may include a communication circuit. The electronic device may include a display. The electronic device may include a memory that stores instructions and includes one or more storage media. The electronic device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive sensor data acquired by at least one sensor of a wearable device connected to the electronic device from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a hovering input according to a movement of a body part wearing the wearable device based on using the sensor data as input data for an artificial intelligence model of the electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate an output signal corresponding to the hovering input. The above sensor data may include at least one of biometric information of the body part, motion information of the body part, or posture information of the body part.

[0008] A method performed by an electronic device is provided. The method may include receiving sensor data acquired by at least one sensor of a wearable device connected to the electronic device from the wearable device. The method may include identifying a hovering input according to a movement of a body part wearing the wearable device based on using the sensor data as input data of an artificial intelligence model of the electronic device. The method may include generating an output signal corresponding to the hovering input. The sensor data may include at least one of biometric information of the body part, motion information of the body part, or posture information of the body part.

[0009] A non-transitory computer-readable storage medium storing one or more programs is provided. The one or more programs may include instructions that, when executed by an electronic device including at least one processor, cause the electronic device to receive sensor data acquired by at least one sensor of a wearable device connected to the electronic device from the wearable device. The one or more programs may include instructions that, when executed by the electronic device including at least one processor, cause the electronic device to identify a hovering input according to a movement of a body part wearing the wearable device based on using the sensor data as input data of an artificial intelligence model of the electronic device. The one or more programs may include instructions that, when executed by the electronic device including at least one processor, cause the electronic device to generate an output signal corresponding to the hovering input. The sensor data may include at least one of biometric information of the body part, motion information of the body part, or posture information of the body part.

[0010] An electronic device is provided. The electronic device may include a communication circuit. The electronic device may include a display. The electronic device may include a memory storing instructions and including one or more storage media. The electronic device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain sensor data regarding a body part of a user wearing a wearable device connected to the electronic device from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify an operation mode of the wearable device based on the body part identified based on the sensor data and a grip posture of the body part. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to change a screen displayed through the display based on a hovering input identified according to a movement of the body part wearing the wearable device in the identified operating mode.

[0011] A method performed by an electronic device is provided. The method may include an operation of acquiring sensor data regarding a body part of a user wearing a wearable device connected to the electronic device from the wearable device. The method may include an operation of identifying an operation mode of the wearable device based on the body part identified based on the sensor data and a grip posture of the body part. The method may include an operation of changing a screen displayed through the display based on a hovering input identified based on a movement of the body part wearing the wearable device in the identified operation mode.

[0012] A non-transitory computer-readable storage medium storing one or more programs is provided. The one or more programs may include instructions that, when executed by an electronic device including at least one processor, cause the electronic device to obtain sensor data about a body part of a user wearing a wearable device connected to the electronic device from the wearable device. The one or more programs may include instructions that, when executed by the electronic device including at least one processor, cause the electronic device to identify an operation mode of the wearable device based on the body part identified based on the sensor data and a grip posture of the body part. The one or more programs may include instructions that, when executed by the electronic device including at least one processor, cause the electronic device to change a screen displayed on the display based on a hovering input identified based on a movement of the body part wearing the wearable device in the identified operation mode.

[0013] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0014] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0015] Figure 2 illustrates a simplified block diagram of an electronic device and a wearable device.

[0016] Figure 3 illustrates examples of electronic devices and wearable devices.

[0017] Figure 4 illustrates an example of a grip posture of a body part of a user wearing a wearable device.

[0018] FIG. 5 illustrates a flowchart of an operation of an electronic device that changes a screen displayed by a display of the electronic device based on sensor data of the wearable device.

[0019] FIGS. 6A to 6D illustrate examples of changing a screen displayed by a display of an electronic device based on sensor data of a wearable device.

[0020] Figure 7 illustrates an example of a grip posture of a body part of a user wearing a wearable device.

[0021] FIG. 8 illustrates a flowchart of an operation of an electronic device that changes a screen displayed by a display of the electronic device based on sensor data of the wearable device.

[0022] FIGS. 9A to 9C illustrate examples of changing a screen displayed by a display of an electronic device based on sensor data of a wearable device.

[0023] FIG. 10 illustrates a flowchart of operations of an electronic device that controls a display of an electronic device based on sensor data of a wearable device.

[0024] FIG. 11 illustrates a flowchart of an operation of an electronic device for changing a screen displayed by a display of the electronic device based on sensor data of the wearable device.

[0025] Figure 12 illustrates a screen of an electronic device for setting an operation mode of a wearable device.

[0026] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0027] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0028] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled, but this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." A condition described as "more than" may be replaced with "more than," a condition described as "less than" may be replaced with "less than," and a condition described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of elements from A (including A) to B (including B). hereinafter, "C" and / or "D" mean at least one of "C" or "D," that is, including {"C", "D", "C" and "D"}.

[0029] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0030] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155),

[0031] The electronic device (101) may include a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may add one or more other components. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0032] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0033] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0034] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0035] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0036] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0037] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0038] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0039] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0040] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0041] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0042] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0044] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0045] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0046] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0047] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0048] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0049] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0050] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0051] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0052] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0053] Figure 2 illustrates a simplified block diagram of an electronic device and a wearable device.

[0054] Referring to FIG. 2, the electronic device (101) may include a processor (211), a memory (212), a communication circuit (213), a display (214), and an artificial intelligence module (215). For example, the processor (211), the memory (212), the communication circuit (213), the display (214), and the artificial intelligence module (215) may be electrically and / or operably coupled with each other by a communication bus. Hereinafter, the operative coupling of hardware components may mean that a direct or indirect connection is established between the hardware components, either wired or wireless, so that a second hardware component is controlled by a first hardware component among the hardware components. The artificial intelligence module (215) illustrated in FIG. 2 is illustrated as a hardware component, but the present disclosure is not limited thereto. For example, the artificial intelligence module (215) may correspond to a software component. Although the hardware components illustrated in FIG. 2 are illustrated based on different blocks, the present disclosure is not limited thereto. For example, some of the hardware components of the electronic device (101) illustrated in FIG. 2 (e.g., at least a portion of the processor (211), the memory (212), the communication circuit (213), the display (214), and / or the artificial intelligence module (215)) may be included in a single integrated circuit such as a system on chip (SoC) or a system in package (SIP). The type and number of hardware components included in the electronic device (101) are not limited to those illustrated in FIG. 2. For example, the electronic device (101) may include only some of the hardware components illustrated in FIG. 2.

[0055] In one embodiment, the processor (211) of the electronic device (101) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP), a microcontroller (MCU), and / or a neural processing unit (NPU). The number of processors (211) may be one or more. For example, the processor (211) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core. The processor (211) of FIG. 2 may be substantially identical to the content of the processor (120) of FIG. 1.

[0056] In one embodiment, the processor (211) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to individually and / or collectively perform the various functions described below in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, as well as situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor is capable of executing program instructions to accomplish or perform various functions.

[0057] In one embodiment, the memory (212) of the electronic device (101) may include hardware components for storing data and / or instructions input to and / or output from the processor (211). For example, the memory (212) may include volatile memory, such as random-access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multimedia card (eMMC).

[0058] In one embodiment, one or more instructions (or commands) representing operations and / or actions performed by the processor (211) of the electronic device (101) may be stored within the memory (212) of the electronic device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, being installed within the electronic device (101) may mean that one or more instructions provided in the form of an application are stored within the memory (212), and that one or more applications are stored in a format executable by the processor (211) of the electronic device (101). The specific details of the memory (212) of FIG. 2 may be substantially identically applied to the details of the memory (130) of FIG. 1.

[0059] In one embodiment, the communication circuit (213) of the electronic device (101) may include circuitry for supporting transmission and / or reception of electrical signals between the electronic device (101) and an external electronic device (e.g., a wearable device (201)) different from the electronic device (101). The communication circuit (213) may include at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (213) may support transmission and / or reception of electrical signals based on various types of communication means, such as Ethernet, Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). Specific details of the communication circuit (213) of FIG. 2 may be substantially identically applied to the communication module (109) and / or antenna module (197) of FIG. 1.

[0060] In one embodiment, the display (214) of the electronic device (101) may include a display panel, a touch sensor, and / or a processing circuit. In one embodiment, the display panel may be used to display visual information (e.g., an image, a screen, an object, a visual object, a user interface (UI), and / or a graphical user interface (GUI)). For example, the display (214) may change a screen displayed through the display (214) in response to a hovering input. For example, the display (214) may display indication points caused by the hovering input through the screen of the display (214). For example, the display (214) may change the position of a cursor (or pointer, mouse pointer) displayed through the display (214) in response to the hovering input. The specific details of the display (214) of FIG. 2 can be applied substantially identically to the details of the display module (160) of FIG. 1.

[0061] In one embodiment, the artificial intelligence module (215) may be a unit (function code, separate device, circuit, or set of instructions) for performing functions. For example, the artificial intelligence module (215) may output an operation mode (e.g., pencil mode or mouse mode) of the wearable device (201) based on sensor data acquired from the wearable device (201). For example, the artificial intelligence module (215) may output a grip posture of a body part wearing the wearable device (201) based on sensor data acquired from the wearable device (201). For example, the artificial intelligence module (215) may output an instruction according to a hovering input according to the movement of a body part of a user wearing the wearable device (201) based on a sensor data set and sensor data acquired from the wearable device (201). For example, the artificial intelligence module (215) may output a pen point (or hovering input coordinates, hovering input distance) for the pencil mode based on the sensor data set and the sensor data. For example, the artificial intelligence module (215) may output the positions of dots caused on the screen of the display (214) by the hovering input based on the sensor data set and the sensor data. For example, the artificial intelligence module (215) may output the sizes of dots caused on the screen of the display (214) by the hovering input based on the sensor data set and the sensor data. For example, the artificial intelligence module (215) may be referred to as an artificial intelligence model or an equivalent technical term. The specific details of the artificial intelligence module (215) of FIG. 2 may be substantially identically applied to the details of the auxiliary processor (123) of FIG. 1.

[0062] Referring to FIG. 2, the wearable device (101) may include a processor (221), a memory (222), a communication circuit (223), and a sensor (224). For example, the processor (221), the memory (222), the communication circuit (223), and the sensor (224) may be electrically and / or operably coupled with each other by a communication bus. Hereinafter, the hardware components being operably coupled may mean that a direct connection or an indirect connection is established between the hardware components, either wired or wireless, so that a second hardware component is controlled by a first hardware component among the hardware components. Although the hardware components illustrated in FIG. 2 are illustrated based on different blocks, the present disclosure is not limited thereto. For example, some of the hardware components of the wearable device (201) illustrated in FIG. 2 (e.g., at least a portion of the processor (221), the memory (222), the communication circuit (223), and / or the display (214)) may be included in a single integrated circuit such as a system on chip (SoC) or a system in package (SIP). The type and number of hardware components included in the wearable device (201) are not limited to those illustrated in FIG. 2. For example, the wearable device (201) may include only some of the hardware components illustrated in FIG. 2.

[0063] In one embodiment, the processor (221) of the wearable device (201) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP), a microcontroller (MCU), and / or a neural processing unit (NPU). The number of processors (221) may be one or more. For example, the processor (221) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core. The processor (221) of FIG. 2 may be substantially identical to the processor (120) of FIG. 1.

[0064] In one embodiment, the processor (221) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to individually and / or collectively perform the various functions described below in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, as well as situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor is capable of executing program instructions to accomplish or perform various functions.

[0065] In one embodiment, the memory (222) of the wearable device (201) may include hardware components for storing data and / or instructions input to and / or output from the processor (221). For example, the memory (222) may include volatile memory, such as random-access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multimedia card (eMMC).

[0066] In one embodiment, one or more instructions (or commands) representing operations and / or actions performed by the processor (221) of the electronic device (201) may be stored within the memory (222) of the wearable device (201). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, being installed within the wearable device (201) may mean that one or more instructions provided in the form of an application are stored within the memory (222), and that one or more applications are stored in a format executable by the processor (221) of the wearable device (201). The specific details of the memory (222) of FIG. 2 may be substantially identically applied to the details of the memory (130) of FIG. 1.

[0067] In one embodiment, the communication circuit (223) of the wearable device (201) may include circuitry for supporting transmission and / or reception of electrical signals between the wearable device (201) and the electronic device (101). The communication circuit (223) may include at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (213) may support transmission and / or reception of electrical signals based on various types of communication means, such as Ethernet, Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). Specific details of the communication circuit (223) of FIG. 2 may be substantially identically applied to the communication module (109) and / or antenna module (197) of FIG. 1.

[0068] In one embodiment, the sensor (224) of the wearable device (201) may include a first pressure sensor (225), a second pressure sensor (226), an inertial measurement unit (IMU) sensor (227), a heart rate sensor (228), and a fingerprint sensor (229). However, this is merely an example, and the present disclosure is not limited thereto. The sensor (224) of the wearable device (201) may include only some of the first pressure sensor (225), the second pressure sensor (226), the IMU sensor (227), the heart rate sensor (228), and the fingerprint sensor (229). In a non-limiting example, the sensor (224) of the wearable device (201) may further include a bone conduction sensor to prevent or reduce unintended actions from being performed due to movements of a body part that is not wearing the wearable device (201). For example, the first pressure sensor (225) of the wearable device (201) may be configured to measure pressure caused by contraction (or relaxation) of a flexor muscle of a body part of a user on which the wearable device (201) is worn. For example, the second pressure sensor (226) may be configured to measure pressure caused by contraction (or relaxation) of an extensor muscle of a body part of a user on which the wearable device (201) is worn. For example, the IMU sensor (227) may be configured to measure IMU data indicating an orientation, a moving direction, and / or a moving speed of the wearable device (201) worn on a body part of a user. The IMU sensor (227) may include an acceleration sensor, a gyro sensor, and a magnetic sensor. However, this is merely an example, and the present disclosure is not limited thereto. The IMU sensor (227) may include only some of the acceleration sensor, the gyro sensor, and the magnetic sensor. For example, the heart rate sensor (228) may be configured to measure the heart rate of a user wearing the wearable device (201).For example, the fingerprint sensor (229) may be configured to detect a fingerprint of a body part of a user wearing the wearable device (201).

[0069] Figure 3 illustrates examples of electronic devices and wearable devices.

[0070] Referring to FIG. 3, the electronic device (101) may correspond to the electronic device (101) of FIGS. 1 and 2. The electronic device (101) may include a display (214).

[0071] In one embodiment, the electronic device (101) may establish a connection with a wearable device (201). The wearable device (201) may be configured to be worn on at least a portion of the user's body (e.g., a finger). For example, the wearable device (201) may be worn on a portion of the user's body. For example, the wearable device (201) may be fastened to a portion of the user's body. For example, the wearable device (201) may be detachable from a portion of the user's body. For example, the wearable device (201) may have a shape corresponding to a portion of the user's body so as to be worn on a portion of the user's body. For example, the wearable device (201) may be worn by the user and thus come into contact with a portion of the user's body. For example, a wearable device (201) may be configured to acquire information about the user (e.g., sensor data) by being worn by the user through a part of the user's body. For example, the wearable device (201) may provide information about the user to an electronic device (101) connected to the wearable device (201).

[0072] In one embodiment, the wearable device (201) may include a housing (310) that includes a first side (311) facing a portion of a user's body (e.g., a finger) and a second side (312) opposite the first side (311). For example, the wearable device (201) may include a ring-shaped housing (310). Although FIG. 2 illustrates a ring-shaped wearable device (201), the present disclosure is not limited thereto. The shape of the wearable device (201) may include various shapes (e.g., square, oval) that may be worn on a portion of a user's body (e.g., a finger, a wrist, an earlobe).

[0073] In one embodiment, at least a portion of the first surface (311) may be in contact with a portion of the body of a user wearing the wearable device (201). For example, the first surface (311) may surround a portion of the body of a user wearing the wearable device (201). For example, the first surface (311) may cover a portion of the body of a user wearing the wearable device (201). For example, the first surface (311) may be configured to pressurize a portion of the body of a user wearing the wearable device (201), thereby fastening the wearable device (201) to a portion of the body of the user. For example, the first surface (311) may be deformable by a portion of the body of a user wearing the wearable device (201). For example, the first surface (311) may be referred to as the inner circumference surface of the housing (310).

[0074] In one embodiment, the second surface (312) may form the exterior of the wearable device (201) together with the first surface (311). For example, the second surface (312) may form a ring-shaped housing (310) together with the first surface (311). For example, the second surface (312) may be a surface spaced apart from a part of the body of a user wearing the wearable device (201). For example, the second surface (312) may be exposed to the outside when the wearable device (201) is worn by the user. For example, the second surface (312) may be composed of at least one of titanium, stainless steel, and ceramic. However, the present disclosure is not limited thereto. The second surface (312) may be composed of a material for protection against external impact and / or scratches. For example, the second surface (312) may be referred to as the outer circumference surface of the housing (310).

[0075] In one embodiment, the first side (311) may be composed of the same and / or similar material as the second side (312). For example, at least a portion of the first side (311) may be composed of at least one of a molding material, transparent plastic, metal, and / or glass. However, the present disclosure is not limited thereto.

[0076] In one embodiment, the wearable device (201) may include one or more components between the first side (311) and the second side (312). For example, the wearable device (201) may include a processor (221), a memory (222), a communication circuit (223), and / or a sensor (224) between the first side (311) and the second side (312). However, the present disclosure is not limited thereto. For example, the wearable device (201) may include only some of the processor (221), the memory (222), the communication circuit (223), and the sensor (224).

[0077] In one embodiment, the wearable device (201) may include a hole (320) formed by the first surface (311) for passing a portion of the body of a user wearing the wearable device (201). For example, the hole (320) may be penetrated by a portion of the body of a user wearing the wearable device (201). The wearable device (201) may be configured to be coupled to a portion of the body of the user by including a hole (320) configured to pass a portion of the body of the user.

[0078] Fig. 4 illustrates an example of a grip posture of a body part of a user wearing a wearable device. Fig. 4 illustrates that sensor data acquired by a sensor (224) of a wearable device (201) may differ depending on the grip posture of the body part of the user in pencil mode (or air pencil mode). Although Fig. 4 illustrates an input means (406) to illustrate a grip posture formed by body parts of the user, the input means (406) is illustrated to illustrate an operation according to the present disclosure. The grip postures illustrated in Fig. 4 are formed by body parts of the user without the input means (406).

[0079] Referring to FIG. 4, the wearable device (201) may be worn on a body part of the user (e.g., the index finger) (402). However, this is merely an example, and the present disclosure is not limited thereto. For example, the wearable device (201) may be worn on one of the body parts of the user (e.g., the thumb, the index finger, the middle finger, the ring finger, and the little finger).

[0080] In one embodiment, the grip posture (or posture, grip style, hand posture, holding posture, grip method, grasp style) of a user wearing the wearable device (201) may cause pressure on a portion of the wearable device (201). For example, the body part (402) of the user wearing the wearable device (201) may include flexor muscles and extensor muscles. For example, the flexor muscles may be located on the inner side (e.g., toward the palm) of the user's body part (402). For example, the extensor muscles may be located on the outer side (e.g., toward the back of the hand) of the user's body part (402). For example, muscle contraction or relaxation according to the user's grip posture may cause pressure on a portion of the wearable device (201) that comes into contact with the user's body part (402). Meanwhile, although FIG. 4 illustrates a first grip posture (410) and a second grip posture (420), the present disclosure is not limited thereto. For example, the grip posture may further include any posture according to the user or the user's status.

[0081] For example, in a first grip posture (410) of a user wearing a wearable device (201), the flexor muscles of the user's body part (402) may contract to control the input means (406). For example, in a first grip posture (410) of a user wearing a wearable device (201), the extensor muscles of the user's body part (402) may relax to control the input means (406). The wearable device (201) may obtain pressure data caused by the contraction of the flexor muscles of the body part (402) and the relaxation of the extensor muscles in the first grip posture (410) using pressure sensors. For example, the first pressure sensor (225) of the wearable device (201) may obtain first pressure data representing the pressure caused by the contraction of the flexor muscles of the body part (402) in the first grip posture (410). The first pressure sensor (225) of the wearable device (201) may be positioned at a portion (404) of the first face (311) of the wearable device (201) that contacts the flexor muscle of the body part (402) to obtain first pressure data. For example, the second pressure sensor (226) of the wearable device (201) may be positioned at a portion (405) of the first face (311) of the wearable device (201) that contacts the flexor muscle of the body part (402) to obtain second pressure data.

[0082] For example, in a second grip posture (420) of a user wearing a wearable device (201), the flexor muscles of the user's body part (402) may contract to control the input means (406). For example, in a second grip posture (420) of a user wearing a wearable device (201), the extensor muscles of the user's body part (402) may relax to control the input means (406). The wearable device (201) may obtain pressure data caused by the contraction of the flexor muscles of the body part (402) and the relaxation of the extensor muscles in the second grip posture (420) using one or more pressure sensors. For example, the first pressure sensor (225) of the wearable device (201) may obtain third pressure data representing the pressure caused by the contraction of the flexor muscles of the body part (402) in the second grip posture (420). The first pressure sensor (225) of the wearable device (201) may be positioned at a portion (404) of the first face (311) of the wearable device (201) that contacts the flexor muscle of the body part (402) to obtain third pressure data. For example, the second pressure sensor (226) of the wearable device (201) may be positioned at a portion (405) of the first face (311) of the wearable device (201) that contacts the flexor muscle of the body part (402) to obtain fourth pressure data.

[0083] For example, in the first grip posture (410), the input means (406) may be supported by the user's body parts (401) and (402). On the other hand, in the second grip posture (420), the input means (406) may be supported by the user's body parts (401), (402), and (403). Therefore, the pressure data acquired in the first grip posture (410) and the pressure data acquired in the second grip posture (420) may be different. For example, the first pressure data indicating the contraction of the flexor muscle of the body part (402) in the first grip posture (410) may be different from the third pressure data indicating the contraction of the flexor muscle of the body part (402) in the second grip posture (420). For example, the second pressure data indicating relaxation of the extensor muscles of the body part (402) in the first grip posture (410) may be different from the fourth pressure data indicating relaxation of the extensor muscles of the body part (402) in the second grip posture (420). Meanwhile, although FIG. 4 illustrates obtaining pressure data using the first pressure sensor (225) and the second pressure sensor (226), the present disclosure is not limited thereto. For example, the wearable device (201) may include one pressure sensor. In another example, the wearable device (201) may include at least three sensors.

[0084] In one embodiment, a grip posture (or posture, grip style, hand posture, holding posture, grip method, holding style) of a user wearing a wearable device (201) may cause a change in the orientation of the wearable device (201). For example, the orientation of the wearable device (201) in a first grip posture (410) may be different from the orientation of the wearable device (201) in a second grip posture (420). For example, in the first grip posture (410) of the user wearing the wearable device (201), first IMU data acquired by an IMU sensor (227) may be different from second IMU data acquired by the IMU sensor (227) in the second grip posture (420).

[0085] As described above, sensor data obtained by the control of the input means (430) obtained by the sensor (224) of the wearable device (201) worn on the user's body part (402) (e.g., the index finger) may vary depending on the user's grip posture. Since the sensor data obtained by the sensor (224) of the wearable device (201) may vary depending on the user's grip posture, an artificial intelligence model may be used to identify a hovering input based on the sensor data. Below, a method and device for controlling the operation of an electronic device (101) according to a hovering input using an artificial intelligence model are described.

[0086] FIG. 5 illustrates a flowchart of operations of an electronic device that changes a screen displayed on a display of the electronic device based on sensor data of a wearable device. The operations of FIG. 5 may be performed by the electronic device (101) of FIGS. 1 and 2 . For example, at least some of the operations may be controlled by the processor (211) of the electronic device (101). In the following description, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed. For example, at least two operations may be performed in parallel.

[0087] Referring to FIG. 5, in operation 501, the electronic device (101) can obtain sensor data about a body part (e.g., a finger) of a user wearing a wearable device (201) connected to the electronic device (101) from the wearable device (201).

[0088] In one embodiment, the wearable device (201) may be worn on a body part of the user (e.g., an index finger). However, this is merely an example, and the present disclosure is not limited thereto. For example, the wearable device (201) may be worn on one of the body parts of the user (e.g., a thumb, an index finger, a middle finger, a ring finger, and a litter finger).

[0089] In one embodiment, the sensor data may include data caused by a gesture (e.g., drag) of a body part on which the wearable device (201) is worn. For example, the sensor data may include first pressure data acquired by a first pressure sensor (225) of the wearable device (201), second pressure data acquired by a second pressure sensor (226) of the wearable device (201), IMU data acquired by an inertial measurement unit (IMU) sensor (227) of the wearable device (201), heart rate data acquired by a heart rate sensor (228), and fingerprint data acquired by a fingerprint sensor (229). The IMU data acquired by the IMU sensor (227) may include gyro data, acceleration data, and / or magnetism data for the wearable device (201). However, this is merely an example, and the present disclosure is not limited thereto. The sensor data may include only some of the first pressure data, the second pressure data, the gyro data, the acceleration data, the magnetic data, the heart rate data, and the fingerprint data. For example, the first pressure data, the second pressure data, the heart rate data, and the fingerprint data may be referenced as biometric information about a body part of a user wearing the wearable device (201). For example, the IMU data may be referenced as motion information and / or posture information about a body part of a user wearing the wearable device (201). In a non-limiting example, the sensor data may further include bone conduction data acquired from a bone conduction sensor of the wearable device (201) to prevent or reduce unintended actions from being performed due to movements of a body part not wearing the wearable device (201).

[0090] For example, the first pressure data may represent a pressure caused by contraction (or relaxation) of a flexor muscle of a body part of a user wearing the wearable device (201). For example, the second pressure data may represent a pressure caused by relaxation (or contraction) of an extensor muscle of a body part of a user wearing the wearable device (201). For example, the IMU data may represent an orientation of a wearable device (201) worn on a body part of a user, a direction of movement of the wearable device (201), and / or a speed of movement of the wearable device (201). For example, the heart rate data may represent a heart rate of a user wearing the wearable device (201). For example, the fingerprint data may represent fingerprint data of a body part of a user wearing the wearable device (201).

[0091] In operation 502, the electronic device (101) may identify a sensor data set for a pencil mode based on the user's status information. The pencil mode may refer to a mode in which dots (or display points, points) are displayed on a screen of a display (214) of the electronic device (101) based on a hovering input by a gesture (or motion) of a body part of a user wearing the wearable device (201).

[0092] In one embodiment, the electronic device (101) can identify user status information based on sensor data acquired from the wearable device (201). For example, the status information can indicate a grip posture (e.g., the first grip posture (410) of FIG. 4) corresponding to the sensor data among grip postures (or postures, grip styles, hand postures, holding postures, grip methods, grasp styles). For example, since different grip postures cause different sensor data, the grip posture can indicate the user's status. For example, the electronic device (101) can identify a grip posture corresponding to the sensor data using an artificial intelligence model. For example, input data of the artificial intelligence model can include sensor data acquired from the wearable device (201). In one example, input data of an artificial intelligence model for identifying a grip posture may include IMU data (e.g., gyro data) representing posture information of a user wearing a wearable device (201). For example, output data of the artificial intelligence model may represent the grip posture of a body part wearing the wearable device (201).

[0093] In one embodiment, the electronic device (101) may identify a sensor data set for the pencil mode based on the state information. For example, the sensor data set may include a user history data set regarding grip posture and / or a server data set regarding grip posture obtained from a server (e.g., cloud).

[0094] For example, the user history data set may include sensor data collected (or accumulated, obtained, trained) by the electronic device (101) with respect to a grip posture and data about operations performed by the electronic device (101) in response to the collected sensor data. For example, the data about operations performed by the electronic device (101) may include location information of fan points identified based on the collected sensor data, location information of dots identified based on the collected sensor data, and size information of dots identified based on the collected sensor data (e.g., the number of pixels of the display (214) corresponding to the dot). In one example, the user history data set may include first sensor data collected by the electronic device (101) from the wearable device (201) regarding a grip posture (e.g., the first grip posture (410) of FIG. 4), data regarding actions performed by the electronic device (101) in response to the first sensor data, second sensor data collected by the electronic device (101) from the wearable device (201) regarding the grip posture, and data regarding actions performed by the electronic device (101) in response to the second sensor data.

[0095] For example, the server data set may include sensor data collected by another user regarding a grip posture and data regarding actions performed by an electronic device of the other user in response to the collected sensor data. For example, the data regarding actions performed by an electronic device of the other user may include location information of a fan point identified based on the collected sensor data, location information of dots identified based on the collected sensor data, and size information of dots identified based on the collected sensor data. In one example, the server data set may include first sensor data collected by a first user regarding a grip posture (e.g., the first grip posture (410) of FIG. 4), data regarding actions performed by a first electronic device of the first user in response to the first sensor data, second sensor data collected by a second user regarding the grip posture, and data regarding actions performed by a second electronic device of the second user in response to the second sensor data.

[0096] In operation 503, the electronic device (101) can identify the positions of dots caused on the screen of the display (214) by a hovering input according to the movement of a body part wearing the wearable device (201), based on the sensor data set for the pencil mode and the sensor data acquired from the wearable device (201).

[0097] In one embodiment, the electronic device (101) can identify a pen point (or, hovering input coordinate, hovering input distance) for a hovering input. For example, the pen point may refer to a virtual point at which a hovering input by a body part wearing the wearable device (201) is identified. For example, the pen point may be used to identify the positions of dots caused on the screen of the display (214) by a hovering input according to the movement of the body part wearing the wearable device (201). For example, the electronic device (101) can identify the pen point for a hovering input using an artificial intelligence model. Input data of the artificial intelligence model may include a sensor data set for the pencil mode and sensor data acquired from the wearable device (201). Output data of the artificial intelligence model may indicate the distance and direction at which the hovering input is identified.

[0098] For example, a fan point for a hovering input may be associated with a number of dots caused on a screen of a display (214) of an electronic device (101) by the hovering input. In one example, the number of dots caused by a movement of a body part spaced a first distance from the fan point may be greater than the number of dots caused by a movement of a body part spaced a second distance shorter than the first distance from the fan point. However, this is merely an example, and the present disclosure is not limited thereto.

[0099] For example, a pan point for a hovering input may be associated with a distance between dots caused on a screen of a display (214) of an electronic device (101) by the hovering input. In one example, a distance between dots caused by a movement of a body part spaced a first distance from the pan point may be longer than a distance between dots caused by a movement of a body part spaced a second distance shorter than the first distance from the pan point. However, this is merely an example, and the present disclosure is not limited thereto.

[0100] In one example, the electronic device (101) may set the fan point further away based on the sensor data set and the sensor data as the speed of the body part wearing the wearable device (201) decreases. In one example, the electronic device (101) may set the fan point closer based on the sensor data set and the sensor data as the speed of the body part increases. In one example, the electronic device (101) may set the fan point closer based on the sensor data set and the sensor data as the pen pressure of the body part increases. In one example, the electronic device (101) may set the fan point further away based on the sensor data set and the sensor data as the pen pressure of the body part decreases. In one example, the electronic device (101) may set the fan point closer based on the sensor data set and the sensor data as the inclination of the body part increases. In one example, the electronic device (101) may set the fan point further away from the body part as the inclination of the body part decreases based on the sensor data set and the sensor data. However, this is merely an example, and the present disclosure is not limited to the above-described example.

[0101] For example, the electronic device (101) may use an artificial intelligence model to identify the positions of dots caused on the screen of the display (214) of the electronic device (101) by a hovering input according to the movement of a body part of a user wearing the wearable device (201). The input data of the artificial intelligence model may include a sensor data set for the pencil mode, sensor data acquired from the wearable device (201), and / or position information of the fan point. The output data of the artificial intelligence model may indicate the positions of dots caused on the screen of the display (214) of the electronic device (101) by the hovering input.

[0102] In operation 504, the electronic device (101) can change the screen displayed through the display (214) based on the positions of the dots. For example, the electronic device (101) can display dots in pixel units at positions on the screen of the display (214). The dots displayed on the screen of the display (214) can correspond to the movement of a body part of a user wearing the wearable device (201).

[0103] In one embodiment, the electronic device (101) may change the size of a dot displayed on the screen of the display (214) using an artificial intelligence model. The size of the dot may correspond to the number of pixels of the display (214) on which the dot is displayed. For example, input data of the artificial intelligence model may include a sensor data set and sensor data. For example, output data of the artificial intelligence model may indicate the number of pixels corresponding to the dot.

[0104] In one example, the electronic device (101) may increase the number of pixels of the display (214) on which dots are displayed based on the sensor data set and the sensor data as the pressure applied to the body part of the user wearing the wearable device (201) increases. In one example, the electronic device (101) may decrease the number of pixels of the display (214) on which dots are displayed based on the sensor data set and the sensor data as the pressure applied to the body part decreases. In one example, the electronic device (101) may decrease the number of pixels of the display (214) on which dots are displayed based on the sensor data set and the sensor data as the inclination of the body part increases. In one example, the electronic device (101) may increase the number of pixels of the display (214) on which dots are displayed based on the sensor data set and the sensor data as the inclination of the body part decreases. However, this is merely an example, and the present disclosure is not limited to the above-described examples. In other words, in the above-described example, the electronic device (101) may increase the number of pixels of the display (214) on which dots are displayed as the pressure applied by the body part decreases. In addition, in the above-described example, the electronic device (101) may increase the number of pixels of the display (214) on which dots are displayed as the inclination of the body part increases. In addition, in the above-described example, the electronic device (101) may decrease the number of pixels of the display (214) on which dots are displayed as the inclination of the body part decreases.

[0105] In one embodiment, the electronic device (101) may learn, using an artificial intelligence model, location information of a fan point according to sensor data acquired from a wearable device (201), size information of dots according to the sensor data (e.g., information on the number of pixels corresponding to a dot), and location information of dots on a display (214) according to the sensor data. For example, the location information of a fan point according to the sensor data, the size information of dots according to the sensor data, and the location information of dots according to the sensor data may be referenced as a sensor data pattern.

[0106] In a non-limiting example, the electronic device (101) may generate an output signal corresponding to a hovering input to display dots on a screen of a display of another electronic device (102). For example, the output signal may include position information of the dots on the display according to sensor data. For example, the electronic device (101) may transmit the generated output signal to the other electronic device (102). For example, the output signal may cause the other electronic device (102) to display dots on the screen of its display according to the position information.

[0107] Figures 6A to 6D illustrate examples of changing the screen displayed by the display of an electronic device based on sensor data of a wearable device. Figures 6A to 6D illustrate an input means (e.g., a pen) to explain grip postures formed by parts of a user's body, but the input means is illustrated to explain operations according to the present disclosure. The grip postures illustrated in Figures 6A to 6C are formed by parts of a user's body without an input means.

[0108] In FIG. 6A, a pen point (or, hovering input coordinate, hovering input distance) for a hovering input is described. The pen point may refer to a virtual point where a hovering input by a body part wearing the wearable device (201) is identified. The pen point may be used to identify the positions of dots caused on the screen of the display (214) of the electronic device (101) by the hovering input.

[0109] Referring to FIG. 6A, in one example, a first fan point for a pencil mode may be spaced a first distance from a body part of a user wearing a wearable device (201). Depending on movement of the body part, a position (or coordinate) of the first fan point may change from a first location (601) to a second location (602). The electronic device (101) may display dots (607) at positions on a screen of the display (214) corresponding to a trajectory of the first fan point using sensor data acquired (or measured) by the wearable device (201) while the position of the first fan point changes from the first location (601) to the second location (602).

[0110] In one example, the second fan point for the pencil mode may be spaced a second distance from a body part of a user wearing the wearable device (201) that is further than the first distance. Depending on the movement of the body part, the location (or coordinates) of the second fan point may change from a third location (603) to a fourth location (604). The electronic device (101) may display dots (608) at locations on the screen of the display (214) corresponding to the trajectory of the second fan point using sensor data acquired (or measured) by the wearable device (201) while the location of the second fan point changes from the third location (603) to the fourth location (604). In one example, the number of dots corresponding to the trajectory of the first fan point may be less than the number of dots corresponding to the trajectory of the second fan point. In one example, the distance between dots by the first fan point may be shorter than the distance between dots by the second fan point.

[0111] In one example, the third fan point for the pencil mode may be spaced a third distance closer than the first distance from a body part of a user wearing the wearable device (201). Depending on the movement of the body part, the position of the third fan point may change from a fifth position (605) to a sixth position (606). The electronic device (101) may display dots (609) at positions on the screen of the display (214) corresponding to the trajectory of the third fan point using sensor data acquired (or measured) by the wearable device (201) while the position of the third fan point changes from the fifth position (605) to the sixth position (606).

[0112] As described above, the positions of dots caused on the screen of the display (214) of the electronic device (101) by the hovering input may differ depending on the fan point. For example, dots (608) according to the second fan point and dots (609) according to the third fan point may differ from dots (607) intended by the user. In order to display the dots (607) intended by the user on the screen of the display (214), the electronic device (101) may identify the first fan point using an artificial intelligence model. For example, input data of the artificial intelligence model may include sensor data acquired (or measured) by the wearable device (201) and / or a sensor data set for pencil mode. The sensor data set may include a user history data set stored in the electronic device (101) and / or a server data set acquired from a server (e.g., cloud). The user history data set may include sensor data collected by the electronic device (101) regarding the grip posture of a user's body part (e.g., an index finger). The server data set may include sensor data collected by other users regarding the grip posture of a body part (e.g., an index finger). For example, the output data of the artificial intelligence model may indicate the position and / or direction of the first fan point. The electronic device (101) may identify the first fan point based on the speed, pressure, and / or angle of the user's body part by using the artificial intelligence model.

[0113] In FIG. 6B, an example of changing a screen displayed through a display (214) of an electronic device (101) according to a hovering input caused by a movement of a body part (402) of a user wearing a wearable device (201) is described. According to the movement of the body part (402) of a user wearing a wearable device (201), the position of a fan point for the hovering input may change from a position (611) to a position (612). While the position of the fan point changes from a position (611) to a position (612), the sensor (224) of the wearable device (201) may obtain sensor data for the body part (402). The electronic device (101) may obtain sensor data for the body part (402) from the wearable device (201). The electronic device (101) can identify a grip posture (e.g., the first grip posture (410) of FIG. 4) for the body part (402) based on sensor data acquired from the wearable device (201). In one example, an artificial intelligence model can be used to identify the grip posture. The electronic device (101) can identify a sensor data set associated with the grip posture for the body part (402). For example, the sensor data set can include a user history data set for the grip posture and a server data set for the grip posture acquired from a server (e.g., cloud). The electronic device (101) can display dots corresponding to the movement of the body part (402) (or the movement of the fan point) on the screen of the display (214) based on the sensor data set and the sensor data acquired from the wearable device (201). In one example, an artificial intelligence model can be used to identify the positions of the dots corresponding to the movement of the body part (402). In one example, an artificial intelligence model may be used to identify the number of pixels in a display (214) corresponding to a dot.For example, the trajectory (616) between dots (614) and dots (615) displayed on the screen of the display (214) may correspond to the trajectory (613) between the positions (611) and (612) of the fan point for the hovering input.

[0114] In FIG. 6C, an example of changing a screen displayed through a display (214) of an electronic device (101) according to a hovering input caused by a movement of a body part (402) of a user wearing a wearable device (201) is described. According to the movement of the body part (402) of a user wearing a wearable device (201), the position of a fan point for the hovering input may change from a position (621) (e.g., a first position) to a position (622) (e.g., a second position). While the position of the fan point changes from a position (621) to a position (622), a sensor (224) of the wearable device (201) may obtain sensor data for the body part (402). The electronic device (101) may obtain sensor data for the body part (402) from the wearable device (201). The electronic device (101) can identify a grip posture (e.g., the first grip posture (410) of FIG. 4) for the body part (402) based on sensor data acquired from the wearable device (201). In one example, an artificial intelligence model can be used to identify the grip posture. The electronic device (101) can identify a sensor data set associated with the grip posture for the body part (402). For example, the sensor data set can include a user history data set for the grip posture and a server data set for the grip posture acquired from a server. The electronic device (101) can display dots corresponding to the movement of the body part (402) (or the movement of the fan point) on the screen of the display (214) based on the sensor data set and the sensor data acquired from the wearable device (201). In one example, an artificial intelligence model can be used to identify the positions of the dots corresponding to the movement of the body part (402). For example, the trajectory (626) between dots (624) and dots (625) displayed on the screen of the display (214) may correspond to the trajectory (623) between the positions (621) and (622) of the fan point for the hovering input.The electronic device (101) can change the size of dots displayed on the screen of the display (214) based on the sensor data set and the sensor data. The size of the dots can correspond to the number of pixels of the display (214) on which the dots are displayed. In one example, the electronic device (101) can reduce the size of the dots displayed on the screen of the display (204) by identifying a decrease in pen pressure by a body part (402) using an artificial intelligence model while the position of the fan point changes from position (621) to position (622). In one example, the user's posture can change from a first posture (630) to a second posture (640) while the position of the fan point changes from position (612) to position (622). The electronic device (101) can reduce the size of dots displayed on the screen of the display (204) by identifying an increase in the inclination of the wearable device (201) using an artificial intelligence model while the position of the fan point changes from position (621) to position (622). However, this is merely an example and the present disclosure is not limited thereto.

[0115] Referring to FIG. 6d, in one example, the wearable device (201) may be worn on the user's index finger, and the wearable device (631) may be worn on the user's thumb. The contents of the wearable device (201) of FIG. 2 may be substantially identically applied to the wearable device (631).

[0116] For example, the electronic device (101) may receive (or acquire) first sensor data from the wearable device (201). The electronic device (101) may identify an index finger on which the wearable device (201) is worn based on the first sensor data. The electronic device (101) may acquire (or identify) a first sensor data set according to a grip posture of the index finger based on the first sensor data. The first sensor data set may include a first user history data set according to the grip posture of the index finger stored in the electronic device (101) and / or a first server data set according to the grip posture of the index finger acquired from a server (e.g., cloud). The first user history data set may include sensor data according to the grip posture of the user's index finger collected by the electronic device (101). The first server data set may include sensor data collected by another user regarding the grip posture of the index finger.

[0117] For example, the electronic device (101) may receive (or acquire) second sensor data from the wearable device (631). The electronic device (101) may identify the thumb on which the wearable device (631) is worn based on the second sensor data. The electronic device (101) may acquire (or identify) a second sensor data set according to the grip posture of the thumb based on the second sensor data. The second sensor data set may include a second user history data set according to the grip posture of the thumb stored in the electronic device (101) and / or a second server data set according to the grip posture of the thumb acquired from a server. The second user history data set may include sensor data according to the grip posture of the user's thumb collected by the electronic device (101). The second server data set may include sensor data collected by another user regarding the grip posture of the thumb. The electronic device (101) can display dots on the screen of the display (214) of the electronic device (101) by a hovering input according to a fan point (632) identified based on the first sensor data, the second sensor data, the first sensor data set, and / or the second sensor data set.

[0118] Fig. 7 illustrates an example of a grip posture of a body part of a user wearing a wearable device. Fig. 7 illustrates that sensor data acquired by a sensor (224) of a wearable device (201) may differ depending on the grip posture of a body part of the user in mouse mode (or air mouse mode). Although Fig. 7 illustrates an input means (e.g., a mouse) to illustrate a grip posture formed by body parts of the user, the input means is illustrated to illustrate an operation according to the present disclosure. The grip postures illustrated in Fig. 7 are formed by body parts of the user without an input means.

[0119] Referring to FIG. 7, the wearable device (201) may be worn on a body part of a user (e.g., an index finger). However, this is merely an example, and the present disclosure is not limited thereto. For example, the wearable device (201) may be worn on one of the body parts of a user (e.g., a thumb, an index finger, a middle finger, a ring finger, and a little finger).

[0120] In one embodiment, the grip posture (or posture, grip style, hand posture, holding posture, grip method, grasp style) of a user wearing the wearable device (201) may cause pressure on a portion of the wearable device (201). For example, the body part (701) of the user wearing the wearable device (201) may include flexor muscles and extensor muscles. For example, the flexor muscles may be located on the inner side (e.g., toward the palm) of the user's body part (701). For example, the extensor muscles may be located on the outer side (e.g., toward the back of the hand) of the user's body part (701). For example, muscle contraction or relaxation according to the user's grip posture may cause pressure on a portion of the wearable device (201) that comes into contact with the user's body part (701). Meanwhile, FIG. 7 illustrates a first grip posture (710) and a second grip posture (720), but the present disclosure is not limited thereto. For example, the grip posture may further include any posture according to the user or the user's status.

[0121] For example, in a first grip posture (710) of a user wearing a wearable device (201), the flexor muscles of the user's body part (701) may contract to control the input means (702). For example, in a first grip posture (710) of a user wearing a wearable device (201), the extensor muscles of the user's body part (701) may relax to control the input means (702). The wearable device (201) may obtain pressure data representing the pressure caused by the contraction of the flexor muscles of the body part (701) and the relaxation of the extensor muscles in the first grip posture (710) using pressure sensors. For example, the first pressure sensor (225) of the wearable device (201) may obtain first pressure data representing the pressure caused by the contraction of the flexor muscles of the body part (701) in the first grip posture (710). The first pressure sensor (225) of the wearable device (201) may be positioned at a portion (703) of the first face (311) of the wearable device (201) that contacts the flexor muscle of the body part (701) to obtain first pressure data. For example, the second pressure sensor (226) of the wearable device (201) may be positioned at a portion (704) of the first face (311) of the wearable device (201) that contacts the flexor muscle of the body part (701) to obtain second pressure data.

[0122] For example, in a second grip posture (720) of a user wearing a wearable device (201), the flexor muscles of the user's body part (701) may contract to control the input means (702). For example, in a second grip posture (720) of a user wearing a wearable device (201), the extensor muscles of the user's body part (701) may relax to control the input means (702). The wearable device (201) may obtain pressure data representing the pressure caused by the contraction of the flexor muscles of the body part (701) and the relaxation of the flexor muscles of the body part (701) in the second grip posture (720) using one or more pressure sensors. For example, the first pressure sensor (225) of the wearable device (201) may obtain third pressure data representing the pressure caused by the contraction of the flexor muscles of the body part (701) in the second grip posture (720). The first pressure sensor (225) of the wearable device (201) may be positioned at a portion (703) of the first face (311) of the wearable device (201) that contacts the flexor muscle of the body part (701) to obtain third pressure data. For example, the second pressure sensor (226) of the wearable device (201) may be positioned at a portion (704) of the first face (311) of the wearable device (201) that contacts the flexor muscle of the body part (701) to obtain fourth pressure data. Meanwhile, although FIG. 7 illustrates obtaining pressure data using a first pressure sensor (225) and a second pressure sensor (226), the present disclosure is not limited thereto. For example, the wearable device (201) may include one pressure sensor. In another example, the wearable device (201) may include at least three sensors.

[0123] For example, the posture of the body part (701) in the first grip posture (710) may be different from the posture of the body part (701) in the second grip posture (720). Therefore, the pressure caused by the user's body part (701) to a part of the wearable device (201) in the first grip posture (710) may be different from the pressure caused by the user's body part (701) to a part of the wearable device (201) in the second grip posture (720). For example, the first pressure data representing the pressure caused by the contraction of the flexor muscles of the body part (701) in the first grip posture (710) may be different from the third pressure data representing the pressure caused by the contraction of the flexor muscles of the body part (701) in the second grip posture (720). For example, in the first grip posture (710), the second pressure data representing the pressure caused by the relaxation of the extensor muscles of the body part (701) may be different from the fourth pressure data representing the pressure caused by the relaxation of the extensor muscles of the body part (701) in the second grip posture (720).

[0124] In one embodiment, a grip posture (or posture, grip style, hand posture, holding posture, grip method, holding style) of a user wearing a wearable device (201) may cause a change in the orientation of the wearable device (201). For example, the orientation of the wearable device (201) in a first grip posture (710) may be different from the orientation of the wearable device (201) in a second grip posture (720). For example, first inertial measurement unit (IMU) data acquired by an IMU sensor (227) of the wearable device (201) in a first grip posture (410) may be different from second IMU data acquired by an IMU sensor (227) of the wearable device (201) in a second grip posture (420).

[0125] As described above, the sensor data acquired by the sensor (224) of the wearable device (201) worn on the user's body part (701) may vary depending on the user's grip posture. Since the sensor data acquired by the sensor (224) of the wearable device (201) may vary depending on the user's grip posture, an artificial intelligence model may be used to control the operation of the electronic device (101) according to the hovering input based on the sensor data. Below, a method and device for controlling the operation of the electronic device (101) according to the hovering input using an artificial intelligence model are described.

[0126] FIG. 8 illustrates a flowchart of operations of an electronic device that changes a screen displayed on a display of the electronic device based on sensor data of a wearable device. The operations of FIG. 8 may be performed by the electronic device (101) of FIGS. 1 and 2 . For example, at least some of the operations may be controlled by the processor (211) of the electronic device (101). In the following description, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed. For example, at least two operations may be performed in parallel.

[0127] Referring to FIG. 8, in operation 801, the electronic device (101) can obtain sensor data about a body part (e.g., a finger) of a user wearing a wearable device (201) connected to the electronic device (101) from the wearable device (201).

[0128] In one embodiment, the wearable device (201) may be worn on a body part of the user (e.g., an index finger). However, this is merely an example, and the present disclosure is not limited thereto. For example, the wearable device (201) may be worn on one of the body parts of the user (e.g., a thumb, an index finger, a middle finger, a ring finger, and a litter finger).

[0129] In one embodiment, the sensor data may include data caused by a gesture (e.g., drag, click) of a body part on which the wearable device (201) is worn. For example, the sensor data may include first pressure data acquired by a first pressure sensor (225) of the wearable device (201), second pressure data acquired by a second pressure sensor (226) of the wearable device (201), IMU data acquired by an inertial measurement unit (IMU) sensor (227) of the wearable device (201), heart rate data acquired by a heart rate sensor (228), and fingerprint data acquired by a fingerprint sensor (229). The IMU data acquired by the IMU sensor (227) may include gyro data, acceleration data, and / or magnetism data for the wearable device (201). However, this is merely an example, and the present disclosure is not limited thereto. The sensor data may include only some of the first pressure data, the second pressure data, the gyro data, the acceleration data, the magnetic data, the heart rate data, and the fingerprint data. For example, the first pressure data, the second pressure data, the heart rate data, and the fingerprint data may be referenced as biometric information about a body part of a user wearing the wearable device (201). For example, the IMU data may be referenced as motion information and / or posture information about a body part of a user wearing the wearable device (201). In a non-limiting example, the sensor data may further include bone conduction data acquired from a bone conduction sensor of the wearable device (201) to prevent or reduce unintended actions from being performed due to movements of a body part not wearing the wearable device (201).

[0130] For example, the first pressure data may represent a pressure caused by contraction (or relaxation) of a flexor muscle of a body part of a user wearing the wearable device (201). For example, the second pressure data may represent a pressure caused by relaxation (or contraction) of an extensor muscle of a body part of a user wearing the wearable device (201). For example, the IMU data may represent an orientation of a wearable device (201) worn on a body part of a user, a direction of movement of the wearable device (201), and / or a speed of movement of the wearable device (201). For example, the heart rate data may represent a heart rate of a user wearing the wearable device (201). For example, the fingerprint data may represent fingerprint data of a body part of a user wearing the wearable device (201).

[0131] In operation 802, the electronic device (101) may identify a sensor data set for a mouse mode according to the user's status information. The mouse mode may refer to a mode that controls a cursor (or pointer, mouse pointer) displayed on a screen of a display (214) of the electronic device (101) according to a hovering input by a gesture (or motion) of a body part of a user wearing the wearable device (201).

[0132] In one embodiment, the electronic device (101) may identify user status information based on sensor data acquired from the wearable device (201). For example, the status information may indicate a grip posture (e.g., the first grip posture (710) of FIG. 7) corresponding to the sensor data among grip postures (or postures, grip styles, hand postures, holding postures, grip methods, grasp styles) and a heart rate of the user wearing the wearable device (201). For example, since different grip postures cause different sensor data, the grip posture may indicate the user's status. For example, since the speed (e.g., drag speed, click speed) of a body part wearing the wearable device (201) is different at different heart rates, the heart rate may indicate the user's status.

[0133] In one embodiment, the electronic device (101) may identify a sensor data set for the mouse mode based on the state information. For example, the sensor data set may include a user history data set for grip posture and / or a server data set for grip posture obtained from a server (e.g., cloud).

[0134] For example, the user history data set may include sensor data collected (or accumulated, obtained, trained) by the electronic device (101) based on the user's heart rate with respect to a grip posture, and data about actions performed by the electronic device (101) in response to the collected sensor data. For example, the data about actions performed by the electronic device (101) may include position information of a cursor identified based on the collected data (e.g., position information on a screen of the display (214) where the cursor is displayed) and click information identified based on the collected data. In one example, the user history data set may include first sensor data collected by the electronic device (101) from the wearable device (201) based on a heart rate for a grip posture (e.g., the first grip posture (710) of FIG. 7), data about actions performed by the electronic device (101) in response to the first sensor data, second sensor data collected by the electronic device (101) from the wearable device (201) based on a heart rate for a grip posture, and data about actions performed by the electronic device (101) in response to the second sensor data.

[0135] For example, the server data set may include sensor data collected by another user regarding a grip posture and data regarding actions performed by an electronic device of the other user in response to the collected sensor data. For example, the data regarding actions performed by an electronic device of the other user may include cursor position information based on the collected sensor data and click information identified based on the collected data. In one example, the server data set may include first sensor data collected by a first user regarding a grip posture (e.g., the first grip posture (710) of FIG. 7), data regarding actions performed by a first electronic device of the first user in response to the first sensor data, second sensor data collected by a second user regarding a grip posture, and data regarding actions performed by a second electronic device of the second user in response to the second sensor data.

[0136] In operation 803, the electronic device (101) can identify a hovering input according to a movement of a body part of a user wearing the wearable device (201) based on a sensor data set for the mouse mode and sensor data acquired from the wearable device (201).

[0137] In one embodiment, the electronic device (101) may identify a hovering input that causes movement of a cursor displayed on a screen of the display (214) based on a sensor data set and the sensor data. For example, the electronic device (101) may use an artificial intelligence model to identify a movement direction, speed, and / or movement distance of the cursor according to the sensor data set and the sensor data. For example, input data of the artificial intelligence model may include a sensor data set and the sensor data. For example, output data of the artificial intelligence model may indicate a movement direction and sensitivity of the cursor. For example, the sensitivity may indicate a movement distance of the cursor on the screen of the display (214) of the electronic device (101) corresponding to a movement distance of a wearable device (201) worn on a body part of a user. In one example, the sensitivity may be expressed in dots per inch (DPI). DPI may represent the number of pixels by which a cursor moves on the screen when the wearable device (201) is identified as having moved 1 inch based on a sensor data set and sensor data. However, this is merely an example and the present disclosure is not limited thereto. For example, sensitivity may be expressed in counts per inch (CPI) and / or acceleration.

[0138] In one embodiment, the electronic device (101) can identify a hovering input that causes a click of a cursor displayed on the screen of the display (214) based on the sensor data set and the sensor data. For example, the electronic device (101) can identify a click of the cursor based on the sensor data set and the sensor data using an artificial intelligence model. For example, input data of the artificial intelligence model can include the sensor data set and the sensor data. For example, output data of the artificial intelligence model can indicate a click or a double click of the cursor.

[0139] In operation 804, the electronic device (101) may change the screen displayed through the display (214) based on the hovering input. In one embodiment, the electronic device (101) may display a cursor on the screen of the display (214) of the electronic device (101) based on the hovering input indicating movement of the cursor. For example, the electronic device (101) may display the cursor at positions on the screen of the display (214) corresponding to the movement direction and movement distance of the wearable device (201). In one embodiment, the electronic device (101) may perform an action corresponding to a visual object displayed at the position of the cursor on the screen of the display (214) based on the hovering input indicating a click of the cursor.

[0140] FIGS. 9A to 9C illustrate examples of changing a screen displayed on a display of an electronic device based on sensor data from a wearable device. Although FIGS. 9A to 9C illustrate input means for explaining grip postures formed by parts of a user's body, the input means is illustrated for explaining operations according to the present disclosure. The grip postures illustrated in FIGS. 9A to 9C are formed by parts of a user's body without an input means.

[0141] In FIG. 9A, an example of changing a screen displayed through a display (214) of an electronic device (101) according to a hovering input caused by movement of a body part (701) of a user wearing a wearable device (201) is described. Depending on the movement of the user's body part (701), the position of the wearable device (201) may change from a position (901) (e.g., a first position) to a position (902) (e.g., a second position). While the position of the wearable device (201) changes from a position (901) to a position (902), a sensor (224) of the wearable device (201) may obtain sensor data regarding the body part (701). The electronic device (101) may obtain sensor data regarding the body part (701) from the wearable device (201). The electronic device (101) can identify a grip posture (e.g., the first grip posture (710) or the second grip posture (720) of FIG. 7) for the body part (701) and the user's heart rate based on sensor data acquired from the wearable device (201). In one example, an artificial intelligence model can be used to identify the grip posture. The electronic device (101) can identify a sensor data set associated with the grip posture and the heart rate for the body part (701). For example, the sensor data set can include a user history data set and a server data set including sensor data on grip postures of other users acquired from a server (e.g., cloud). The electronic device (101) can identify locations on the screen of the display (214) corresponding to movements of the wearable device (201) worn on the body part (701) based on the sensor data set and the sensor data acquired from the wearable device (201). In one example, an artificial intelligence model may be used to identify on-screen locations of a display (214) corresponding to movements of a wearable device (201).For example, a trajectory (906) between the positions (904) and (905) of a cursor displayed on the screen of the display (214) may correspond to a trajectory (903) between the positions (901) and (902) of the wearable device (201) according to the movement of the body part (701). The electronic device (101) may display a cursor based on positions on the screen of the display (214) corresponding to the movement of the wearable device (201).

[0142] In FIG. 9B, an example of changing a screen displayed through a display (214) of an electronic device (101) according to a hovering input caused by a movement of a body part (701) of a user wearing a wearable device (201) is described. The body part (701) of a user wearing a wearable device (201) can perform a gesture (or motion) corresponding to a click. The sensor (224) of the wearable device (201) can obtain sensor data for the body part (701) while the gesture is performed. The electronic device (101) can identify a grip posture (e.g., the first grip posture (710) or the second grip posture (720) of FIG. 7) for the body part (701) and the user's heart rate based on the sensor data obtained from the wearable device (201). In one example, an artificial intelligence model may be used to identify a grip posture. The electronic device (101) may identify a sensor data set associated with the grip posture for the body part (701) and the heart rate of the user wearing the wearable device (201). For example, the sensor data set may include a user history data set and a server data set containing sensor data on the grip postures of other users obtained from a server. The electronic device (101) may identify a hovering input indicating a click of a cursor due to the movement of the wearable device (201) worn on the body part (701) based on the sensor data set and the sensor data obtained from the wearable device (201). Based on the identified hovering input, the electronic device (101) may execute an application corresponding to a visual object (911) displayed at the location of the cursor on the screen (910) of the display (214) of the electronic device (101). The electronic device (101) can change the screen displayed through the display (214) from screen (910) to screen (920) in response to the execution of the application.

[0143] In Fig. 9c, the wearable device (201) can be worn on the user's index finger, and the wearable device (931) can be worn on the user's middle finger. The wearable device (931) can be substantially identically applied to the wearable device (201) of Fig. 2. In Fig. 9c, the index finger of the user wearing the wearable device (201) can perform a gesture (or motion) corresponding to a left-click.

[0144] For example, the wearable device (201) may acquire first sensor data for the index finger while the gesture is performed. The electronic device (101) may receive (or acquire) the first sensor data from the wearable device (201). Based on the first sensor data, the electronic device (101) may acquire (or identify) a first sensor data set according to the grip posture of the index finger. The first sensor data set may include a first user history data set stored in the electronic device (101) and / or a first server data set acquired from a server (e.g., cloud). The first user history data set may include sensor data according to the grip posture of the user's index finger collected by the electronic device (101). The first server data set may include sensor data collected by another user about the grip posture of the index finger.

[0145] For example, the wearable device (931) may acquire second sensor data for the middle finger while a gesture corresponding to a left click by the index finger is performed. The electronic device (101) may receive (or acquire) the second sensor data from the wearable device (201). Based on the second sensor data, the electronic device (101) may acquire (or identify) a second sensor data set according to the grip posture of the middle finger. The second sensor data set may include a second user history data set stored in the electronic device (101) and / or a second server data set acquired from a server. The second user history data set may include sensor data according to the grip posture of the middle finger collected by the electronic device (101). The second server data set may include sensor data collected by another user about the grip posture of the middle finger.

[0146] For example, the electronic device (101) can identify a hovering input according to the movement of a body part of the user by using an artificial intelligence model that uses the first sensor data, the second sensor data, the first sensor data set, and / or the second sensor data set as input data. In one example, the electronic device (101) can identify a hovering input corresponding to a left click according to the movement of the user's index finger based on acceleration data and pressure data of the first sensor data. In one example, the electronic device (101) can control not to perform a right click by learning the movement of the middle finger according to the movement of the user's index finger for a left click based on gyro data and bone conduction data of the second sensor data.

[0147] FIG. 10 illustrates a flowchart of operations of an electronic device that controls a display of an electronic device based on sensor data of a wearable device. The operations of FIG. 10 may be performed by the electronic device (101) of FIGS. 1 and 2 . For example, at least some of the operations may be controlled by the processor (211) of the electronic device (101). In the following description, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed. For example, at least two operations may be performed in parallel.

[0148] Referring to FIG. 10, in operation 1001, the electronic device (101) can identify the operation mode of the wearable device (201).

[0149] In one embodiment, the operating mode may be a first mode or a second mode. For example, the first mode may be referred to as a pencil mode (or an air pencil mode). For example, the second mode may be referred to as a mouse mode (or an air mouse mode). For example, the pencil mode may refer to a mode in which dots (or display points, points) are displayed on the screen of the display (214) of the electronic device (101) according to a hovering input by a gesture (or motion) of a body part of a user wearing the wearable device (201). For example, the mouse mode may refer to a mode that controls a cursor (or pointer, mouse pointer) displayed on the screen of the display (214) of the electronic device (101) according to a hovering input by a gesture (or motion) of a body part of a user wearing the wearable device (201). However, this is merely an example, and the present disclosure is not limited thereto. The operation mode may be one of a first mode, a second mode, a third mode (e.g., a virtual reality sports game mode), and a fourth mode (e.g., a virtual reality musical instrument playing mode).

[0150] In one embodiment, the electronic device (101) may receive (or acquire) sensor data about a body part of a user wearing the wearable device (201) connected to the electronic device (101) from the wearable device (201). For example, the sensor data may include data caused by a gesture (or motion) of the body part on which the wearable device (201) is worn. For example, the sensor data may include first pressure data acquired by a first pressure sensor (225) of the wearable device (201), second pressure data acquired by a second pressure sensor (226) of the wearable device (201), IMU data acquired by an IMU (inertial measurement unit) sensor (227) of the wearable device (201), heartbeat data acquired by a heartbeat sensor (228) of the wearable device (201), and fingerprint data acquired by a fingerprint sensor (229) of the wearable device (201). However, this is merely an example, and the present disclosure is not limited thereto. The sensor data may include only some of the first pressure data, the second pressure data, the IMU data, the heartbeat data, and the fingerprint data.

[0151] In one embodiment, the electronic device (101) can identify a body part of a user wearing the wearable device (201) based on sensor data. For example, the wearable device (201) can be worn on one of the user's body parts (e.g., a thumb, an index finger, a middle finger, a ring finger, a litter finger). In one example, the electronic device (101) can identify a body part of a user wearing the wearable device (201) by using the sensor data as input data of an artificial intelligence model. In another example, the electronic device (101) can identify a body part of a user wearing the wearable device (201) based on fingerprint data acquired by a fingerprint sensor (229) of the wearable device (201).

[0152] In one embodiment, the electronic device (101) can identify a grip posture for a body part of a user wearing the wearable device (201) based on sensor data. For example, the electronic device (101) can identify a grip posture for a body part of a user wearing the wearable device (201) by using the sensor data as input data of an artificial intelligence model. In one example, the electronic device (101) can identify a first grip posture (e.g., grip posture (410) of FIG. 4) based on the sensor data. In one example, the electronic device (101) can identify a second grip posture (e.g., grip posture (420) of FIG. 4) based on the sensor data. In one example, the electronic device (101) can identify a third grip posture (e.g., grip posture (710) of FIG. 7) based on the sensor data. In one example, the electronic device (101) may identify a fourth grip posture (e.g., grip posture (720) of FIG. 7) based on sensor data. However, this is merely an example, and the present disclosure is not limited to the above-described example. The grip posture may further include any posture depending on the user or user status.

[0153] In one embodiment, the electronic device (101) can identify the operating mode of the wearable device (201) based on the body part on which the wearable device (201) is worn and the grip posture for the body part. For example, an artificial intelligence model can be used to identify the operating mode. For example, input data of the artificial intelligence model for identifying the operating mode can include first pressure data, second pressure data, and / or IMU data (e.g., gyro data). For example, output data of the artificial intelligence model can indicate one of the first mode and the second mode.

[0154] In operation 1002, the electronic device (101) can control a screen displayed through the display (214) of the electronic device (101) in the first mode. For example, the electronic device (101) can control a screen displayed through the display (214) of the electronic device (101) in the first mode based on identification that the operation mode of the wearable device (201) is the first mode. For example, the screen displayed through the display (214) in the first mode can be controlled according to the method illustrated in FIG. 5.

[0155] In operation 1003, the electronic device (101) can control a screen displayed through the display (214) of the electronic device (101) in the second mode. For example, the electronic device (101) can control a screen displayed through the display (214) of the electronic device (101) in the second mode based on identification that the operating mode of the wearable device (201) is the second mode. For example, the screen displayed through the display (214) in the second mode can be controlled according to the method illustrated in FIG. 8.

[0156] In operation 1004, the electronic device (101) may perform a mode verification procedure. For example, the mode verification procedure may be performed based on a predetermined cycle. For example, the electronic device (101) may analyze sensor data acquired from the wearable device (201) using an artificial intelligence model to perform the mode verification procedure. For example, input data of the artificial intelligence model may include sensor data acquired from the wearable device (201) and an operation mode (e.g., a first mode or a second mode) of the wearable device (201). For example, output data of the artificial intelligence model may indicate whether a mode identification procedure is performed. In one example, the artificial intelligence model may output output data indicating that the mode identification procedure is performed by identifying that sensor data according to a different mode than the first mode is received while operations according to the first mode are performed. In one example, the artificial intelligence model may output output data indicating that the mode identification procedure is not performed by identifying that sensor data according to the first mode is received while operations according to the first mode are being performed. For example, the electronic device (101) may perform operation 1001 when the output data of the artificial intelligence model indicates that the mode identification procedure is performed. For example, the electronic device (101) may control a screen displayed through the display (214) according to the operation mode of the wearable device (201) when the output data of the artificial intelligence model indicates that the mode identification procedure is not performed.

[0157] FIG. 11 illustrates a flowchart of operations of an electronic device that changes a screen displayed on a display of the electronic device based on sensor data of a wearable device. The operations of FIG. 11 may be performed by the electronic device (101) of FIGS. 1 and 2 . For example, at least some of the operations may be controlled by the processor (211) of the electronic device (101). In the following description, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed. For example, at least two operations may be performed in parallel.

[0158] Referring to FIG. 11, in operation 1101, the electronic device (101) can identify a body part of a user wearing the wearable device (201).

[0159] In one embodiment, the wearable device (201) may be worn on a body part of the user. For example, the wearable device (201) may be worn on one of the body parts of the user (e.g., the thumb, the index finger, the middle finger, the ring finger, and the little finger). However, this is merely an example, and the present disclosure is not limited thereto. For example, the wearable device (201) may be worn on a body part other than the above-mentioned body parts (e.g., the toes).

[0160] In one embodiment, the electronic device (101) may receive (or acquire) sensor data from the wearable device (201). The sensor data may include data caused by a movement (or gesture, motion) of a body part on which the wearable device (201) is worn. For example, the sensor data may include first pressure data acquired by a first pressure sensor (225) of the wearable device (201), second pressure data acquired by a second pressure sensor (226) of the wearable device (201), IMU data acquired by an IMU (inertial measurement unit) sensor (227) of the wearable device (201), heartbeat data acquired by a heartbeat sensor (228), and fingerprint data acquired by a fingerprint sensor (229). The IMU data acquired by the IMU sensor (227) may include gyro data, acceleration data, and / or magnetic data for the wearable device (201). However, this is merely an example, and the present disclosure is not limited thereto. The sensor data may include only some of the first pressure data, the second pressure data, the gyro data, the acceleration data, the magnetic data, the heartbeat data, and the fingerprint data. For example, the first pressure data, the second pressure data, the heartbeat data, and the fingerprint data may be referenced as biometric information for a body part of a user wearing the wearable device (201). For example, the IMU data may be referenced as motion information and / or posture information for a body part of a user wearing the wearable device (201). In a non-limiting example, the sensor data may further include bone conduction data obtained from a bone conduction sensor of the wearable device (201) to prevent or reduce unintended actions from being performed due to movement of a body part that is not wearing the wearable device (201).For example, the first pressure data may represent a pressure caused by contraction (or relaxation) of a flexor muscle of a body part of a user wearing the wearable device (201). For example, the second pressure data may represent a pressure caused by relaxation (or contraction) of an extensor muscle of a body part of a user wearing the wearable device (201). For example, the IMU data may represent an orientation of a wearable device (201) worn on a body part of a user, a direction of movement of the wearable device (201), and / or a speed of movement of the wearable device (201). For example, the heart rate data may represent a heart rate of a user wearing the wearable device (201). For example, the fingerprint data may represent fingerprint data of a body part of a user wearing the wearable device (201).

[0161] In one embodiment, the electronic device (101) can identify a body part (e.g., a thumb) of a user wearing the wearable device (201) based on sensor data. In one example, the electronic device (101) can identify a body part of a user wearing the wearable device (201) based on movement pattern information of body parts learned by an artificial intelligence model according to the sensor data. In another example, the electronic device (101) can identify a body part of a user wearing the wearable device (201) based on fingerprint data acquired by a fingerprint sensor (229).

[0162] In operation 1102, the electronic device (101) can identify the operation mode of the wearable device (201).

[0163] In one embodiment, the operating mode may be one of a first mode or a second mode. For example, the first mode may be referred to as a pencil mode (or an air pencil mode). The pencil mode may refer to a mode in which dots (or display points, points) are displayed on the screen of the display (214) of the electronic device (101) according to a hovering input by a movement (or gesture, motion) of a body part of a user wearing the wearable device (201). For example, the second mode may be referred to as a mouse mode (or an air mouse mode). The mouse mode may refer to a mode that controls a cursor (or pointer, mouse pointer) displayed on the screen of the display (214) of the electronic device (101) according to a hovering input by a movement (or gesture, motion) of a body part of a user wearing the wearable device (201). However, this is merely an example, and the present disclosure is not limited thereto. The operation mode may further include other modes (e.g., virtual reality sports game mode, virtual reality instrument playing mode) in addition to the above modes.

[0164] In one embodiment, the electronic device (101) can identify a grip posture for a body part of a user wearing the wearable device (201) based on sensor data. For example, the electronic device (101) can identify a grip posture for a body part of a user wearing the wearable device (201) by using the sensor data as input data of an artificial intelligence model. In one example, the electronic device (101) can identify a first grip posture (e.g., grip posture (410) of FIG. 4) based on the sensor data. In one example, the electronic device (101) can identify a second grip posture (e.g., grip posture (420) of FIG. 4) based on the sensor data. In one example, the electronic device (101) can identify a third grip posture (e.g., grip posture (710) of FIG. 7) based on the sensor data. In one example, the electronic device (101) may identify a fourth grip posture (e.g., grip posture (720) of FIG. 7) based on sensor data. However, this is merely an example, and the present disclosure is not limited to the above-described example. The grip posture may further include any posture depending on the user or user status.

[0165] In one embodiment, the electronic device (101) can identify the operating mode of the wearable device (201) based on the body part wearing the wearable device (201) and the grip posture of the body part. For example, an artificial intelligence model can be used to identify the operating mode. For example, input data of the artificial intelligence model for identifying the operating mode can include sensor data and / or a sensor data set. For example, output data of the artificial intelligence model can indicate one of the operating modes of the wearable device (201).

[0166] In operation 1103, the electronic device (101) may identify user status information based on sensor data. For example, the status information may indicate a grip posture and / or heart rate corresponding to the sensor data among the grip posture (or posture, grip style, hand posture, holding posture, grip method, grasp style) of the body part wearing the wearable device (201). For example, since different grip postures cause different sensor data, the grip posture may indicate the user's status. For example, since the movement (e.g., speed, angle, pen pressure) of the body part wearing the wearable device (201) is different at different heart rates, the heart rate may indicate the user's status.

[0167] In operation 1104, the electronic device (101) may identify a sensor data set based on the state information. For example, the sensor data set may include a user history data set for grip posture (and / or heart rate) stored in the electronic device (101) and / or a server data set for grip posture (and / or heart rate) obtained from a server (e.g., cloud). The user history data set may include sensor data collected (or accumulated, obtained, trained) by the electronic device (101) for grip posture and / or data for actions performed by the electronic device (101) in response to the collected sensor data. The server data set may include sensor data collected by another user for grip posture and / or data for actions performed by the electronic device of another user in response to the collected sensor data.

[0168] In operation 1105, the electronic device (101) can determine input variables and weights of an artificial intelligence model according to the operation mode.

[0169] In one embodiment, the electronic device (101) may determine input variables of an artificial intelligence model according to an operation mode. In one example, the electronic device (101) may determine input variables of the artificial intelligence model as IMU values ​​(e.g., acceleration values, gyro values, and / or magnetic values ​​of the wearable device (201)) and pressure values ​​based on identifying the operation mode of the wearable device (101) as a first mode (e.g., pencil mode). In one example, the electronic device (101) may determine input variables of the artificial intelligence model as IMU values, pressure values, heart rate, and bone conduction values ​​based on identifying the operation mode of the wearable device (101) as a second mode (e.g., mouse mode). However, this is merely an example and the present disclosure is not limited thereto. The electronic device (101) may use only some of the sensor data acquired from the wearable device (201) as input data for the artificial intelligence model, depending on the operation mode of the wearable device (201).

[0170] In one embodiment, the electronic device (101) may determine weights of an artificial intelligence model according to an operating mode. For example, the weights of an artificial intelligence model according to a first mode may be different from the weights of an artificial intelligence model according to a second mode.

[0171] In operation 1106, the electronic device (101) may change the screen displayed through the display (214) based on the hovering input. For example, the electronic device (101) may change the screen displayed through the display (214) according to the method illustrated in FIG. 5 based on the identification that the operation mode of the wearable device (201) is the first mode. For example, the electronic device (101) may change the screen displayed through the display (214) according to the method illustrated in FIG. 8 based on the identification that the operation mode of the wearable device (201) is the second mode.

[0172] In operation 1107, the electronic device (101) may update an artificial intelligence model based on user history data. For example, the user history data may include sensor data acquired from a wearable device (201) and data representing operations of the electronic device (101) performed based on the sensor data. For example, the electronic device (101) may determine (or change) weights of the artificial intelligence model based on the user history data. For example, the electronic device (101) may transmit the user history data to a server (e.g., a cloud).

[0173] Figure 12 illustrates a screen of an electronic device for setting an operation mode of a wearable device.

[0174] Referring to FIG. 12, the electronic device (101) may display a screen for setting the operation mode of the wearable device (201) through the display (214). In one example, the electronic device (101) may display a first user interface (UI) (1201) (or an object, a visual object), a second UI (1202), a third UI (1203), a fourth UI (1204), a fifth UI (1205), and a sixth UI (1206) on the screen. However, this is merely an example, and the present disclosure is not limited thereto. For example, the electronic device (101) may display only some of the UIs on the screen. The first UI (1201) may indicate an air pencil mode of the wearable device (201). The second UI (1202) may indicate an air mouse mode of the wearable device (201). The third UI (1203) may indicate a piano mode of the wearable device (201). The fourth UI (1204) may indicate an automatic mode of the wearable device (201). The fifth UI (1205) may indicate a first custom mode of the wearable device (201). The sixth UI (1206) may indicate a second custom mode of the wearable device (201). The electronic device (101) may share the first custom mode and / or the second custom mode with other users by transmitting information about the first custom mode and / or the second custom mode to a server.

[0175] In one embodiment, the electronic device (101) may determine the operation mode of the wearable device (201) based on obtaining a user input for the UI. For example, the electronic device (101) may determine the operation mode of the wearable device (201) as an air pencil mode based on obtaining a user input for the first UI (1201). The electronic device (101) may change the screen displayed through the display (214) according to the method illustrated in FIG. 5 based on determining the operation mode of the wearable device (201) as the air pencil mode. In another example, the electronic device (101) may determine the operation mode of the wearable device (201) as an air mouse mode based on obtaining a user input for the second UI (1202). The electronic device (101) can change the screen displayed through the display (214) according to the method illustrated in FIG. 8 based on determining the operation mode of the wearable device (201) as the air mouse mode. In another example, the electronic device (101) can determine the operation mode of the wearable device (201) as the automatic mode based on obtaining a user input for the fourth UI (1204). In the automatic mode, the electronic device (101) can determine the operation mode of the wearable device (201) according to the body part wearing the wearable device (201) and the grip posture of the body part using sensor data obtained from the wearable device (201). In one example, the electronic device (101) may determine the operation mode of the wearable device (201) as an air pencil mode when the grip posture of the body part wearing the wearable device (201) is identified as a first grip posture (410) or a second grip posture (420). In another example, the electronic device (101) may determine the operation mode of the wearable device (201) as an air mouse mode when the grip posture of the body part wearing the wearable device (201) is identified as a third grip posture (710) or a fourth grip posture (720).

[0176] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present disclosure belongs.

[0177] As described above, the electronic device (101) may include a communication circuit (213). The electronic device (101) may include a display (214). The electronic device (101) may include a memory (212) that stores instructions and includes one or more storage media. The electronic device (101) may include at least one processor (211) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain sensor data about a body part of a user wearing a wearable device connected to the electronic device from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a sensor data set for a pencil mode based on state information of the user identified based on the sensor data. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify positions of dots on a screen of the display caused by a hovering input according to movement of the body part, based on the sensor data set and the sensor data obtained from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to change the screen displayed through the display based on the positions of the dots.

[0178] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify positions of the dots corresponding to positions of the fan points based on an artificial intelligence model using the sensor data set and input data including the sensor data.

[0179] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine the number of pixels of the display for each dot of the dots based on an artificial intelligence model using the sensor data set and input data including the sensor data.

[0180] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine a number of dots caused on the screen of the display by the hovering input by identifying a fan point for the hovering input based on an artificial intelligence model using the sensor data set and input data including the sensor data.

[0181] For example, the state information may indicate a grip posture corresponding to the sensor data among a plurality of grip postures of the body part of the user wearing the wearable device. For example, the sensor data set may include a user history data set and a server data set, and the user history data set may include a plurality of sensor data for a grip posture corresponding to the sensor data among a plurality of grip postures. For example, the server data set may include a plurality of sensor data for the grip postures of another user.

[0182] For example, the sensor data may include first pressure data caused by a flexor muscle of the body part wearing the wearable device, second pressure data caused by an extensor muscle of the body part wearing the wearable device, and IMU (inertial measurement unit) data indicating the orientation, acceleration, and angular velocity of the wearable device.

[0183] The electronic device (101) as described above may include a communication circuit (213). The electronic device (101) may include a display (214). The electronic device (101) may include a memory (212) that stores instructions and includes one or more storage media. The electronic device (101) may include at least one processor (211) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive sensor data acquired by at least one sensor of the wearable device connected to the electronic device from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a hovering input according to a movement of a body part wearing the wearable device based on using the sensor data as input data of an artificial intelligence model of the electronic device. The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate an output signal corresponding to the hovering input. The sensor data may include at least one of biometric information of the body part, motion information of the body part, or posture information of the body part.

[0184] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify the body part of a user wearing the wearable device based on the sensor data. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to learn, using the artificial intelligence model, a sensor data pattern for the motion information of the body part and the posture information of the body part.

[0185] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit the output signal to another electronic device for controlling the operation of the other electronic device.

[0186] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify one of a plurality of operating modes based on the identified hovering input. The plurality of operating modes may include at least two of a pencil mode, a mouse mode, a game mode, or an instrument play mode.

[0187] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the identification of a movement distance and a movement direction of a pen point in the pencil mode, the identification of a movement distance, a movement direction, or a click of a cursor in the mouse mode, or the identification of a movement distance, a movement direction, and a movement speed of a body part in the musical instrument playing mode, based on the identified hovering input using the sensor data as the input data of the artificial intelligence model of the electronic device.

[0188] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the hovering input identified based on using the sensor data as input data of the artificial intelligence model of the electronic device to determine the number of pixels of the display for a dot caused on the screen of the display by the hovering input in the pencil mode, determine the sensitivity of the cursor in the mouse mode, or determine the volume of a sound output in the musical instrument playing mode.

[0189] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain first sensor data from the first wearable device for a different body part than the body part from the first wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify the hovering input according to the movement of the wearable device based on using the sensor data and the first sensor data as the input data of the artificial intelligence model.

[0190] The electronic device (101) as described above may include a communication circuit (213). The electronic device (101) may include a display (214). The electronic device (101) may include a memory (212) that stores instructions and includes one or more storage media. The electronic device (101) may include at least one processor (211) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain sensor data about a body part of a user wearing a wearable device connected to the electronic device from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify an operation mode of the wearable device based on the body part identified based on the sensor data and a grip posture of the body part. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to change a screen displayed through the display based on a hovering input identified based on movement of the body part wearing the wearable device in the identified operating mode.

[0191] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a set of sensor data for a pencil mode of the identified grip posture, based on the identification that the operating mode of the wearable device is a pencil mode. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, based on the set of sensor data and the sensor data obtained from the wearable device, locations of dots on the screen of the display caused by the hovering input according to the movement of the body part. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to change the screen displayed through the display by displaying the dots at the locations on the screen.

[0192] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify positions of the dots corresponding to positions of the fan points based on an artificial intelligence model using the sensor data set and input data including the sensor data.

[0193] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine the number of pixels of the display on which the dots are displayed based on an artificial intelligence model using the sensor data set and input data including the sensor data.

[0194] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify state information of the user based on the sensor data upon identification that the operating mode of the wearable device is a mouse mode. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a sensor data set for the mouse mode upon the state information. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify the hovering input according to movement of the body part based on the sensor data set and the sensor data acquired from the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the cursor at locations corresponding to movement of the body part upon identification that the hovering input represents movement of a cursor displayed on the screen of the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute an application of a visual object displayed at a location of the cursor on the screen of the display upon identification that the hovering input represents a click of the cursor displayed on the screen of the display.

[0195] For example, the status information may indicate the heart rate of the user wearing the wearable device and the grip posture corresponding to the sensor data among a plurality of grip postures of the body part.

[0196] An electronic device according to the present disclosure can perform the functions of various input means using a wearable device. The electronic device according to the present disclosure can perform the functions of an input means by using a wearable device without carrying the input means.

[0197] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0198] The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0199] When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured to be executed by one or more processors in an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure. The one or more programs may be provided as a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0200] These programs (software modules, software) may be stored in random access memory, non-volatile memory including flash memory, read only memory (ROM), electrically erasable programmable read only memory (EEPROM), magnetic disc storage devices, compact disc-ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, magnetic cassettes, or may be stored in memories formed by a combination of some or all of these. In addition, each configuration memory may include multiple copies.

[0201] Additionally, the program may be stored on an attachable storage device that is accessible via a communication network, such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a storage area network (SAN), or a combination thereof. Such a storage device may be connected to a device implementing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to a device implementing an embodiment of the present disclosure.

[0202] In the specific embodiments of the present disclosure described above, components included in the disclosure are expressed singularly or plurally, depending on the specific embodiment presented. However, the singular or plural expressions are selected to suit the presented situation for convenience of explanation, and the present disclosure is not limited to singular or plural components. Components expressed in plural may be composed of singular elements, or components expressed in singular may be composed of plural elements.

[0203] According to embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0204] Meanwhile, although the detailed description of the present disclosure has described specific embodiments, it is obvious that various modifications are possible within the scope of the present disclosure.

Claims

In electronic devices, communication circuit; display; A memory storing instructions and including one or more storage media; and At least one processor comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Acquire sensor data on a body part of a user wearing a wearable device connected to the electronic device from the wearable device, Identifying a sensor data set for pencil mode based on the user's status information identified based on the sensor data; Based on the sensor data set and the sensor data obtained from the wearable device, the positions of dots caused on the screen of the display by a hovering input according to the movement of the body part are identified, and Causing the screen displayed through the display to be changed based on the positions of the dots; Electronic devices. In the first paragraph, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: By identifying the positions of the fan points for the hovering input based on an artificial intelligence model using the sensor data set and input data including the sensor data, thereby causing the positions of the dots corresponding to the positions of the fan points to be identified, Electronic devices. In the first paragraph, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on the artificial intelligence model using the above sensor data set and input data including the above sensor data, causing the number of pixels of the display for the dots to be determined, Electronic devices. In the first paragraph, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: By identifying a fan point for the hovering input based on an artificial intelligence model using the sensor data set and input data including the sensor data, the number of dots caused on the screen of the display by the hovering input is determined. Electronic devices. In the first paragraph, The above state information indicates a grip posture corresponding to the sensor data among a plurality of grip postures of the body part of the user wearing the wearable device. Electronic devices. In the first paragraph, The above sensor data set includes a user history data set and a server data set, The above user history data set includes a plurality of sensor data for a grip posture corresponding to the sensor data among a plurality of grip postures, and The above server data set includes multiple sensor data on the grip posture of another user, Electronic devices. In the first paragraph, The sensor data includes first pressure data caused by a flexor muscle of the body part wearing the wearable device, second pressure data caused by an extensor muscle of the body part wearing the wearable device, and IMU (inertial measurement unit) data indicating the orientation, acceleration, and angular velocity of the wearable device. Electronic devices. A method performed by an electronic device including a display, An action of acquiring sensor data on a body part of a user wearing a wearable device connected to the electronic device from the wearable device; An operation of identifying a sensor data set for a pencil mode based on the user's status information identified based on the sensor data; An operation of identifying the positions of dots on the screen of the display caused by a hovering input according to the movement of the body part, based on the sensor data set and the sensor data obtained from the wearable device; and An operation of changing the screen displayed through the display based on the positions of the dots, method. In the 8th paragraph, the operation of identifying the positions of the dots is as follows: An operation of identifying locations of fan points for the hovering input based on an artificial intelligence model using the sensor data set and input data including the sensor data, method. In paragraph 8, Further comprising an operation of determining the number of pixels of the display for each dot of the dots based on an artificial intelligence model using the sensor data set and input data including the sensor data. method. In paragraph 8, Further comprising an operation of determining the number of dots caused on the screen of the display by the hovering input by identifying a fan point for the hovering input based on an artificial intelligence model using the sensor data set and input data including the sensor data. method. In paragraph 8, The above state information indicates a grip posture corresponding to the sensor data among a plurality of grip postures of the body part of the user wearing the wearable device. method. In paragraph 8, The above sensor data set includes a user history data set and a server data set, The above user history data set includes a plurality of sensor data for a grip posture corresponding to the sensor data among a plurality of grip postures, and The above server data set includes multiple sensor data on the grip posture of another user, method. In paragraph 8, The sensor data includes first pressure data caused by a flexor muscle of the body part wearing the wearable device, second pressure data caused by an extensor muscle of the body part wearing the wearable device, and IMU (inertial measurement unit) data indicating the orientation, acceleration, and angular velocity of the wearable device. method. A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs, when executed by an electronic device including at least one processor, Acquire sensor data on a body part of a user wearing a wearable device connected to the electronic device from the wearable device, Identifying a sensor data set for pencil mode based on the user's status information identified based on the sensor data; Based on the sensor data set and the sensor data obtained from the wearable device, the positions of dots caused on the screen of the display by a hovering input according to the movement of the body part are identified, and Including instructions that cause the electronic device to change the screen displayed through the display based on the positions of the dots. Non-transitory computer-readable storage medium.

Images