Electronic device and method for operating electronic device

Abstract

The electronic device comprises a camera, a communication interface, a memory, and a processor. When the processor executes a program or at least one instruction stored in the memory, the electronic device is capable of: obtaining a user image by photographing at least one user; obtaining at least one piece of gesture information corresponding to each of the at least one user by estimating a gesture from the user image; obtaining inertial information from an external electronic device; obtaining mapping gesture information corresponding to the inertial information on the basis of the at least one piece of gesture information and the inertial information; and obtaining a final gesture for controlling the electronic device of the user corresponding to the mapping gesture information on the basis of the mapping gesture information and the inertial information.

Description

Electronic device and method of operation of electronic device

[0001] The present disclosure relates to an electronic device and a method of operating the electronic device. Specifically, it relates to an electronic device that estimates a gesture of a user using the electronic device and a method of operating the electronic device.

[0002] With recent technological advancements, technology that captures and utilizes human gestures from images obtained by photographing a user with a camera is being widely used.

[0003] In addition, technologies that acquire and utilize information about the movement of devices containing inertial measurement sensors through inertial measurement sensors are also widely used.

[0004] In addition, recently, technology that fuses information obtained from different types of sensors to obtain device movements or human gestures is also being widely used.

[0005] Technology that controls the operation of electronic devices based on human gestures or device movements obtained in this way is also being utilized.

[0006] One embodiment of the present disclosure provides an electronic device. The electronic device may include a camera. The electronic device may include a communication interface including a circuit. The electronic device may include a memory in which a program or at least one instruction is stored. The electronic device may include at least one processor. By having at least one processor execute the program or at least one instruction stored in the memory individually or collectively, the electronic device may capture at least one user using the electronic device through the camera and acquire a user image. The electronic device may estimate a gesture from the user image and acquire at least one gesture information corresponding to each of the at least one user. The electronic device may acquire inertial information from an external electronic device through a communication interface. Based on at least one gesture information and inertial information, the electronic device may acquire mapping gesture information corresponding to the inertial information among the at least one gesture information. Based on the mapping gesture information and inertial information, the electronic device may acquire a final gesture for controlling the electronic device of the user corresponding to the mapping gesture information among the at least one user.

[0007] In one embodiment of the present disclosure, a method for operating an electronic device may be provided. The method for operating an electronic device may include the step of obtaining a user image by photographing at least one user using the electronic device through a camera. The method for operating an electronic device may include the step of estimating a gesture from the user image and obtaining at least one gesture information corresponding to each of the at least one user. The method for operating an electronic device may include the step of obtaining inertial information from an external electronic device through a communication interface. The method for operating an electronic device may include the step of obtaining mapping gesture information corresponding to the inertial information among the at least one gesture information, based on the at least one gesture information and inertial information. The method for operating an electronic device may include the step of obtaining a final gesture for controlling the electronic device of the user corresponding to the mapping gesture information among the at least one user, based on the mapping gesture information and inertial information.

[0008] In one embodiment of the present disclosure, a computer-readable recording medium may be provided on which a program for performing at least one of the embodiments of the operation method of the disclosed electronic device is recorded on a computer.

[0009] The technical problems to be solved in this document are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure belongs from the description below.

[0010] The present disclosure may be understood from the combination of the following detailed description and the accompanying drawings, where reference numerals denote structural elements.

[0011] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0012] FIG. 2 is a block diagram for explaining the configuration of an electronic device according to one embodiment of the present disclosure.

[0013] FIG. 3 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0014] FIG. 4 is a diagram illustrating gesture information estimated from a user image according to one embodiment of the present disclosure.

[0015] FIG. 5 is a diagram illustrating inertial information obtained from an external electronic device according to one embodiment of the present disclosure.

[0016] FIG. 6 is a flowchart for explaining the operation of acquiring mapping sub-gesture information corresponding to a specific body area where an external electronic device is located, among a plurality of sub-gesture information according to one embodiment of the present disclosure.

[0017] FIG. 7 is a flowchart illustrating an operation to compare sub-gesture information corresponding to a preset reference body area among gesture information, according to one embodiment of the present disclosure, with inertia information.

[0018] FIG. 8 is a diagram illustrating an operation to compare sub-gesture information corresponding to a preset reference body area among gesture information, according to one embodiment of the present disclosure, with inertia information.

[0019] FIG. 9 is a flowchart illustrating an operation to obtain mapping gesture information based on at least one gesture information and inertia information using an extended Kalman filter according to one embodiment of the present disclosure.

[0020] FIG. 10 is a diagram illustrating an operation to acquire mapping gesture information based on at least one sub-gesture information corresponding to a preset reference body area among a plurality of sub-gesture information, according to one embodiment of the present disclosure, and inertia information.

[0021] FIG. 11 is a flowchart illustrating an operation to obtain gesture information corresponding to the largest value among at least one filtering Kalman gain that is greater than a preset reference value, as mapping gesture information, according to one embodiment of the present disclosure.

[0022] FIG. 12 is a flowchart illustrating the operation of an electronic device transmitting a request signal requesting inertial information as the period for acquiring inertial information is greater than a preset threshold period, according to one embodiment of the present disclosure.

[0023] FIG. 13 is a diagram illustrating the operation of an electronic device acquiring inertial information transmitted via a broadcasting method and the operation of an electronic device requesting inertial information, according to one embodiment of the present disclosure.

[0024] FIG. 14 is a flowchart for explaining the operation of an electronic device that transmits a request signal requesting inertial information as the acquired gesture information includes a preset start gesture according to one embodiment of the present disclosure.

[0025] FIG. 15 is a drawing for illustrating a preset start gesture that determines whether to perform an operation of an electronic device according to one embodiment of the present disclosure.

[0026] FIG. 16 is a flowchart for explaining the operation of an external electronic device that transmits inertial information according to one embodiment of the present disclosure, wherein a preset start gesture is included in a gesture obtained based on inertial information.

[0027] FIG. 17 is a flowchart illustrating the operation of an electronic device that obtains a final gesture based on a correction weight proportional to the reliability of a user image obtained from a pose estimation model, according to one embodiment of the present disclosure.

[0028] FIG. 18 is a drawing for explaining the operation between an electronic device and an external electronic device according to one embodiment of the present disclosure.

[0029] The terms used in this disclosure will be briefly explained, and an embodiment of this disclosure will be described in detail.

[0030] Throughout this disclosure, unless specifically stated otherwise, "or" is inclusive and not exclusive. Accordingly, "A or B" may mean "A, B, or both" unless clearly indicated otherwise by the context.

[0031] In the present disclosure, the expression “at least one of a, b, or c” may refer to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b, and c all”, or variations thereof.

[0032] In describing the present disclosure, technical details that are well known in the technical field to which the present disclosure belongs and are not directly related to the present disclosure are omitted. This is intended to convey the essence of the present disclosure more clearly without obscuring it by omitting unnecessary explanations.

[0033] The terms used in this disclosure have been selected to be as widely used as possible, taking into account the functions in the embodiments of this disclosure; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description section of the relevant embodiments of this disclosure. Therefore, the terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0034] In the accompanying drawings of this disclosure, some components are exaggerated, omitted, or schematically depicted. Additionally, the size of each component does not entirely reflect its actual size. Identical or corresponding components in each drawing are given the same reference numerals.

[0035] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art as described in this specification.

[0036] Throughout this disclosure, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part," "module," etc., as used in this disclosure refer to a unit that processes at least one function or operation, and may be implemented in hardware or software, or as a combination of hardware and software.

[0037] The expression “configured to” as used in this disclosure may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “system configured to” may mean that the system is “capable of” together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in memory.

[0038] In addition, when a component is described in the present disclosure as being “connected” or “connected” to another component, it should be understood that the component may be directly connected to or directly connected to the other component, but unless otherwise specifically stated, it may also be connected or connected through another component in between.

[0039] In one embodiment of the present disclosure, each block in each flowchart and combinations of flowcharts may be executed by one or more computer programs comprising computer-executable instructions. One or more computer programs may be loaded into a processor of a general-purpose computer, a computer for special purposes, or other programmable data processing equipment, and the instructions executed through the processor of the computer or other programmable data processing equipment may generate means for performing the functions described in the flowchart block(s). One or more computer programs may be stored all in a single memory or may be divided and stored in a plurality of different memories.

[0040] Additionally, each block of the flowchart may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). In one embodiment of the present disclosure, the functions mentioned in the blocks may occur out of order. For example, two blocks shown in succession may be executed substantially simultaneously or in reverse order according to function.

[0041] All functions or operations described in this document may be processed by a single processor or a combination of multiple processors.

[0042] Functions related to artificial intelligence according to the present disclosure are operated through processors and memory. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0043] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using a number of training data by a learning algorithm. Such learning may be performed on the electronic device itself in which the artificial intelligence model according to the present disclosure is used, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0044] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network may include a Deep Neural Network (DNN), such as a Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, but is not limited to the examples mentioned above.

[0045] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, an embodiment of the present disclosure may be implemented in various different forms and is not limited to the embodiment described herein. Furthermore, in order to clearly explain an embodiment of the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the present disclosure are denoted by similar reference numerals.

[0046] Embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0047] FIG. 1 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0048] Referring to FIG. 1, in one embodiment of the present disclosure, an electronic device (100) may be a device that displays an image (200) and provides it to a plurality of users (300, 310, 320). The electronic device (100) may include a display (110) that displays the image (200). The electronic device (100) may display the image (200) through the display (110) and provide the image (200) to the users (300, 310, 320).

[0049] In one embodiment of the present disclosure, FIG. 1 is illustrated that the electronic device (100) is in the shape of a television. However, the present disclosure is not limited thereto. The electronic device (100) may be implemented as an electronic device of various shapes, such as a mobile device, a smartphone, a laptop computer, a desktop, a tablet PC, digital signage, a projector, and a wearable device.

[0050] In one embodiment of the present disclosure, FIG. 1 illustrates three users (300, 310, 320) using the electronic device (100). However, the present disclosure is not limited thereto, and the number of users using the electronic device (100) may be fewer than three or more than three, or at least one user. For convenience of explanation, the number of users using the electronic device (100) is described as three.

[0051] In one embodiment of the present disclosure, the electronic device (100) can estimate the gestures of a plurality of users (300, 310, 320) using the electronic device (100).

[0052] Specifically, the electronic device (100) may include a camera (120). The electronic device (100) may obtain user images by photographing users (300, 310, 320) through the camera (120). The electronic device (100) may estimate the gestures of multiple users (300, 310, 320) from the user images. The gestures of multiple users (300, 310, 320) may be gestures for controlling the operation of the electronic device (100). The electronic device (100) may estimate the gestures of multiple users (300, 310, 320) from the user images through a pose estimation model.

[0053] At this time, feature points can be detected from the user image through a pose estimation model. A "feature point" may refer to a point within the image that is distinguishable from the surrounding background or is easily identifiable. Feature points may include joints (skeletons), hands, faces, etc. In one embodiment of the present disclosure, a "gesture" may refer to a shape corresponding to a form created by a plurality of feature points and lines connecting each of the plurality of feature points.

[0054] In one embodiment of the present disclosure, the operation of an electronic device (100) may be controlled based on gestures of a plurality of estimated users (300, 310, 320). The plurality of users (300, 310, 320) using the electronic device (100) may include a first user (300), a second user (310), and a third user (320).

[0055] In one embodiment of the present disclosure, the operation of an electronic device (100), for example, changing the type of image displayed on a display (110), changing the duration of the image, changing the sound or volume of the image, the type of application executed on the electronic device (100), or control within the application, etc., may be controlled by an estimated gesture of each of the first user (300), the second user (310), and the third user (320). At this time, the content of the operation controlled in correspondence with the type of estimated gesture may be pre-set.

[0056] However, if part of the body of each of the multiple users (300, 310, 320) being photographed by the camera (120) is obscured by an obstacle or another user, or if the quality or resolution of the user image obtained through the camera (120) is low, the accuracy of the gestures of the multiple users (300, 310, 320) estimated from the user image may be low. Accordingly, the operation of the electronic device (100) may be controlled differently from the intention of the multiple users (300, 310, 320).

[0057] In one embodiment of the present disclosure, at least one user among a plurality of users (300, 310, 320) using an electronic device (100) may use another electronic device (400) different from the electronic device (100). In this case, the other electronic device (400) may be referred to as an external electronic device (400). In one embodiment of the present disclosure, a third user (320) may use the electronic device (100) while wearing the external electronic device (400).

[0058] In one embodiment of the present disclosure, the external electronic device (400) may be a wearable device that a user can wear. The external electronic device (400) may be an electronic device having various shapes, such as a watch, earphones, a bracelet, a ring, a belt, etc. The external electronic device (400) may include a smart watch, a smart ring, wireless earphones, a head-mounted display device, etc. In one embodiment of the present disclosure, a third user (320) may use the electronic device (100) while wearing an external electronic device (400) in the form of a smart watch on their left wrist.

[0059] However, the present disclosure is not limited thereto, and the external electronic device (400) may be an electronic device that can be moved together with the user as they use the electronic device (100) and move their body, such as a mobile device or a smartphone. Hereinafter, for convenience of explanation, the external electronic device (400) will be described as a wearable device.

[0060] In one embodiment of the present disclosure, the external electronic device (400) may include an inertia measurement unit (IMU). The external electronic device (400) may obtain inertia information by measuring changes in inertia according to the movement of a user wearing the external electronic device (400). The external electronic device (400) may obtain inertia information by measuring changes in inertia according to the movement of the left hand of the third user (320), specifically the movement of fingers or palm included in the left hand, the movement of the left wrist, etc.

[0061] An external electronic device (400) can provide inertial information to an electronic device (100) through a communication interface. In one embodiment of the present disclosure, the external electronic device (400) transmits inertial information to the surroundings in a broadcast manner, and the electronic device (100) can obtain inertial information through a communication interface included in the electronic device (100). Hereinafter, the transmission method of inertial information will be described later in FIGS. 5 and FIGS. 13.

[0062] In one embodiment of the present disclosure, the inertia information is provided in a broadcast manner, and the inertia information acquired by the electronic device (100) may not include information on which of the plurality of users (300, 310, 320) is wearing the external electronic device (400). Additionally, the inertia information may not include information on which change in inertia resulting from the movement of a part of the body is acquired by the external electronic device (400).

[0063] In one embodiment of the present disclosure, an electronic device (100) can obtain a plurality of gesture information from gestures of a plurality of users (300, 310, 320) estimated from a user image.

[0064] In one embodiment of the present disclosure, "gesture information" may include information regarding changes in the relative positions of a plurality of feature points included in the gesture. In one embodiment of the present disclosure, as at least one user uses the electronic device (100), the electronic device (100) may acquire at least one gesture information. Hereinafter, gesture information will be described in reference to FIGS. 4 and FIGS. 7.

[0065] The electronic device (100) of the present disclosure can obtain mapping gesture information corresponding to the inertial information among the gesture information, based on a plurality of gesture information and inertial information corresponding to each of the plurality of users. Accordingly, the electronic device (100) can determine which of the plurality of users (300, 310, 320) is wearing an external electronic device (400) that provides inertial information. In one embodiment of the present disclosure, the electronic device (100) can determine that the gesture information of the third user (320) corresponds to the obtained inertial information by comparing the gesture information of the first user (300), the gesture information of the second user (310), and the gesture information of the third user (320) with the inertial information.

[0066] In one embodiment of the present disclosure, an electronic device (100) can obtain mapping gesture information corresponding to inertial information among a plurality of gesture information using an Extended Kalman Filter (EKF). The electronic device (100) can obtain gesture information corresponding to the Kalman gain having the largest value among a plurality of Kalman gains of inertial information for each of the plurality of gesture information as mapping gesture information.

[0067] Additionally, the electronic device (100) can determine whether the inertia information includes a change in inertia according to the movement of which part of the user's body. In one embodiment of the present disclosure, each of the plurality of gesture information may include a plurality of sub-gesture information corresponding to a plurality of body regions of a person. Based on the plurality of sub-gesture information and inertia information included in each of the plurality of gesture information, the electronic device (100) can obtain mapping sub-gesture information corresponding to the region where the external electronic device (400) is located among the plurality of sub-gesture information. In one embodiment of the present disclosure, the electronic device (100) can compare the plurality of sub-gesture information and inertia information included in each of the gesture information of the first user (300), the gesture information of the second user (310), and the gesture information of the third user (320), and confirm that the sub-gesture information corresponding to the left hand region of the third user (320) corresponds to the inertia information.

[0068] At this time, the electronic device (100) selects at least one sub-gesture information corresponding to a reference body area that is pre-set as an area where an external electronic device (400) can be located among a plurality of sub-gesture informations, and can obtain mapping sub-gesture information corresponding to inertial information among the selected at least one sub-gesture information using an extended Kalman filter.

[0069] Accordingly, the electronic device (100) can reduce the amount of computation and power consumption required to acquire mapping sub-gesture information corresponding to inertial information. Below, the operation of acquiring mapping gesture information corresponding to inertial information among a plurality of gesture information and the operation of acquiring mapping sub-gesture information will be described later in FIGS. 2, 9, and 10.

[0070] In one embodiment of the present disclosure, an electronic device (100) can obtain a gesture of a user corresponding to the mapping gesture information among a plurality of users (300, 310, 320) based on mapping gesture information and inertia information. At this time, the gesture estimated based on the mapping gesture information and inertia information may be referred to as a "final gesture." Additionally, the electronic device (100) can obtain a final gesture by the body area where the user's external electronic device (400) corresponding to the mapping gesture information is located, based on mapping sub-gesture information and inertia information. In one embodiment of the present disclosure, the electronic device (100) can obtain a final gesture of a third user (320) using mapping gesture information and inertia information. The electronic device (100) can obtain a final gesture by the left hand of the third user (320) using mapping sub-gesture information and inertia information.

[0071] In one embodiment of the present disclosure, an electronic device (100) can obtain a final gesture based on mapping gesture information and inertial information using an extended Kalman filter. At this time, the mapping gesture information may include mapping sub-gesture information, and the electronic device (100) can obtain a final gesture based on mapping sub-gesture information and inertial information using an extended Kalman filter. Below, the operation of obtaining a final gesture will be described later in FIGS. 2 and FIGS. 16.

[0072] In one embodiment of the present disclosure, the accuracy of the user's final gesture recognized by the electronic device (100) can be increased by using inertial information obtained from an external electronic device (400) together with gesture information obtained from a user image. Specifically, even if part of the user's body included in the user image is obscured by an obstacle or another user, or if the resolution of the user image is low, the accuracy of the operation to recognize the user's final gesture can be increased by considering inertial information together. In addition, since the user image can be considered together when recognizing the user's final gesture, the drift error caused by accumulated errors over time that may occur when recognizing the user's gesture by considering only inertial information can be prevented or reduced.

[0073] Additionally, in one embodiment of the present disclosure, the electronic device (100) can utilize inertial information provided by an external electronic device (400) that has not previously been paired with the electronic device (100) by utilizing mapping gesture information corresponding to inertial information provided in a broadcast manner among at least one user's gesture information to acquire the final gesture. Accordingly, the electronic device (100) according to the present disclosure can acquire the user's final gesture by utilizing inertial information provided by an external electronic device (400) newly worn by a user using the electronic device (100) or inertial information provided by an external electronic device (400) worn by a user who has come to use the electronic device (100) for the first time.

[0074] The electronic device (100) of the present disclosure can control the operation of the electronic device (100) based on a final gesture.

[0075] However, the present disclosure is not limited thereto, and the electronic device (100) may be implemented as an electronic device of various shapes, such as a desktop, set-top box, or server device, which does not include a display. In this case, the electronic device (100) may provide the acquired final gesture to an external electronic device through an input / output interface. Additionally, the electronic device (100) may provide the acquired final gesture to an external server through a communication interface.

[0076] In addition, the electronic device (100) can estimate the gestures of the first user (300) and the second user (310) from the user image.

[0077] FIG. 2 is a block diagram for explaining the configuration of an electronic device according to one embodiment of the present disclosure.

[0078] Referring to FIGS. 1 and 2, in one embodiment of the present disclosure, an electronic device (100) may include a display (110), a camera (120), a memory (130), at least one processor (e.g., including a processing circuit, 140), an input / output interface (e.g., including an input / output circuit, 150), and a communication interface (e.g., including a communication circuit, 160).

[0079] However, not all components illustrated in FIG. 2 are essential components. The electronic device (100) may be implemented by more components than those illustrated in FIG. 2, or by fewer components. In one embodiment of the present disclosure, the electronic device (100) may not include a display (110).

[0080] A display (110), camera (120), memory (130), at least one processor (140), input / output interface (150), and communication interface (170) included in the electronic device (100) can each be electrically connected to one another.

[0081] In one embodiment of the present disclosure, the display (110) may include any one of a liquid crystal display, a plasma display, an organic light emitting diode display, or an inorganic light emitting diode display. However, the present disclosure is not limited thereto, and the display (110) may include other types of displays capable of displaying an image (200).

[0082] In one embodiment of the present disclosure, at least one processor (140) includes various processing circuits and controls a display (110) to display an image (200), so that the electronic device (100) can be provided to a plurality of users (300, 310, 320).

[0083] In one embodiment of the present disclosure, the camera (120) can obtain user images by photographing a plurality of users (300, 310, 320) using the electronic device (100). In one embodiment of the present disclosure, the camera (120) may include an RGB camera capable of obtaining an image containing RGB information. However, the present disclosure is not limited thereto, and the camera (120) may include a stereo camera comprising two RGB cameras, an RGB-Depth camera that obtains an image containing RGB information and depth information, or a black and white camera that obtains a black and white image, and is not limited to any one of these.

[0084] At least one processor (140) controls a camera (120) to photograph at least one user (300) using the electronic device (100), so that the electronic device (100) can acquire a user image.

[0085] In one embodiment of the present disclosure, the memory (130) may store instructions, data structures, and program code that can be read by at least one processor (140). In one embodiment of the present disclosure, the memory (130) may be one or more. Operations performed by the electronic device (100) may be implemented by at least one processor (140) executing the instructions or code of a program stored in the memory (130).

[0086] In one embodiment of the present disclosure, the memory (130) may include at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), Mask ROM, Flash ROM, etc.), a hard disk drive (HDD), or a solid-state drive (SSD).

[0087] In one embodiment of the present disclosure, the memory (130) may not exist separately and may be configured to be included in at least one processor (140).

[0088] In one embodiment of the present disclosure, instructions or program code for performing functions or operations of an electronic device (100) may be stored in the memory (130). The instructions, algorithms, data structures, program code, and application programs stored in the memory (130) may be implemented in a programming or scripting language such as, for example, C, C++, Java, Python, assembler, etc.

[0089] In one embodiment of the present disclosure, various types of modules that can be used to perform the operation of the electronic device (100) may be stored in the memory (130).

[0090] In one embodiment of the present disclosure, the memory (130) may store an image acquisition module (131), a gesture estimation module (132), a gesture mapping module (134), and a final gesture acquisition module (135), and each module may include various processing circuits and / or executable program instructions. However, not all modules illustrated in FIG. 2 are required. More modules than those illustrated in FIG. 2 may be stored in the memory (130), or fewer modules may be stored.

[0091] In one embodiment of the present disclosure, the memory (130) may further store a module composed of instructions or program codes regarding an operation or function of selecting at least one sub-gesture information corresponding to a preset reference body area among a plurality of sub-gesture informations, or a module composed of instructions or program codes regarding an operation or function of determining whether the period for acquiring inertia information is longer than a preset threshold period.

[0092] In one embodiment of the present disclosure, a 'module' included in the memory (130) may mean a unit that processes a function or operation performed by at least one processor (140). The 'module' included in the memory (130) may be implemented as software such as instructions, algorithms, data structures, or program code.

[0093] In one embodiment of the present disclosure, the image acquisition module (131) may be composed of instructions or program code regarding an operation or function of acquiring a user image by photographing at least one user using the electronic device (100) through a camera (120).

[0094] In one embodiment of the present disclosure, by having at least one processor (140) execute instructions or program code of an image acquisition module (131), the electronic device (100) can acquire a user image by photographing at least one user using the electronic device (100) through a camera (120).

[0095] In one embodiment of the present disclosure, the gesture estimation module (132) may be composed of instructions or program code regarding an operation or function of estimating at least one user’s gesture from an acquired user image.

[0096] In one embodiment of the present disclosure, the gesture estimation module (132) may include instructions or program code regarding an operation or function of detecting a plurality of feature points and connecting the detected plurality of feature points to detect a person's pose.

[0097] In one embodiment of the present disclosure, the gesture estimation module (132) may include a pose estimation model that estimates a person's pose. However, the present disclosure is not limited thereto, and the gesture estimation module (132) may include a pre-trained artificial intelligence model (133) that infers at least one user's pose included in a user image. In one embodiment of the present disclosure, the gesture estimation module (132) may include models such as OpenPose, AlphaPose, and CPN (Cascaded Pyramid Network). The gesture estimation module (132) may estimate a person's pose from an image using a deep learning-based model such as CNN (Convolutional Neural Network), Transformer, Vision Transformer, etc., and the artificial intelligence model (133) in the present disclosure is not limited to the examples described above.

[0098] In one embodiment of the present disclosure, by having at least one processor (140) execute instructions or program code of a gesture estimation module (132), the electronic device (100) can estimate at least one user gesture included in an acquired user image.

[0099] In one embodiment of the present disclosure, the gesture estimation module (132) may include instructions or program code regarding an operation or function for acquiring gesture information including relative positional changes of a plurality of feature points included in at least one estimated user gesture.

[0100] In one embodiment of the present disclosure, the relative positions of a plurality of feature points included in a gesture may change over time according to linear movement in at least one of the x-axis, y-axis, or z-axis of the plurality of feature points included in the gesture. Additionally, the relative positions of a plurality of feature points may change over time according to rotational movement around at least one of the x-axis, y-axis, or z-axis of the plurality of feature points included in the gesture. In one embodiment of the present disclosure, the gesture information may include information regarding changes in the relative positions of a plurality of feature points included in at least one gesture.

[0101] In one embodiment of the present disclosure, by having at least one processor (140) execute instructions or program code of a gesture estimation module (132), the electronic device (100) can obtain at least one gesture information corresponding to each of at least one user based on at least one user's gesture estimated from a user image.

[0102] In one embodiment of the present disclosure, the gesture mapping module (134) may include instructions or program code regarding an operation or function of acquiring mapping gesture information corresponding to inertial information among at least one gesture information based on inertial information acquired through a communication interface (160) and at least one gesture information.

[0103] In one embodiment of the present disclosure, the inertial information obtained through the communication interface (160) may include 6 DoF (6 Degrees of Freedom) values ​​including acceleration values ​​of three-dimensional axes (x-axis, y-axis, and z-axis) and angular velocity values ​​of rotation (roll, yaw, pitch) around the three-dimensional axes according to the movement of a user wearing an external electronic device (400).

[0104] However, the present disclosure is not limited thereto, and the inertial information obtained through the communication interface (160) may include 3DoF (3 Degrees of Freedom) values ​​including information on acceleration values ​​in three dimensions (x-axis, y-axis, and z-axis) according to the movement of a user wearing an external electronic device (400).

[0105] In one embodiment of the present disclosure, the inertial information according to the gesture of a user wearing an external electronic device (400) and the gesture information obtained from a user image obtained by capturing the gesture of the user wearing the external electronic device (400) with a camera (120) may not differ significantly since they are obtained from the same gesture.

[0106] On the other hand, the inertial information based on the gesture of a user wearing an external electronic device (400) and the gesture information obtained from a user image obtained by capturing the gesture of another person, not the user wearing the external electronic device (400), with a camera (120) can be obtained from different gestures, so there may be a large difference.

[0107] In one embodiment of the present disclosure, the gesture mapping module (134) may include instructions or program code regarding an operation or function of obtaining mapping gesture information corresponding to inertial information among at least one gesture information, based on at least one gesture information obtained from the gesture estimation module (132) and inertial information obtained from the communication interface (160). The gesture mapping module (134) may include instructions or program code regarding an operation or function of obtaining the gesture information with the smallest difference from the inertial information among at least one gesture information as mapping gesture information.

[0108] In one embodiment of the present disclosure, the gesture mapping module (134) may include instructions or program code regarding an operation or function of obtaining at least one kalman gain of inertia information for each of at least one gesture information using an extended kalman filter, and obtaining gesture information corresponding to the kalman gain having the largest value among the at least one kalman gains as mapping gesture information.

[0109] In one embodiment of the present disclosure, by having at least one processor (140) execute instructions or program code of a gesture mapping module (134), the electronic device (100) can obtain mapping gesture information corresponding to the inertial information among at least one gesture information based on at least one gesture information and inertial information.

[0110] In one embodiment of the present disclosure, at least one gesture information may include a plurality of sub-gesture information corresponding to each of a plurality of body regions of each user. The gesture mapping module (134) may include instructions or program code regarding an operation or function of obtaining at least one Kalman gain of inertia information for each of the plurality of sub-gesture information included in the at least one gesture information using an extended Kalman filter, and obtaining sub-gesture information corresponding to the Kalman gain having the largest value among the at least one Kalman gain as mapping sub-gesture information.

[0111] In one embodiment of the present disclosure, by having at least one processor (140) execute instructions or program code of a gesture mapping module (134), the electronic device (100) can obtain mapping sub-gesture information corresponding to inertial information among a plurality of sub-gesture information based on a plurality of sub-gesture information and inertial information.

[0112] In one embodiment of the present disclosure, the final gesture acquisition module (135) may include instructions or program code regarding an operation or function to acquire the final gesture of a user corresponding to the mapping gesture information among at least one user based on mapping gesture information and inertia information.

[0113] The final gesture acquisition module (135) includes a sensor fusion model and may include instructions or program code regarding an action or function to acquire the user's final gesture from mapping gesture information and inertial information through the same. In one embodiment of the present disclosure, the final gesture acquisition module (135) may include an extended Kalman filter algorithm.

[0114] However, the present disclosure is not limited thereto, and the final gesture acquisition module (135) may include a pre-trained artificial intelligence model (133) to infer features by combining data acquired from different types of sensors. In one embodiment of the present disclosure, the final gesture acquisition module (135) may include a deep learning-based model such as a Convolutional Neural Network (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), etc., and the artificial intelligence model in the present disclosure is not limited to the examples described above.

[0115] In one embodiment of the present disclosure, by having at least one processor (140) execute instructions or program code of the final gesture acquisition module (135), the electronic device (100) can acquire the final gesture of at least one user corresponding to the mapping gesture information based on a plurality of sub-gesture information and inertia information.

[0116] In one embodiment of the present disclosure, at least one processor (140) may be configured to control a series of processes to operate an electronic device (100) according to the embodiments described below, and may be composed of one or more processors.

[0117] In one embodiment of the present disclosure, at least one processor (140) may be composed of at least one of a Central Processing Unit, a microprocessor, a Graphic Processing Unit, an Application Processor (AP), Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), Communication Processor (CP), a Neural Processing Unit, or an AI-dedicated processor designed with a hardware structure specialized for the learning and processing of an AI model, but is not limited thereto. Accordingly, the processor 140 may include various processing circuits and / or a plurality of processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits comprising at least one processor, and at least one of the at least one processor may be configured to perform various functions described herein individually and / or collectively in a distributed manner. When “one processor,” “at least one processor,” and “one or more processors” as used herein are described as being configured to perform multiple functions, these terms include, but are not limited to, situations where, for example, one processor performs part of the mentioned functions and other processor(s) perform other parts of the mentioned functions, and situations where a single processor can perform all the mentioned functions.Additionally, at least one processor may include a combination of processors performing various mentioned / disclosed functions, and may be performed, for example, in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0118] In one embodiment of the present disclosure, if one or more processors included in at least one processor (140) are artificial intelligence dedicated processors, said artificial intelligence dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0119] In one embodiment of the present disclosure, at least one processor (140) may be composed of a circuit such as a System on Chip (SoC) or an Integrated Circuit (IC).

[0120] In one embodiment of the present disclosure, at least one processor (140) can execute various types of modules stored in memory (130). At least one processor (140) can execute at least one instruction constituting the various types of modules stored in memory (130) individually or collectively. By executing a program or at least one instruction stored in memory (130), at least one processor (140) can process data according to a predefined operation rule or artificial intelligence model.

[0121] In one embodiment of the present disclosure, at least one processor (140) may include a plurality of processors. In one embodiment of the present disclosure, at least one module among a plurality of modules in memory (130) may be executed by any one of the plurality of processors. The remaining modules among the plurality of modules stored in memory (130) may be executed by another processor among the plurality of processors.

[0122] In one embodiment of the present disclosure, the input / output interface (150) includes various types of input / output circuits and, under the control of at least one processor (140), may receive a user image obtained by photographing a user using the electronic device (100) from an external electronic device, etc. The electronic device (100) may also obtain a user image from an external electronic device, etc. through the input / output interface (150), rather than a user image obtained by a camera (120).

[0123] Additionally, the input / output interface (150) may provide the final gesture obtained according to the present disclosure to an external electronic device under the control of at least one processor (140). At this time, the external electronic device may control the operation of the external electronic device based on the final gesture obtained from the electronic device (100).

[0124] In one embodiment of the present disclosure, the input / output interface (150) may perform input / output operations with an external electronic device using at least one of an input / output method including an HDMI port (High-Definition Multimedia Interface port), DVI (Digital Visual Interface), a component jack, a PC port, or a USB port (Universal Serial Bus port). However, the present disclosure is not limited to the above-mentioned input / output methods.

[0125] In one embodiment of the present disclosure, the communication interface (160) includes various types of communication circuits and can perform data communication with an external server or an external electronic device under the control of at least one processor (140).

[0126] The communication interface (160) can perform data communication with an external server or an external electronic device using at least one of the data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet), WiMAX (World Interoperability for Microwave Access), SWAP (Shared Wireless Access Protocol), WiGig (Wireless Gigabit Alliance), and RF communication.

[0127] In one embodiment of the present disclosure, the communication interface (160) may provide the final gesture obtained according to the present disclosure to an external server or an external electronic device under the control of at least one processor (140).

[0128] FIG. 3 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0129] Referring to FIGS. 1, 2 and 3, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include the step (S100) of obtaining a user image by photographing at least one user using the electronic device (100) through a camera (120).

[0130] In step S100, at least one processor (140) executes instructions or program code of the image acquisition module (131), so that the electronic device (100) can acquire a user image by photographing at least one user using the electronic device (100) through the camera (120).

[0131] In one embodiment of the present disclosure, the method of operation of an electronic device (100) may include the step (S200) of estimating at least one user's gesture from a user image and obtaining at least one gesture information corresponding to each of at least one user.

[0132] In step S200, by having at least one processor (140) execute instructions or program code of the gesture estimation module (132), the electronic device (100) can estimate at least one user's gesture from a user image and obtain at least one gesture information corresponding to each of at least one user.

[0133] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (S300) of obtaining inertial information from an external electronic device (400) through a communication interface (160).

[0134] In step S300, at least one processor (140) controls the operation of the communication interface (160), so that the electronic device (100) can obtain inertial information from an external electronic device (400) through the communication interface (160). At this time, the inertial information may be transmitted by the external electronic device (400) in a broadcast manner.

[0135] In one embodiment of the present disclosure, the method of operation of an electronic device (100) may include the step (S400) of obtaining mapping gesture information corresponding to the inertial information among at least one gesture information, based on at least one gesture information and inertial information.

[0136] In step S400, by having at least one processor (140) execute instructions or program code of the gesture mapping module (134), the electronic device (100) can obtain mapping gesture information corresponding to the inertial information among at least one gesture information based on at least one gesture information and inertial information. After calculating at least one Kalman gain of the inertial information for each of at least one gesture information using an extended Kalman filter, the electronic device (100) can obtain gesture information corresponding to the Kalman gain having the largest value among the calculated at least one Kalman gains as mapping gesture information.

[0137] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (S500) of obtaining the final gesture of a user corresponding to the mapping gesture information among at least one user based on mapping gesture information and inertia information.

[0138] In step S500, by having at least one processor (140) execute instructions or program code of the final gesture acquisition module (135), the electronic device (100) can acquire the final gesture of at least one user corresponding to the mapping gesture information based on mapping gesture information and inertial information. The electronic device (100) can acquire the final gesture of the user corresponding to the mapping gesture information based on mapping gesture information and inertial information by using an extended Kalman filter.

[0139] In this case, it goes without saying that at least one user and at least one gesture information do not mean one user and one gesture information. It goes without saying that at least one user may mean one user or two or more users, and at least one gesture information may mean one gesture information or two or more gesture information.

[0140] FIG. 4 is a diagram illustrating gesture information estimated from a user image according to one embodiment of the present disclosure.

[0141] Referring to FIG. 1, FIG. 2 and FIG. 4, in one embodiment of the present disclosure, FIG. 4 shows a user image obtained by photographing three users (300, 310, 320) using an electronic device (100), a plurality of feature points included in each of the three users (300, 310, 320) detected from the user image through the electronic device (100), and a line connecting each of the plurality of feature points.

[0142] In one embodiment of the present disclosure, the electronic device (100) can estimate a shape corresponding to a plurality of detected feature points and a shape formed by connecting the plurality of feature points with lines as a gesture of three users (300, 310, 320).

[0143] In one embodiment of the present disclosure, an electronic device (100) can obtain gesture information including information about the relative positional change over time of a plurality of feature points included in a gesture from a gesture estimated from a user image.

[0144] In one embodiment of the present disclosure, the user image may include multiple body regions of each of the three users (300, 310, 320). In one embodiment of the present disclosure, the multiple body regions may include a left hand region (321), a right hand region (322), a left ear region (323), a right ear region (324), a waist region (325), a left foot region (326), a right foot region (327), etc. However, the present disclosure is not limited thereto and may include various body regions in addition to the body regions shown in FIG. 4.

[0145] FIG. 4 is illustrated as including the full body of each of the three users (300, 310, 320) in the user image, but the present disclosure is not limited thereto. Depending on the angle of view of the camera (120), the distance between the camera (120) and the three users (300, 310, 320), the arrangement between the camera (120) and the three users (300, 310, 320), etc., the user image may include only a portion of the body of each of the three users (300, 310, 320).

[0146] In one embodiment of the present disclosure, the gesture information may include a plurality of sub-gesture information corresponding to a plurality of body regions of each of the three users (300, 310, 20). In this case, the "sub-gesture information" may include information regarding the relative positional changes of a plurality of feature points located in the corresponding body regions.

[0147] In one embodiment of the present disclosure, the sub-gesture information corresponding to the left-hand area (321) may include information regarding relative positional changes of a plurality of feature points detected from a left-hand image located in the left-hand area (321). In one embodiment of the present disclosure, the sub-gesture information corresponding to the right-hand area (322) may include information regarding relative positional changes of a plurality of feature points detected from a right-hand image located in the right-hand area (322).

[0148] FIG. 5 is a diagram illustrating inertial information obtained from an external electronic device according to one embodiment of the present disclosure.

[0149] Referring to FIG. 1, FIG. 2 and FIG. 5, in one embodiment of the present disclosure, FIG. 5 illustrates a part of the body (500) of a user wearing an external electronic device (400) among at least one user using an electronic device (100) and the external electronic device (400).

[0150] In one embodiment of the present disclosure, the external electronic device (400) may be a smart watch, and the part of the user's body (500) on which the external electronic device (400) is worn may be the left wrist. However, the present disclosure is not limited thereto, and it is understood that the location on the body on which the external electronic device (400) is worn may vary depending on the type or shape of the external electronic device (400).

[0151] In one embodiment of the present disclosure, the external electronic device (400) may include an inertial measurement sensor. In one embodiment of the present disclosure, the inertial measurement sensor may include an acceleration sensor and an angular velocity sensor (e.g., a gyroscope). The inertial measurement sensor may obtain acceleration values ​​of three-dimensional axes (x-axis, y-axis, and z-axis) according to the user's movement through the acceleration sensor. The inertial measurement sensor may obtain angular velocity values ​​of rotation (roll, yaw, pitch) around three-dimensional axes according to the user's movement through the angular velocity sensor. The external electronic device (400) may obtain inertial information (510) by detecting the movement of the body of the user wearing the external electronic device (400), for example, the movement of the left wrist (500) up, down, left, and right, or the rotation of the left wrist (500) in a counter-clockwise or clockwise direction.

[0152] In one embodiment of the present disclosure, as a slight movement or rotation occurs in the left wrist (500) according to the movement of the fingers included in the left hand, the external electronic device (400) may detect the movement of the fingers included in the left hand and obtain inertial information (510).

[0153] Additionally, the external electronic device (400) may recognize the gesture of the left hand of the user wearing the external electronic device (400) through a pre-trained artificial intelligence model that can infer what movement or gesture the fingers included in the left hand are making according to the acquired inertial information (510).

[0154] In one embodiment of the present disclosure, an external electronic device (400) can transmit inertial information (510) obtained through a communication interface to the surroundings of the external electronic device (400). At this time, the inertial information (510) can be transmitted to the surroundings of the external electronic device (400) in a broadcast manner through the communication interface.

[0155] In one embodiment of the present disclosure, an external electronic device (400) may broadcast inertia information (510) included in an advertising packet of Bluetooth Low Energy (BLE) through a communication interface. Additionally, the external electronic device (400) may broadcast inertia information (510) included in an advertising packet of Wi-Fi Aware through a communication interface. However, the present disclosure is not limited thereto, and it is understood that the external electronic device (400) may use various communication methods to provide inertia information (510) in a broadcast manner to unpaired surrounding electronic devices.

[0156] In one embodiment of the present disclosure, the external electronic device (400) can reduce the power consumption required to perform the operation of providing inertia information (510) to other electronic devices in the vicinity by periodically broadcasting a packet containing inertia information (510) without maintaining a connection with other electronic devices in the vicinity.

[0157] Electronic devices located around an external electronic device (400) can obtain inertial information (510) transmitted by the external electronic device (400). In one embodiment of the present disclosure, an electronic device (100) can also obtain inertial information (510) provided by the external electronic device (400) through a communication interface (160).

[0158] FIG. 6 is a flowchart illustrating an operation to acquire mapping sub-gesture information corresponding to a specific body region where an external electronic device, which generates inertial information, is located among a plurality of sub-gesture information according to an embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 3, and redundant descriptions are omitted.

[0159] Referring to FIGS. 1, FIGS. 2, FIGS. 3 and FIGS. 6, in one embodiment of the present disclosure, a method of operation of an electronic device (100) may include a step (S410) of obtaining mapping sub-gesture information corresponding to a specific body area where an external electronic device (400) is located among a plurality of sub-gesture information included in each of at least one gesture information, based on at least one gesture information and inertia information.

[0160] In step S410, by having at least one processor (140) execute instructions or program code of the gesture mapping module (134), the electronic device (100) can obtain mapping sub-gesture information corresponding to a specific body area where the external electronic device (400) is located, among a plurality of sub-gesture information included in each of at least one gesture information, based on at least one gesture information and inertia information.

[0161] In one embodiment of the present disclosure, step S410 may be performed after step S300.

[0162] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (S510) of obtaining a final gesture of a specific body area of ​​at least one user corresponding to the mapping sub-gesture information based on mapping sub-gesture information.

[0163] In step S510, by having at least one processor (140) execute instructions or program code of the final gesture acquisition module (135), the electronic device (100) can acquire the final gesture of a specific body area of ​​at least one user corresponding to the mapping sub-gesture information based on mapping sub-gesture information and inertia information.

[0164] In one embodiment of the present disclosure, with reference to FIGS. 4 and FIGS. 5, in step S510, the electronic device (100) may obtain a final gesture by the left wrist (500) or left hand of the third user (320).

[0165] FIG. 7 is a flowchart illustrating an operation of comparing sub-gesture information corresponding to a preset reference body area among gesture information and inertia information according to an embodiment of the present disclosure. FIG. 8 is a diagram illustrating an operation of comparing sub-gesture information corresponding to a preset reference body area among gesture information and inertia information according to an embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 3, and redundant descriptions are omitted.

[0166] Referring to FIGS. 2, FIGS. 3, FIGS. 4 and FIGS. 7, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S310) of selecting at least one sub-gesture information corresponding to a preset reference body area among a plurality of sub-gesture information included in each of at least one gesture information. In one embodiment of the present disclosure, step S310 may be performed after step S300.

[0167] In one embodiment of the present disclosure, the "reference body area" may be a pre-set area among a plurality of body areas of a person where an external electronic device (400) is determined to be located. In one embodiment of the present disclosure, the reference body area may be pre-set to include an area expected to be a place where a person can wear the external electronic device (400), for example, a hand area including fingers for wearing a smart ring, a wrist area for wearing a smart watch, an ear area for wearing earphones or headphones, a head area for wearing a head-mounted display device, etc. However, the present disclosure is not limited thereto, and the reference body area may be set to include other areas. Furthermore, the size of the area set may vary, such as the hand area and the wrist area being divided into a single area.

[0168] In step S310, by having at least one processor (140) execute instructions or program code of memory (130), the electronic device (100) can select at least one sub-gesture information corresponding to a preset reference body area among a plurality of sub-gesture information included in each of at least one gesture information.

[0169] In one embodiment of the present disclosure, when a reference body region includes a hand region, a wrist region, and an ear region, the electronic device (100) may select three sub-gesture information corresponding to the hand region, wrist region, and ear region, respectively, among a plurality of sub-gesture information included in each of at least one gesture information. However, if there is no sub-gesture information corresponding to the reference body region among the plurality of sub-gesture information, the electronic device (100) may not select information corresponding to the region.

[0170] In one embodiment of the present disclosure, the method of operation of an electronic device (100) may include a step (S420) of obtaining mapping sub-gesture information corresponding to a specific body region among the selected at least one sub-gesture information included in each of the at least one gesture information, based on selected at least one sub-gesture information and inertia information. At this time, step S420 may be included in step S410.

[0171] In step S420, by having at least one processor (140) execute instructions or program code of the gesture mapping module (134), the electronic device (100) can obtain mapping sub-gesture information corresponding to a specific body area among the selected at least one sub-gesture information included in each of the at least one gesture information, based on selected at least one sub-gesture information and inertia information.

[0172] Referring to FIG. 8, in one embodiment of the present disclosure, FIG. 8 illustrates one sub-gesture information (800) corresponding to a reference body area including the left hand and wrist among a plurality of sub-gesture information. In the one sub-gesture information (800), a plurality of feature points (801) and a line (802) connecting each of the plurality of feature points (801) are illustrated.

[0173] Additionally, FIG. 8 illustrates inertial information (810) acquired by an electronic device (100). The inertial information (810) may include 6DoF values, and may include x-axis acceleration values ​​(811), y-axis acceleration values ​​(812), z-axis acceleration values ​​(813), roll angular velocity values ​​(814), pitch angular velocity values ​​(815), and yaw angular velocity values ​​(816).

[0174] In one embodiment of the present disclosure, the electronic device (100) can compare the x-axis acceleration value of one sub-gesture information (800) with the x-axis acceleration value (811) included in the inertia information (810) based on the linear movement information along the x-axis included in one sub-gesture information (800). The electronic device (100) can compare the y-axis acceleration value of one sub-gesture information (800) with the y-axis acceleration value (812) included in the inertia information (810) based on the linear movement information along the y-axis included in one sub-gesture information (800). The electronic device (100) can compare the z-axis acceleration value of one sub-gesture information (800) with the z-axis acceleration value (813) included in the inertia information (810) based on the linear movement information along the z-axis included in one sub-gesture information (800).

[0175] In one embodiment of the present disclosure, the electronic device (100) can compare the roll angular velocity value of one sub-gesture information (800) with the roll angular velocity value (814) included in the inertia information (810) based on rotational motion information about the x-axis included in one sub-gesture information (800). The electronic device (100) can compare the pitch angular velocity value of one sub-gesture information (800) with the pitch angular velocity value (815) included in the inertia information (810) based on rotational motion information about the y-axis included in one sub-gesture information (800). The electronic device (100) can compare the yaw angular velocity value of one sub-gesture information (800) with the yaw angular velocity value (816) included in the inertia information (810) based on rotational motion information about the z-axis included in one sub-gesture information (800).

[0176] In one embodiment of the present disclosure, the electronic device (100) may determine the sub-gesture information that has the smallest difference from the inertia information (810) among a plurality of sub-gesture information as the mapping sub-gesture information. At this time, having the smallest difference may mean that the body area corresponding to the sub-gesture information is the same area as the body area where the external electronic device (400) providing the inertia information (810) is located.

[0177] On the other hand, among the multiple sub-gesture information, the remaining sub-gesture information corresponding to a body region different from the body region where the external electronic device (400) providing the inertial information (810) is located may differ significantly from the inertial information (810).

[0178] The operation of the electronic device (100) in steps S400, S410, and S420 below will be described later in FIG. 9 and FIG. 10.

[0179] FIG. 9 is a flowchart illustrating an operation to obtain mapping gesture information based on at least one gesture information and inertia information using an extended Kalman filter according to an embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 3, and redundant descriptions are omitted.

[0180] Referring to FIGS. 2, FIGS. 3 and FIGS. 9, in one embodiment of the present disclosure, the method of operation of an electronic device (100) may include the step (S430) of obtaining at least one Kalman gain of inertia information for each of at least one gesture information using an extended Kalman filter.

[0181] In one embodiment of the present disclosure, step S430 may be performed after step S300. Additionally, step S430 may be included in step S400.

[0182] In one embodiment of the present disclosure, the "extended Kalman filter" is a recursive filter that estimates the state of a nonlinear system based on measurements containing noise. The extended Kalman filter can predict the current state based on previously obtained estimates and a model of the nonlinear system designed in advance, taking into account the system in which the extended Kalman filter is used. The extended Kalman filter is an algorithm that obtains an estimate of the current state by updating the current estimate to reflect the residual between the current estimate and the currently obtained measurements.

[0183] In one embodiment of the present disclosure, "Kalman gain" may refer to a weight that determines the degree to which the residuals of the prediction and the measurement are reflected when updating the current prediction so that the residuals of the current prediction and the current measurement are reflected through an extended Kalman filter.

[0184] In one embodiment of the present disclosure, the Kalman gain value may increase as the noise included in the current measurement is smaller. The Kalman gain value may decrease as the noise included in the current measurement is larger.

[0185] In one embodiment of the present disclosure, the greater the value of the Kalman gain, the higher the degree to which the current measurement is reflected in the current estimate. The smaller the value of the Kalman gain, the lower the degree to which the current measurement is reflected in the current estimate.

[0186] In one embodiment of the present disclosure, the electronic device (100) may use a Kalman gain included in an extended Kalman filter in an operation to acquire mapping gesture information corresponding to inertial information among at least one gesture information.

[0187] In step S430, by having at least one processor (140) execute instructions or program code of the gesture mapping module (134), the electronic device (100) can obtain at least one kalman gain of inertia information for each of at least one gesture information.

[0188] In one embodiment of the present disclosure, the electronic device (100) can obtain at least one current prediction value in an extended Kalman filter by using at least one gesture information of each of at least one user utilizing the electronic device (100) as a previous measurement value. The electronic device (100) can use inertial information obtained through a communication interface (160) as a current measurement value. Accordingly, the electronic device (100) can obtain at least one Kalman gain of inertial information for each of at least one gesture information.

[0189] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (S440) of obtaining gesture information corresponding to the Kalman gain having the largest value among at least one Kalman gain as mapping gesture information. In one embodiment of the present disclosure, step S440 may be included in step S400.

[0190] In one embodiment of the present disclosure, since the movement, position, or shape, etc., of each of at least one user is different, at least one gesture information may include information having different values ​​for each. Inertia information may include information about the movement of a user wearing an external electronic device (400). Accordingly, at least one Kalman gain of the inertia information for each of at least one gesture information may have different values.

[0191] However, the present disclosure is not limited thereto, and when there is only one user using the electronic device (100), the gesture information acquired by the electronic device (100) is one, and the Kalman gain of the inertia information for the gesture information can also have one value.

[0192] In one embodiment of the present disclosure, the residual between the current predicted value and the current measured value based on gesture information obtained from at least one user wearing an external electronic device (400) may be referred to as the first residual. The residual between the current predicted value and the current measured value based on gesture information obtained from at least one user different from the user wearing an external electronic device (400) may be referred to as the second residual.

[0193] The first residual is the residual between the current prediction and the current measurement obtained based on the same movement of the same user, so it may have the smallest value among at least one residual. The second residual is the residual between the current prediction and the current measurement obtained based on the potentially different movements of different users, so it may have a value greater than the first residual.

[0194] In one embodiment of the present disclosure, if the user to whom gesture information is acquired and the user to whom inertia information is acquired are the same user, the inertia information may be valid information, not noise information, when acquiring the gesture of the said user. If the user to whom gesture information is acquired and the user to whom inertia information is acquired are different users, the inertia information may be noise information when acquiring the gesture of the said user.

[0195] In one embodiment of the present disclosure, when the user to whom gesture information is obtained and the user to whom inertia information is obtained are the same user, the accuracy of the obtained gesture may be increased by considering the gesture information and inertia information together. On the other hand, when the user to whom gesture information is obtained and the user to whom inertia information is obtained are different users, the accuracy of the obtained user's gesture may be decreased by considering the gesture information and inertia information together.

[0196] Therefore, the larger the residual between the current prediction and the current measurement, the greater the noise contained in the current measurement, which may result in a smaller Kalman gain for the inertial information regarding the gesture information obtained through the extended Kalman filter. Conversely, the smaller the residual between the current prediction and the current measurement, the smaller the noise contained in the current measurement, which may result in a larger Kalman gain for the inertial information regarding the gesture information obtained through the extended Kalman filter. In this case, the case where the Kalman gain is greatest may be when the user for whom the gesture information was obtained is the same as the user for whom the inertial information was obtained, resulting in the smallest residual value.

[0197] In step S440, by having at least one processor (140) execute the instructions or program code of the gesture mapping module (134), the electronic device (100) can obtain gesture information corresponding to the Kalman gain having the largest value among at least one Kalman gain as mapping gesture information.

[0198] However, the present disclosure is not limited thereto. The above description may be an example in which inertial information transmitted from one external electronic device surrounding the electronic device (100) is obtained through a communication interface (160). In one embodiment of the present disclosure, the electronic device (100) may obtain a plurality of inertial information transmitted from two or more external electronic devices surrounding it. In this case, the electronic device (100) may perform steps S430 and S440 for each of the plurality of inertial information. The electronic device (100) may obtain the gesture information having the largest Kalman gain value among at least one gesture information as mapping gesture information corresponding to each of the plurality of inertial information.

[0199] In one embodiment of the present disclosure, step S500 may be performed after step S440.

[0200] In one embodiment of the present disclosure, an extended Kalman filter may be used to fuse data acquired from different types of sensors to obtain a desired result. In step S500, by having at least one processor (140) execute instructions or program code of the final gesture acquisition module (125), the electronic device (100) may acquire a final gesture from mapping gesture information and inertial information using the extended Kalman filter.

[0201] In one embodiment of the present disclosure, the electronic device (100) may use mapping gesture information obtained at a first time point as a measurement value and obtain an estimate of the first time point by updating the initial prediction value based on the difference between the prediction value of the initial state and the mapping gesture information obtained at the first time point and the Kalman gain. The electronic device (100) may use inertia information obtained at a second time point as a measurement value and obtain an estimate of the second time point by updating the prediction value of the second time point based on the estimate of the first time point and the system model and the difference between the inertia information obtained at the second time point and the Kalman gain. At this time, the prediction value of the initial state may be set to an arbitrary value. Furthermore, it is obvious that inertia information at the first time point may be used as a measurement value and mapping gesture information at the second time point may be used as a measurement value.

[0202] In one embodiment of the present disclosure, the electronic device (100) can repeatedly perform the above operation based on mapping gesture information and inertia information obtained at a plurality of points in time to obtain an estimated value of the finally obtained point in time as the user's final gesture.

[0203] In one embodiment of the present disclosure, at least one gesture information may include a plurality of sub-gesture information corresponding to a plurality of body regions.

[0204] In one embodiment of the present disclosure, step S430 may include the step of obtaining at least one Kalman gain of inertia information for each of the plurality of sub-gesture information included in each gesture information using an extended Kalman filter.

[0205] In one embodiment of the present disclosure, by having at least one processor (140) execute instructions or program code of the gesture mapping module (134), the electronic device (100) can obtain at least one kalman gain of inertia information for each of the plurality of sub-gesture information included in each gesture information. Through this, the electronic device (100) can determine which body region of the user using the electronic device (100) the external electronic device (400) is worn in.

[0206] In one embodiment of the present disclosure, step S440 may include the step of obtaining sub-gesture information corresponding to the Kalman gain having the largest value among at least one Kalman gain as mapping sub-gesture information.

[0207] In one embodiment of the present disclosure, by executing instructions or program code of a gesture mapping module (134), the electronic device (100) can obtain sub-gesture information corresponding to the Kalman gain having the largest value among at least one Kalman gain as mapping sub-gesture information.

[0208] In one embodiment of the present disclosure, by comparing inertial information obtained through a communication interface (160) with gesture information obtained through a user image obtained through a camera (120), the electronic device (100) can identify a user wearing an external electronic device (400) among at least one user using the electronic device (100).

[0209] Additionally, the electronic device (100) can obtain, through an operation of comparing each of the multiple sub-gesture information corresponding to each of the multiple body regions included in the inertial information and gesture information, the sub-gesture information obtained from the user image of the body region corresponding to the position where the external electronic device (400) is worn among the multiple sub-gesture information obtained from the user image of the multiple body regions of the user wearing the external electronic device (400) as mapping sub-gesture information.

[0210] In one embodiment of the present disclosure, step S500 may include the step of obtaining a final gesture from mapping sub-gesture information and inertial information using an extended Kalman filter. The electronic device (100) may obtain a final gesture from mapping sub-gesture information and inertial information using an extended Kalman filter.

[0211] In one embodiment of the present disclosure, the electronic device (100) can acquire a gesture of a body region corresponding to mapping sub-gesture information by using not only the sub-gesture information acquired from the user image of the body region but also the inertial information acquired from an external electronic device (400) located in the body region. Through this, the electronic device (100) can acquire a gesture recognition result with high accuracy that takes into account both the user image through the camera (120) and the inertial information through the inertial measurement sensor of the external electronic device (400).

[0212] FIG. 10 is a diagram illustrating an operation to acquire mapping gesture information based on at least one sub-gesture information corresponding to a preset reference body area among a plurality of sub-gesture information, according to one embodiment of the present disclosure, and inertia information.

[0213] Referring to FIGS. 2, FIGS. 7, FIGS. 8 and FIG. 10, in one embodiment of the present disclosure, FIG. 10 illustrates the operation of an electronic device (100) when four users use the electronic device (100) and inertia information (1040) is obtained through the communication interface (160) of the electronic device (100). However, the present disclosure is not limited thereto, and the electronic device (100) may be used by a number of users different from four (e.g., one or five). Furthermore, the electronic device (100) may obtain other inertia information obtained from other external electronic devices in addition to the inertia information (1040). In this case, the electronic device (100) may perform the operation using the inertia information (1040) on other inertia information as well.

[0214] In one embodiment of the present disclosure, the electronic device (100) can obtain gesture information for each of the four users based on user images obtained by photographing the four users.

[0215] In one embodiment of the present disclosure, the gesture information of each of the four users may include a plurality of sub-gesture information corresponding to a plurality of body regions of a person.

[0216] In one embodiment of the present disclosure, the electronic device (100) may obtain inertia information (1040) through a communication interface (160). At this time, the inertia information (1040) may be obtained by measuring the change in inertia according to the movement of a specific body area of ​​one of the four users as the external electronic device (400) is positioned in a specific body area of ​​the user.

[0217] In one embodiment of the present disclosure, the electronic device (100) can obtain multiple Kalman gains of inertia information for each of the multiple sub-gesture information included in the gesture information of each of the four users using an extended Kalman filter. The electronic device (100) can obtain the sub-gesture information corresponding to the Kalman gain having the largest value among the multiple Kalman gains as mapping sub-gesture information corresponding to the inertia information (1040).

[0218] In one embodiment of the present disclosure, the electronic device (100) may select at least one sub-gesture information corresponding to a preset reference body region among a plurality of sub-gesture information included in each gesture information. The electronic device (100) may obtain at least one Kalman gain of inertia information for each of the selected at least one sub-gesture information using an extended Kalman filter. The electronic device (100) may obtain the sub-gesture information corresponding to the Kalman gain having the largest value among the at least one Kalman gain as mapping sub-gesture information corresponding to inertia information (1040).

[0219] By using at least one sub-gesture information selected to correspond to a reference body area, rather than multiple sub-gesture information, the amount of computation required for the electronic device (100) to acquire mapping sub-gesture information can be reduced.

[0220] In one embodiment of the present disclosure, FIG. 10 illustrates four sub-gesture information (1001, 1011, 1021, 1031) and inertia information (1040) selected corresponding to a left hand area included in a reference body area among a plurality of body areas of four users. At this time, other body areas may be included in the reference body area, and in this case, the electronic device (100) may obtain Kalman gain using the sub-gesture information and inertia information selected corresponding to the other body area.

[0221] The four users may be referred to as the first user, the second user, the third user, and the fourth user, and the four sub-gesture information (1001, 1011, 1021, 1031) may be referred to as the first sub-gesture information (1001) corresponding to the first user, the second sub-gesture information (1011) corresponding to the second user, the third sub-gesture information (1021) corresponding to the third user, and the fourth sub-gesture information (1031) corresponding to the fourth user.

[0222] In one embodiment of the present disclosure, the first case (1000) may be an operation of obtaining Kalman gains of the first sub-gesture information (1001) and inertia information (1040) using an extended Kalman filter. The second case (1010) may be an operation of obtaining Kalman gains of the second sub-gesture information (1011) and inertia information (1040) using an extended Kalman filter. The third case (1020) may be an operation of obtaining Kalman gains of the third sub-gesture information (1021) and inertia information (1040) using an extended Kalman filter. The fourth case (1030) may be an operation of obtaining Kalman gains of the fourth sub-gesture information (1031) and inertia information (1040) using an extended Kalman filter.

[0223] In one embodiment of the present disclosure, the value of the residual between the third sub-gesture information (1021) and the inertia information (1040) in the third case (1020) may be smaller than the value of the residual in each of the first case (1000), the second case (1010), and the fourth case (1030). The value of the Kalman gain of the inertia information (1040) for the third sub-gesture information (1021) in the third case (1020) may be larger than the value of the Kalman gain of the inertia information (1040) in each of the first case (1000), the second case (1010), and the fourth case (1030).

[0224] In one embodiment of the present disclosure, the electronic device (100) may determine the third sub-gesture information (1021), which has the largest Kalman gain value of the inertia information (1040), as the mapping sub-gesture information among the first to fourth sub-gesture information (1001, 1011, 1021, 1031). Accordingly, when the electronic device (100) acquires the gestures of four users using the electronic device (100), the gesture by the left hand area of ​​the third user may be acquired using the inertia information (1040) as well as the third sub-gesture information (1021).

[0225] At this time, the Kalman gain value of the inertia information (1040) in the third case (1020) may be greater than the Kalman gain value between the sub-gesture information and the inertia information (1040) corresponding to a body area other than the left hand area among the multiple body areas.

[0226] FIG. 11 is a flowchart illustrating an operation to acquire gesture information corresponding to the largest value among at least one filtering Kalman gain greater than a preset reference value as mapping gesture information, according to an embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 9, and redundant descriptions are omitted.

[0227] Referring to FIGS. 1, FIGS. 2, FIGS. 3, FIGS. 9 and FIGS. 11, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include the step (S431) of obtaining at least one filtered Kalman gain that is greater than a preset reference value among at least one Kalman gain.

[0228] In one embodiment of the present disclosure, step S431 may be performed after step S430. Additionally, step S431 may be included in step S400.

[0229] In one embodiment of the present disclosure, the "reference value" may be a value set to use only inertial information having a Kalman gain higher than that value for the accuracy of a gesture recognized through the electronic device (100), even when considering noise included in inertial information obtained through the communication interface (160), noise due to the distance between the external electronic device (400) and the electronic device (100), or noise included in gesture information obtained by the electronic device (100) through a user image.

[0230] In step S431, by having at least one processor (140) execute instructions or program code of the gesture mapping module (134), the electronic device (100) can obtain at least one filtering Kalman gain that is greater than a preset reference value among at least one Kalman gain.

[0231] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (S441) of obtaining gesture information corresponding to the filtering Kalman gain having the largest value among at least one filtering Kalman gain as mapping gesture information. In one embodiment of the present disclosure, step S441 may be included in step S400.

[0232] In step S441, by having at least one processor (140) execute instructions or program code of the gesture mapping module (134), the electronic device (100) can obtain gesture information corresponding to the filtering Kalman gain having the largest value among at least one filtering Kalman gain as mapping gesture information.

[0233] In one embodiment of the present disclosure, the electronic device (100) may not acquire gesture information corresponding to the Kalman gain having the largest value among at least one Kalman gain, if the Kalman gain is lower than a reference value. In this case, even if inertial information is acquired through the communication interface (160), the electronic device (100) may not use the inertial information when acquiring the user's gesture by determining that it does not correspond to at least one gesture information acquired from the user image (e.g., if it contains a large amount of noise components, or if it is inertial information transmitted from an external electronic device worn by another user who does not use the electronic device (100)).

[0234] Through this, the electronic device (100) can increase the accuracy and reliability of the operation of acquiring the user's gesture.

[0235] In one embodiment of the present disclosure, step S500 may be performed after step S441.

[0236] In one embodiment of the present disclosure, step S431 may include the step of obtaining at least one sub-filtering Kalman gain greater than a preset reference value among a plurality of sub-gesture informations each included in at least one gesture information.

[0237] In one embodiment of the present disclosure, step S441 may include the step of obtaining sub-gesture information corresponding to the sub-filtering Kalman gain having the largest value among at least one sub-filtering Kalman gain as mapping sub-gesture information.

[0238] In one embodiment of the present disclosure, the electronic device (100) may not acquire the gesture information corresponding to the Kalman gain having the largest value among at least one Kalman gain, if the Kalman gain is lower than a reference value. In this case, even if the inertial information is acquired through the communication interface (160), the electronic device (100) may not use the inertial information when acquiring the user's gesture by determining that it does not correspond to at least one sub-gesture information acquired from the user image (for example, if it contains a large amount of noise components, or if it is inertial information transmitted from an external electronic device worn by another user who does not use the electronic device (100), or if the location of the external electronic device is a body region different from a plurality of body regions included in the user image).

[0239] FIG. 12 is a flowchart illustrating the operation of an electronic device transmitting a request signal requesting inertial information when the period for acquiring inertial information is longer than a preset threshold period, according to an embodiment of the present disclosure. FIG. 13 is a diagram illustrating the operation of an electronic device acquiring inertial information transmitted in a broadcasting manner and the operation of an electronic device requesting inertial information, according to an embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 3, and redundant descriptions are omitted.

[0240] Referring to FIGS. 1, FIGS. 2, FIGS. 3 and FIGS. 12, in one embodiment of the present disclosure, the method of operating an electronic device (100) may include a step (S320) of checking whether the period during which inertia information is acquired is longer than a preset threshold period.

[0241] In one embodiment of the present disclosure, step S320 may be performed after step S300.

[0242] In one embodiment of the present disclosure, inertial information may be transmitted via a broadcast method from an external electronic device (400). The period during which the external electronic device (400) transmits inertial information may be gradually increased. The external electronic device (400) may reduce power consumption of the operation of transmitting inertial information by gradually increasing the period during which it transmits inertial information.

[0243] In one embodiment of the present disclosure, the “threshold period” may be a period set for the electronic device (100) to acquire a final gesture so that the electronic device (100) can utilize inertial information when acquiring the user’s final gesture.

[0244] In step S320, by having at least one processor (140) execute instructions or program code of memory (130), the electronic device (100) can determine whether the period during which inertia information is acquired is longer than a preset threshold period.

[0245] In one embodiment of the present disclosure, as it is determined that the period during which inertia information is acquired in step S320 is equal to or shorter than a preset threshold period, the electronic device (100) may perform step S400.

[0246] In one embodiment of the present disclosure, as the period during which inertial information is acquired in step S320 is longer than a preset threshold period, the method of operation of the electronic device (100) may include the step (S330) of transmitting a request signal requesting inertial information through a communication interface (160).

[0247] In one embodiment of the present disclosure, an electronic device (100) may transmit a request signal to surrounding devices in a broadcast manner through a communication interface (160). The electronic device (100) may broadcast by including the request signal in an advertising packet of Bluetooth Low Energy (BLE) through the communication interface (160). Additionally, the electronic device (100) may broadcast by including the request signal in an advertising packet of Wi-Fi Aware through the communication interface (160). However, the present disclosure is not limited thereto, and it is understood that the electronic device (100) may use various communication methods to provide the request signal in a broadcast manner to surrounding external electronic devices that are not paired.

[0248] However, the present disclosure is not limited thereto, and if the acquired inertial information includes information capable of identifying an external electronic device (400), such as the device address, device name, or device identifier of the external electronic device (400), the electronic device (100) may transmit a request signal to the surroundings including said identification information.

[0249] In step S330, the electronic device (100) can transmit a request signal requesting inertial information through the communication interface (160). When the request signal is obtained through the communication interface included in the external electronic device (400), the external electronic device (400) can transmit the inertial information again through the communication interface at a period shorter than the threshold period.

[0250] Referring to FIGS. 2, FIGS. 12 and FIGS. 13, in one embodiment of the present disclosure, FIG. 13 illustrates an electronic device (1310) placed within a specific space (1300), a first user (1320) using the electronic device (1310), a second user (1330), and a third user (1340) not using the electronic device (1310). In one embodiment of the present disclosure, the third user (1340) may be located in a different space distinct from the space containing the electronic device (1310), the first user (1320), and the second user (1330).

[0251] In one embodiment of the present disclosure, the electronic device (1310) can obtain first gesture information corresponding to the first user (1320) and second gesture information corresponding to the second user (1330) based on user images obtained by photographing a first user (1320) and a second user (1330) using the electronic device (1310) with a camera.

[0252] In one embodiment of the present disclosure, a second user (1330) may be wearing a first external electronic device (1331). A third user (1340) may be wearing a second external electronic device (1341). The first external electronic device (1331) may transmit first inertial information (1332) obtained according to the movement of a specific body area of ​​the second user (1330) where the first external electronic device (1331) is located, to the surroundings in a broadcast manner. The second external electronic device (1341) may transmit second inertial information (1342) obtained according to the movement of a specific body area of ​​the third user (1340) where the second external electronic device (1341) is located, to the surroundings in a broadcast manner.

[0253] At this time, the period during which the first external electronic device (1331) transmits the first inertial information (1332) and the period during which the second external electronic device (1341) transmits the second inertial information (1342) can be gradually increased. The first external electronic device (1331) and the second external electronic device (1341) can reduce power consumption by gradually increasing the period during which the first inertial information (1332) and the second inertial information (1342) are transmitted externally.

[0254] In one embodiment of the present disclosure, the electronic device (1310) can obtain first inertial information (1332) and second inertial information (1342) through a communication interface.

[0255] In one embodiment of the present disclosure, the electronic device (1310) can obtain the Kalman gain of the first inertial information (1332) for each of the first gesture information and the second gesture information using an extended Kalman filter. The electronic device (1310) can determine the second gesture information as mapping gesture information corresponding to the first inertial information (1332) by determining that the value of the Kalman gain of the first inertial information (1332) for the second gesture information is greater than the value of the Kalman gain of the first inertial information (1332) for the first gesture information. At this time, the value of the Kalman gain of the first inertial information (1332) for the second gesture information may be greater than a preset reference value.

[0256] In one embodiment of the present disclosure, the electronic device (1310) can obtain the Kalman gain of the second inertia information (1342) for each of the first gesture information and the second gesture information using an extended Kalman filter. The electronic device (1310) can determine that there is no gesture information corresponding to the second inertia information (1342) because the values ​​of the Kalman gains of the second inertia information (1342) for each of the first gesture information and the second gesture information are all smaller than a reference value.

[0257] In one embodiment of the present disclosure, an electronic device (1310) can obtain a gesture of a first user (1320) from first gesture information. The electronic device (1310) can obtain a gesture of a second user (1330) from second gesture information and first inertial information (1332) using an extended Kalman filter. At this time, the gesture of the second user (1330) obtained by fusing the second gesture information and the first inertial information (1332) may be referred to as the final gesture.

[0258] In one embodiment of the present disclosure, the electronic device (1310) may transmit a request signal (1311) requesting the first inertial information (1332) to the outside through a communication interface (160) as the period of the acquired first inertial information (1332) becomes longer than a threshold period. At this time, the request signal (1311) may be transmitted to the surroundings in a broadcast manner.

[0259] In one embodiment of the present disclosure, a first external electronic device (1331) may obtain a request signal (1311) through a communication interface. The first external electronic device (1331) may confirm that the packet of the request signal (1311) contains information requesting first inertia information (1332) and transmit the first inertia information (1332) to the outside at a period shorter than a threshold period.

[0260] In one embodiment of the present disclosure, a second external electronic device (1341) may obtain a request signal (1311) through a communication interface. The second external electronic device (1341) may determine that the packet of the request signal (1311) contains information requesting first inertial information (1332) rather than the second inertial information (1342) provided by itself, and may ignore it.

[0261] FIG. 13 illustrates first to third users (1320, 1330, 1340) located within a specific space (1300), but the present disclosure is not limited thereto. In one embodiment of the present disclosure, only at least one user utilizing the electronic device (1310) within a space containing the electronic device (1310) may be located within the specific space (1300). Additionally, two or more users may be located within a space not containing the electronic device (1310) within the specific space (1300).

[0262] In one embodiment of the present disclosure, a plurality of external electronic devices may be located in a specific space (1300). Each of the plurality of users located within the specific space (1300) may be wearing an external electronic device. However, the present disclosure is not limited thereto, and some of the plurality of users may not be wearing an external electronic device, while the remaining users may be wearing an external electronic device. Additionally, a single user may be wearing two or more external electronic devices on different parts of their body.

[0263] In one embodiment of the present disclosure, each of the plurality of external electronic devices may transmit to the surroundings, in a broadcast manner, a plurality of inertial information acquired according to the movement of a specific body area of ​​each of the plurality of users where the plurality of external electronic devices are located. In this case, each of the plurality of inertial information may include information capable of identifying the external electronic device, such as the device name or device identifier of each of the corresponding plurality of external electronic devices.

[0264] In one embodiment of the present disclosure, the electronic device (1310) can acquire a plurality of inertial information through a communication interface.

[0265] In one embodiment of the present disclosure, the electronic device (1310) may obtain a user image by photographing at least one user located within the space containing the electronic device (1310) among a plurality of users with a camera. Based on the user image, the electronic device (1310) may obtain at least one gesture information corresponding to at least one user located within the space containing the electronic device (1310).

[0266] In one embodiment of the present disclosure, an electronic device (1310) can compare at least one gesture information with a plurality of inertial information to obtain at least one inertial information corresponding to at least one gesture information among the plurality of inertial information. The electronic device (1310) can determine at least one inertial information corresponding to at least one gesture information as at least one mapping inertial information.

[0267] In one embodiment of the present disclosure, the electronic device (1310) may use an extended Kalman filter to obtain a Kalman gain for each of a plurality of inertial information for at least one gesture information, and determine at least one inertial information having the largest value among the obtained Kalman gains as mapping inertial information. At this time, if the electronic device (1310) obtains two or more gesture information corresponding to two or more users, the electronic device (1310) may obtain a Kalman gain for each of the plurality of inertial information for each gesture information, and determine one inertial information having the largest value among the obtained Kalman gains as mapping inertial information corresponding to each gesture information.

[0268] In one embodiment of the present disclosure, the electronic device (1310) can identify an external electronic device that has transmitted mapping inertia information among a plurality of external electronic devices through information such as a device name or device identifier included in the mapping inertia information. At this time, the external electronic device identified as having transmitted mapping inertia information may be an external electronic device worn by a user located within the space containing the electronic device (1310). The external electronic device identified as having transmitted mapping inertia information may be referred to as the identified external electronic device. The electronic device (1310) can identify a user wearing an external electronic device among at least one user located within the space containing the electronic device (1310) by comparing at least one gesture information with a plurality of inertia information.

[0269] In one embodiment of the present disclosure, the electronic device (1310) can obtain a gesture of a user wearing an identification external electronic device from mapping inertial information obtained from an identification external electronic device. In this case, the obtained user gesture may be a gesture using a specific body part wearing the identification external electronic device.

[0270] In one embodiment of the present disclosure, the electronic device (1310) can control the operation of the electronic device (1310) based on a user gesture obtained.

[0271] However, in one embodiment of the present disclosure, the user gesture obtained using mapping inertial information obtained from an external identification electronic device may include a drift error that occurs as the error between the user's movement and the measured inertial information accumulates over time.

[0272] In one embodiment of the present disclosure, the electronic device (1310) can correct a drift error included in a user gesture by using gesture information determined to correspond to mapping inertia information. The electronic device (1310) can correct the drift error by comparing the user gesture obtained from the gesture information with the user gesture obtained from the mapping inertia information. The electronic device (1310) can use the user gesture in which the drift error has been corrected by using the gesture information obtained after using the mapping inertia information.

[0273] Additionally, in one embodiment of the present disclosure, the electronic device (1310) can obtain a gesture of a user wearing an identified external electronic device from gesture information and mapping inertia information. At this time, the user's gesture obtained from the gesture information and mapping inertia information may be referred to as the final gesture. The electronic device (1310) can obtain the final gesture by fusing the gesture information and mapping inertia information. The electronic device (1310) can control the operation of the electronic device (1310) based on the final gesture obtained from the gesture information and mapping inertia information by using an extended Kalman filter.

[0274] FIG. 14 is a flowchart illustrating the operation of an electronic device that transmits a request signal requesting inertia information as the acquired gesture information includes a preset start gesture, according to an embodiment of the present disclosure. FIG. 15 is a diagram illustrating a preset start gesture that determines whether to perform the operation of the electronic device, according to an embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 3, and redundant descriptions are omitted.

[0275] Referring to FIGS. 1, FIGS. 2, FIGS. 3, and FIGS. 14, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S210) of determining whether at least one gesture information includes a preset start gesture.

[0276] In one embodiment of the present disclosure, step S210 may be performed after step S200.

[0277] Referring to FIG. 15, in one embodiment of the present disclosure, a “start gesture” may be a pre-set operation for the electronic device (100) to perform an operation of acquiring a gesture of a user using the electronic device (100) when the said gesture is recognized. The start gesture may be a pre-set gesture for the electronic device (100) to transmit a request signal requesting inertial information to the surroundings of the electronic device (100) through a communication interface (160) when the said gesture is recognized.

[0278] In one embodiment of the present disclosure, the starting gesture may include a first gesture (1500) in the form of a clenched fist, a second gesture (1510) in the form of opening the hand from the clenched fist and then clenching the fist again, a third gesture (1520) in the form of touching the tip of the thumb and the tip of the index finger together and extending the remaining fingers to indicate "OK," and a fourth gesture (1530) in the form of opening the hand from the form indicating "OK" and then indicating "OK" again. However, this is merely one embodiment, and the present disclosure is not limited thereto and may include other forms of gestures. Additionally, the starting gesture may include gestures using other body parts, such as the shape of the mouth, in addition to gestures using the hand.

[0279] In step S210, by having at least one processor (140) execute instructions or program code of memory (130), the electronic device (100) can determine whether at least one gesture information includes a preset start gesture. At this time, the electronic device (100) may also determine whether the start gesture is included in the gesture estimated from the user image through the gesture estimation module (132).

[0280] In one embodiment of the present disclosure, as at least one gesture information in step S210 includes a preset start gesture, the method of operation of the electronic device (100) may include the step (S220) of transmitting a request signal requesting inertial information through a communication interface (160).

[0281] In step S220, the electronic device (100) can transmit a request signal requesting inertial information through the communication interface (160).

[0282] In one embodiment of the present disclosure, step S300 may be performed after step S220. In one embodiment of the present disclosure, in step S220, an external electronic device (400) may acquire a request signal transmitted by the electronic device (100) via a broadcast method through a communication interface (160). As the external electronic device (400) transmits inertial information to the surroundings in response to the request signal, in step S300, the electronic device (100) may acquire inertial information via the communication interface (160).

[0283] In one embodiment of the present disclosure, as at least one gesture information in step S210 does not include a preset start gesture, the method of operation of the electronic device (100) may be terminated without transmitting a request signal. That is, the electronic device (100) may determine that there is no need to recognize or acquire a user's gesture and terminate the operation for acquiring a user's gesture.

[0284] However, the present disclosure is not limited thereto. The method of operation of the electronic device (100) may further include a step of determining whether the application running on the electronic device (100) is an application or program (e.g., a game, a search bar, a note, a drawing board) that is pre-configured to receive gesture input from a user using the electronic device (100), or whether the user uses a menu or function that allows specific input to be performed using a gesture during the operation of the application running on the electronic device (100).

[0285] The electronic device (100) may perform step S220 as it is determined that the electronic device (100) needs to obtain a gesture from the user in the above step.

[0286] FIG. 16 is a flowchart for explaining the operation of an external electronic device that transmits inertial information according to one embodiment of the present disclosure, wherein a preset start gesture is included in a gesture obtained based on inertial information.

[0287] Referring to FIGS. 1, 15 and 16, in one embodiment of the present disclosure, a method of operating an external electronic device (400) may include a step (S10) of obtaining inertial information from at least one user using the external electronic device (400) by using an inertial measurement sensor included in the external electronic device (400).

[0288] In step S10, the external electronic device (400) can obtain inertia information by measuring the change in inertia according to the movement of the body of the user wearing the external electronic device (400).

[0289] In one embodiment of the present disclosure, the method of operation of an external electronic device (400) may include the step of obtaining a gesture of a user wearing the external electronic device (400) based on inertial information obtained after step S10. In one embodiment of the present disclosure, the inertial information may include 6DoF information based on the movement of a body area wearing the external electronic device (400) or the movement of a surrounding area of ​​the body area. The external electronic device (400) may obtain a gesture of a user wearing the external electronic device (400) by comparing the inertial information with a previously stored reference inertial information.

[0290] At this time, "reference inertia information" is a plurality of inertia information obtained when a user performs each of a plurality of specific gestures, and the external electronic device (400) can determine that the user performs a specific gesture corresponding to an inertia information among a plurality of specific gestures as the obtained inertia information is determined to correspond to one of the plurality of inertia information included in the reference inertia information.

[0291] In one embodiment of the present disclosure, when an external electronic device (400) is worn on a user's left wrist, first inertia information measured when the user rotates the left wrist clockwise may be stored as reference inertia information. The external electronic device (400) may determine that the user performs a gesture of rotating the left wrist clockwise if the acquired inertia information corresponds to the first inertia information.

[0292] Additionally, when the user performs a specific gesture with the fingers of the left hand, the left wrist may also move along with the movement of the fingers due to the body structure, or vibration may be transmitted to the left wrist. Accordingly, the second inertial information measured when the user performs a specific gesture with the fingers of the left hand can also be stored as reference inertial information. The external electronic device (400) can determine that the user performs a specific gesture with the fingers of the left hand if the acquired inertial information corresponds to the second inertial information.

[0293] In one embodiment of the present disclosure, the method of operation of an external electronic device (400) may include a step (S20) of determining whether at least one user gesture obtained based on inertia information is included in a preset start gesture (1500, 1510, 1520, 1530).

[0294] In step S20, the external electronic device (400) can determine whether at least one acquired user gesture is included in a preset start gesture (1500, 1510, 1520, 1530).

[0295] In step S20, as it is determined that at least one user gesture is included in the start gesture, the method of operation of the external electronic device (400) may include the step (S30) of transmitting inertial information to the outside through a communication interface included in the external electronic device (400).

[0296] In step S30, the external electronic device (400) can transmit inertial information to the outside via a broadcast method through a communication interface. At this time, the external electronic device (400) can reduce the amount of power consumed in transmitting inertial information to the outside by setting the transmission cycle of the inertial information to the outside to gradually slow down. Accordingly, the usage time of the battery included in the external electronic device (400) can be increased.

[0297] After that, step S100, which is an operation of the electronic device (100) according to the present disclosure, may be performed. However, the present disclosure is not limited thereto, and it is obvious that step S300 according to the present disclosure may be performed after step S30.

[0298] In step S20, as it is determined that at least one user gesture is not included in the start gesture, the operation method of the external electronic device (400) can be terminated without transmitting inertia information to the outside.

[0299] FIG. 17 is a flowchart illustrating the operation of an electronic device for acquiring a final gesture based on a correction weight proportional to the reliability of a user image obtained from a pose estimation model, according to one embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to steps identical to those described in FIG. 3 and FIG. 9, and redundant descriptions are omitted.

[0300] Referring to FIGS. 2, FIGS. 3, FIGS. 9 and FIGS. 17, in one embodiment of the present disclosure, a method of operating an electronic device (100) may include a step (S230) of obtaining at least one gesture information from a user image using a pose estimation model.

[0301] In one embodiment of the present disclosure, step S230 may be performed after step S100. Step S230 may be included in step S200. The pose estimation model used in step S230 may be an artificial intelligence model (133) included in the gesture estimation module (132).

[0302] In step S230, by having at least one processor (140) execute instructions or program code of the gesture estimation module (132), the electronic device (100) can obtain at least one gesture information from a user image using a pose estimation model.

[0303] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (S240) of obtaining confidence in a user image from a pose estimation model.

[0304] In one embodiment of the present disclosure, the "confidence" obtained in step S240 may be a value in which the degree of confidence regarding the gesture information estimated from the user image by the pose estimation model used in step S230 is represented as a value between 0 and 1.

[0305] In one embodiment of the present disclosure, a pose estimation model may generate a probability density function for the coordinates where each feature point constituting a gesture is located from a user image. The pose estimation model may estimate the coordinate having the highest value in the probability density function as the location of the corresponding feature point. In this case, reliability may refer to the probability density at the corresponding coordinate. However, the present disclosure is not limited thereto, and reliability may include other information that allows the result of the pose estimation model to be trusted according to the algorithm of the pose estimation model used to acquire gesture information in step S230.

[0306] In one embodiment of the present disclosure, reliability may be high when the resolution of the user image obtained through the camera (120) is high, when the area of ​​the body region where gesture information included in the user image is obtained is large, or when the body region where gesture information included in the user image is obtained is not obscured by an obstacle or another body region.

[0307] However, if the resolution of the user image obtained through the camera (120) is low, if the area of ​​the body region where gesture information included in the user image is obtained is small, or if the body region where gesture information included in the user image is obtained is obscured by an obstacle or another body region, the reliability obtained may be low.

[0308] In step S240, the electronic device (100) can obtain confidence in the user image from the pose estimation model.

[0309] In one embodiment of the present disclosure, after step S240, the electronic device (100) may perform steps S300 and S400.

[0310] In one embodiment of the present disclosure, the method of operation of the electronic device (100) may include the step (520) of obtaining a final gesture by multiplying a correction weight proportional to the reliability using an extended Kalman filter by the Kalman gain of the mapping gesture information. In one embodiment of the present disclosure, step S520 may be performed after step S400. Step S520 may be included in step S500.

[0311] In one embodiment of the present disclosure, the “correction weight” may be a value multiplied by the Kalman gain of the mapping gesture information, which determines the degree to which the mapping gesture information is reflected when acquiring the user’s final gesture using an extended Kalman filter. The correction weight may have a value between 0 and 1. The closer the value of the correction weight is to 0, the lower the degree to which the mapping gesture information is reflected when acquiring the final gesture.

[0312] In one embodiment of the present disclosure, the value of the correction weight may increase as the reliability increases. The value of the correction weight may decrease as the reliability decreases. Accordingly, when the reliability of the gesture information obtained from the user image is low, the electronic device (100) can increase the accuracy of the final gesture by lowering the proportion in which the gesture information is reflected and increasing the proportion in which the inertial information is reflected when obtaining the final gesture.

[0313] However, the present disclosure is not limited thereto. The correction weight may be set to have a value of 1 when the value of the reliability is greater than a preset first threshold reliability value. When the value of the reliability is equal to or smaller than the first threshold reliability value, the correction weight may be set to have a value closer to 0 as the magnitude of the correction weight decreases.

[0314] Additionally, the correction weight may be set to have a value greater than 1 when the confidence value is greater than the preset second threshold confidence value. In this case, the accuracy of the final gesture can be improved by increasing the proportion of gesture information with high confidence reflected in acquiring the final gesture and decreasing the proportion of inertia information reflected. At this time, the second threshold confidence value may be greater than the first threshold confidence value.

[0315] FIG. 18 is a drawing for explaining the operation between an electronic device and an external electronic device according to one embodiment of the present disclosure. Hereinafter, the same reference numerals are assigned to configurations identical to those described in FIG. 2, and redundant descriptions are omitted.

[0316] Referring to FIGS. 2, FIGS. 13 and FIGS. 18, in one embodiment of the present disclosure, an electronic device (100) may include a camera (120), a memory (130), at least one processor (e.g., including a processing circuit, 140), an input / output interface (e.g., including an input / output circuit, 150), and a communication interface (e.g., including a communication circuit, 160).

[0317] In one embodiment of the present disclosure, the external electronic device (400) may include an inertial measurement sensor (410), a memory (420), at least one processor (e.g., including a processing circuit, 430) and a communication interface (e.g., including a communication circuit, 440).

[0318] However, not all of the components shown in FIG. 18 are essential components. The external electronic device (400) may be implemented with more components than those shown in FIG. 18, or with fewer components.

[0319] The inertial measurement sensor (410), memory (420), at least one processor (430), and communication interface (440) included in the external electronic device (400) can each be electrically connected to each other.

[0320] In one embodiment of the present disclosure, the inertial measurement sensor (410) may include an acceleration sensor and an angular velocity sensor (gyroscope sensor, e.g., a gyroscope). The inertial measurement sensor (410) may obtain 3DoF or 6DoF values ​​by measuring the change in inertia according to the movement of the external electronic device (400).

[0321] In one embodiment of the present disclosure, the memory (420) may store instructions, data structures, and program code that can be read by at least one processor (430). In one embodiment of the present disclosure, the memory (420) may be one or more. Operations performed by the external electronic device (400) may be implemented by at least one processor (430) executing the instructions or code of a program stored in the memory (420).

[0322] In one embodiment of the present disclosure, the memory (420) may not exist separately and may be configured to be included in at least one processor (430). In one embodiment of the present disclosure, the memory (420) may store various types of modules that can be used to perform the operation of an external electronic device (400), such as an inertial information acquisition module.

[0323] In one embodiment of the present disclosure, at least one processor (430) may be configured to control a series of processes to operate an external electronic device (400) and may be composed of one or more processors. In one embodiment of the present disclosure, at least one processor (430) may execute various types of modules stored in memory (420).

[0324] In one embodiment of the present disclosure, at least one processor (430) may be composed of at least one of a Central Processing Unit, a microprocessor, a Graphic Processing Unit, an Application Processor (AP), an Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), a Communication Processor (CP), a Neural Processing Unit, or an AI-dedicated processor designed with a hardware structure specialized for the learning and processing of an artificial intelligence model (AI), but is not limited thereto.

[0325] Accordingly, processor 430 may include various processing circuits and / or multiple processors. For example, the term “processor” as used herein, including in the claims, may include various processing circuits including at least one processor, and at least one of the at least one processor may be configured to perform various functions described herein individually and / or collectively in a distributed manner. When “one processor,” “at least one processor,” and “one or more processors” as used herein are described as being configured to perform multiple functions, these terms include, but are not limited to, situations where, for example, one processor performs part of the mentioned functions and other processor(s) perform other parts of the mentioned functions, and situations where a single processor can perform all the mentioned functions. Additionally, at least one processor may include a combination of processors performing the various mentioned / disclosed functions, for example, and may be performed in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0326] In one embodiment of the present disclosure, at least one processor (430) may be composed of a circuit such as a System on Chip (SoC) or an Integrated Circuit (IC).

[0327] In one embodiment of the present disclosure, the communication interface (440) can perform data communication with an external server or an external electronic device under the control of at least one processor (430).

[0328] The communication interface (440) can perform data communication with an external server or an external electronic device using at least one of the data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, Zigbee, WFD (Wi-Fi Direct), infrared communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet), WiMAX (World Interoperability for Microwave Access), SWAP (Shared Wireless Access Protocol), WiGig (Wireless Gigabit Alliance), and RF communication.

[0329] In one embodiment of the present disclosure, the communication interface (440) can provide acquired inertial information to an external server or an external electronic device under the control of at least one processor (430). The external electronic device (400) can provide the acquired inertial information to an external server or an external electronic device via a broadcast method through the communication interface (440).

[0330] In one embodiment of the present disclosure, the electronic device (100) can obtain user images by photographing a plurality of users using the electronic device (100) through a camera (120). The electronic device (100) can obtain a plurality of gesture information corresponding to each of the plurality of users from the user images by having at least one processor (140) execute instructions or program code of the memory (130).

[0331] In one embodiment of the present disclosure, the electronic device (100) can obtain inertial information provided by an external electronic device (400) through a communication interface (160).

[0332] In one embodiment of the present disclosure, an electronic device (100) can obtain mapping gesture information corresponding to inertial information among a plurality of gesture information by using an extended Kalman filter, by having at least one processor (140) execute instructions or program code of memory (130). At this time, each of the plurality of gesture information may include a plurality of sub-gesture information corresponding to a plurality of body regions, and the electronic device (100) can obtain mapping sub-gesture information corresponding to inertial information among the plurality of gesture information.

[0333] In one embodiment of the present disclosure, the electronic device (100) can obtain the user's final gesture from mapping gesture information or mapping sub-gesture information and inertia information using an extended Kalman filter by having at least one processor (140) execute instructions or program code of memory (130).

[0334] To solve the technical problem described above, an embodiment of the present disclosure provides an electronic device. The electronic device may include a camera. The electronic device may include a communication interface. The electronic device may include a memory in which a program or at least one instruction is stored. The electronic device may include at least one processor. By having at least one processor execute a program or at least one instruction stored in memory individually or collectively, the electronic device may capture at least one user using the electronic device through the camera and acquire a user image. The electronic device may estimate a gesture from the user image and acquire at least one gesture information corresponding to each of the at least one user. The electronic device may acquire inertial information from an external electronic device through a communication interface. Based on at least one gesture information and inertial information, the electronic device may acquire mapping gesture information corresponding to the inertial information among the at least one gesture information. Based on the mapping gesture information and inertial information, the electronic device may acquire a final gesture for controlling the electronic device of the user corresponding to the mapping gesture information among the at least one user.

[0335] In one embodiment of the present disclosure, an electronic device may obtain at least one Kalman gain of inertia information for each of at least one gesture information using an extended Kalman filter. The electronic device may obtain gesture information corresponding to the Kalman gain having the largest value among the at least one Kalman gain as mapping gesture information.

[0336] In one embodiment of the present disclosure, an electronic device may acquire at least one filtered Kalman gain that is greater than a preset reference value among at least one Kalman gain. The electronic device may acquire gesture information corresponding to the filtered Kalman gain having the largest value among at least one filtered Kalman gain as mapping gesture information.

[0337] In one embodiment of the present disclosure, an electronic device can obtain a final gesture from mapping gesture information and inertial information using an extended Kalman filter.

[0338] In one embodiment of the present disclosure, an electronic device may obtain at least one gesture information from a user image using a pose estimation model. The electronic device may obtain confidence in the user image from the pose estimation model. When using an extended Kalman filter, the electronic device may obtain a final gesture by multiplying a correction weight proportional to the confidence by the Kalman gain of the mapping gesture information.

[0339] In one embodiment of the present disclosure, an electronic device can acquire inertial information transmitted via broadcast from an external electronic device through a communication interface.

[0340] In one embodiment of the present disclosure, the period during which inertial information is acquired through a communication interface may become progressively longer. As the period becomes longer than a preset threshold period, the electronic device may transmit a request signal requesting inertial information through the communication interface.

[0341] In one embodiment of the present disclosure, at least one gesture information may include a plurality of sub-gesture informations, each corresponding to a plurality of body regions. Based on at least one gesture information and inertia information, the electronic device may obtain mapping sub-gesture information corresponding to a specific body region where an external electronic device is located, among the plurality of sub-gesture informations included in each of at least one gesture information. Based on the mapping sub-gesture information and inertia information, the electronic device may obtain a final gesture of a specific body region of at least one user corresponding to the mapping sub-gesture information.

[0342] In one embodiment of the present disclosure, an electronic device may select at least one sub-gesture information corresponding to a preset reference body region among a plurality of sub-gesture information included in each of at least one gesture information. Based on the selected at least one sub-gesture information and inertia information, the electronic device may obtain mapping sub-gesture information corresponding to a specific body region among the selected at least one sub-gesture information included in each of at least one gesture information.

[0343] In one embodiment of the present disclosure, the electronic device may transmit a request signal requesting inertial information through a communication interface, as at least one gesture information includes a preset start gesture.

[0344] In order to solve the technical problem described above, an operation method of an electronic device may be provided as an embodiment of the present disclosure. The operation method of an electronic device may include the step of obtaining a user image by photographing at least one user using the electronic device through a camera. The operation method of an electronic device may include the step of estimating a gesture from the user image and obtaining at least one gesture information corresponding to each of the at least one user. The operation method of an electronic device may include the step of obtaining inertial information from an external electronic device through a communication interface. The operation method of an electronic device may include the step of obtaining mapping gesture information corresponding to the inertial information among the at least one gesture information, based on the at least one gesture information and inertial information. The operation method of an electronic device may include the step of obtaining a final gesture for controlling the electronic device of the user corresponding to the mapping gesture information among the at least one user, based on the mapping gesture information and inertial information.

[0345] In one embodiment of the present disclosure, the step of acquiring mapping gesture information may include acquiring at least one Kalman gain of inertia information for each of at least one gesture information using an extended Kalman filter. The step of acquiring mapping gesture information may include acquiring gesture information corresponding to the Kalman gain having the largest value among the at least one Kalman gain as mapping gesture information.

[0346] In one embodiment of the present disclosure, the step of acquiring mapping gesture information may include acquiring at least one filtering Kalman gain that is greater than a preset reference value among at least one Kalman gain. The step of acquiring mapping gesture information may include acquiring gesture information corresponding to the filtering Kalman gain having the largest value among at least one filtering Kalman gain as mapping gesture information.

[0347] In one embodiment of the present disclosure, the step of obtaining a final gesture may include obtaining a final gesture from mapping gesture information and inertia information using an extended Kalman filter.

[0348] In one embodiment of the present disclosure, the step of obtaining at least one gesture information may include the step of obtaining at least one gesture information from a user image using a pose estimation model. The step of obtaining at least one gesture information may include the step of obtaining confidence for the user image from the pose estimation model. The step of obtaining a final gesture may include the step of obtaining a final gesture by multiplying a correction weight proportional to confidence by the Kalman gain of the mapping gesture information using an extended Kalman filter.

[0349] In one embodiment of the present disclosure, in the step of acquiring inertia information, inertia information transmitted via broadcast from an external electronic device is acquired through a communication interface, and the period during which the inertia information is acquired may gradually become longer. The method of operation of the electronic device may include the step of transmitting a request signal requesting inertia information through the communication interface as the period becomes longer than a preset threshold period.

[0350] In one embodiment of the present disclosure, at least one gesture information may include a plurality of sub-gesture informations, each corresponding to a plurality of body regions. A method of operation of an electronic device may include the step of obtaining a mapping sub-gesture information corresponding to a specific body region where an external electronic device is located, among a plurality of sub-gesture informations included in each of the at least one gesture information, based on at least one gesture information and inertia information. A method of operation of an electronic device may include the step of obtaining a final gesture of a specific body region of at least one user corresponding to the mapping sub-gesture information, based on the mapping sub-gesture information and inertia information.

[0351] In one embodiment of the present disclosure, a method of operating an electronic device may include the step of selecting at least one sub-gesture information corresponding to a preset reference body region among a plurality of sub-gesture information included in each of at least one gesture information. The step of acquiring mapping sub-gesture information may include the step of acquiring mapping sub-gesture information corresponding to a specific body region among at least one selected sub-gesture information included in each of at least one gesture information, based on the selected at least one sub-gesture information and inertia information.

[0352] In one embodiment of the present disclosure, a method of operating an electronic device may include the step of transmitting a request signal requesting inertial information through a communication interface, wherein at least one gesture information includes a preset start gesture.

[0353] In order to solve the aforementioned technical problem, a computer-readable recording medium may be provided having a program recorded thereon for performing at least one operation method of an electronic device operation method disclosed in the present disclosure on a computer.

[0354] A program executed by an electronic device described in this disclosure may be implemented by hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0355] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively.

[0356] Software can be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). Computer-readable recording media can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The recording medium can be read by a computer, stored in memory, and executed by a processor.

[0357] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory storage media' simply means that it is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, 'non-transitory storage media' may include a buffer in which data is stored temporarily.

[0358] In addition, the program according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0359] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically through a manufacturer of an electronic device or an electronic market (e.g., Samsung Galaxy Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer of the electronic device, a server of the electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0360] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or module are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

Claims

1. Camera (120); A communication interface (160) including a communication circuitry; Memory (130) where a program or at least one instruction is stored; and It includes at least one processor (140) including a processing circuitry, and By having the above-mentioned at least one processor (140) execute the above-mentioned program or the above-mentioned at least one instruction stored in the memory (130) individually or collectively, the electronic device (100) is, At least one user using the electronic device (100) is photographed through the camera (120) to obtain a user image, and Estimate a gesture from the above user image to obtain at least one gesture information corresponding to each of the at least one user, and Inertial information is obtained from an external electronic device through the above communication interface (160), and Based on the above at least one gesture information and the above inertia information, mapping gesture information corresponding to the above inertia information among the above at least one gesture information is obtained, and An electronic device (100) that obtains a final gesture for controlling the electronic device (100) of a user corresponding to the mapping gesture information among at least one user, based on the mapping gesture information and the inertia information.

2. In Paragraph 1, The above electronic device (100) is, Using an extended Kalman filter, at least one Kalman gain of the inertia information for each of the at least one gesture information is obtained, and An electronic device (100) that acquires gesture information corresponding to the Kalman gain having the largest value among the at least one Kalman gain as the mapping gesture information.

3. In Paragraph 2, The above electronic device (100) is, Among the above at least one Kalman gain, at least one filtered Kalman gain greater than a specified reference value is obtained, and An electronic device (100) that acquires gesture information corresponding to the filtering Kalman gain having the largest value among the above at least one filtering Kalman gain as the mapping gesture information.

4. In either Paragraph 2 or Paragraph 3, The above electronic device (100) is, An electronic device (100) that obtains the final gesture from the mapping gesture information and the inertia information using the extended Kalman filter.

5. In Paragraph 4, The above electronic device (100) is, Using a pose estimation model, at least one gesture information is obtained from the user image, and Confidence for the user image is obtained from the pose estimation model above, and An electronic device (100) that obtains the final gesture by multiplying the Kalman gain of the mapping gesture information by a correction weight proportional to the reliability when using the above-mentioned extended Kalman filter.

6. In any one of paragraphs 1 through 5, The above electronic device (100) is, An electronic device (100) that acquires the inertial information transmitted via broadcast from the above external electronic device through the communication interface (160).

7. In Paragraph 6, The period during which the inertia information is obtained through the communication interface (160) is extended at a specified rate, and The above electronic device (100) is, An electronic device (100) that transmits a request signal requesting inertia information through the communication interface (160) as the above period is longer than the specified threshold period.

8. In any one of paragraphs 1 through 7, The above at least one gesture information includes a plurality of sub-gesture information corresponding to a plurality of body regions, respectively, and The electronic device (100) obtains mapping sub-gesture information corresponding to a specific body area where the external electronic device is located among the plurality of sub-gesture information included in each of the at least one gesture information, based on the at least one gesture information and the inertia information. An electronic device (100) that acquires the final gesture of the specific body area of ​​the user corresponding to the mapping sub-gesture information among the at least one user, based on the mapping sub-gesture information and the inertia information.

9. In Paragraph 8, The above electronic device (100) is, Among the plurality of sub-gesture information included in each of the above at least one gesture information, at least one sub-gesture information corresponding to a designated reference body area is selected, and An electronic device (100) that acquires mapping sub-gesture information corresponding to a specific body area among the selected at least one sub-gesture information included in each of the at least one gesture information, based on the selected at least one sub-gesture information and the inertia information.

10. In any one of paragraphs 1 through 9, The above electronic device (100) is, An electronic device (100) that transmits a request signal requesting the inertial information through the communication interface (160), wherein the above at least one gesture information includes a designated start gesture.

11. In the method of operating the electronic device (100), A step of obtaining a user image by photographing at least one user using the electronic device through a camera (S100); A step (S200) of estimating a gesture from the above user image and obtaining at least one gesture information corresponding to each of the at least one user; A step (S300) of obtaining inertial information from an external electronic device through a communication interface; Based on the at least one gesture information and the inertia information, a step (S400) of obtaining mapping gesture information corresponding to the inertia information among the at least one gesture information; and A method of operation of an electronic device (100) comprising the step (S500) of obtaining a final gesture for controlling the electronic device of a user corresponding to the mapping gesture information among at least one user, based on the mapping gesture information and the inertia information.

12. In Paragraph 11, The step (S400) of obtaining the above mapping gesture information is, A step of obtaining at least one Kalman gain of the inertia information for each of the at least one gesture information using an extended Kalman filter; and A method of operation of an electronic device (100) comprising the step of acquiring gesture information corresponding to the Kalman gain having the largest value among the at least one Kalman gain as the mapping gesture information.

13. In Paragraph 12, The step (S400) of obtaining the above mapping gesture information is, Among the above at least one Kalman gain, a step of obtaining at least one filtered Kalman gain greater than a specified reference value; and A method of operation of an electronic device (100) further comprising the step of acquiring gesture information corresponding to the filtering Kalman gain having the largest value among the at least one filtering Kalman gain as the mapping gesture information.

14. In either Paragraph 12 or Paragraph 13, The step of obtaining the above final gesture (S500) is, A method of operation of an electronic device (100) comprising the step of obtaining the final gesture from the mapping gesture information and the inertia information using the extended Kalman filter.

15. A computer-readable recording medium having a program recorded thereon for performing the method of operation described in any one of claims 11 through 14 on a computer.

Images