IMU- AND EMG-BASED AUGMENTED REALITY INPUT DEVICE AND METHOD

Abstract

An IMU- and EMG-based augmented reality input device is designed to be connected to a wristband worn on a user's arm, equipped with an inertial measurement unit (IMU) sensor and an electromyography (EMG) sensor, and an augmented reality module worn on the user's face, providing a virtual augmented reality over a network and configured to position a virtual cursor on the augmented reality, with the virtual cursor moving according to changes in angle and position of the user's arm as detected by an IMU sensor, and providing input corresponding to a gesture of the user's arm as detected by an EMG sensor.

Description

TECHNICAL FIELDThe present disclosure relates to an inertial measurement unit (IMU) and electromyography (EMG)-based augmented reality input device and a method. More particularly, the present disclosure relates to an IMU and EMG-based augmented reality input device and method that concurrently uses IMU signals and EMG signals.BACKGROUND OF THE INVENTIONAugmented reality is a generic term that includes virtual reality (VR), augmented reality (AR), and mixed reality (MR). In augmented reality, the input is performed using a controller.Some VR and AR controllers may mainly use controllers that require the use of hands. However, controllers requiring the use of hands create a separation from the environment and impede natural interaction environments. To solve these problems, gesture input technology using cameras mounted on AR / VR devices may be used.However, the camera-based gesture input technology requires multiple camera sensors positioned in different directions to cover different positions of the hand, and has a disadvantage that the recognition rate is reduced depending on the environment (light, camera angle, or the like).Moreover, some EMG-based interfaces may have the disadvantage that they can only perform registered gestures and pre-mapped commands.SUMMARYThe present disclosure seeks to provide an IMU- and EMG-based augmented reality input device and a method capable of using an IMU-based efficient 3D pointing method and simultaneously using an EMG-based command transfer function using artificial intelligence, thereby providing an augmented usage environment as the environment in which only one of them is used.An apparatus may include: a first wearable device including an inertial measurement unit (IMU) sensor and an electromyography (EMG) sensor, wherein the first wearable device is configured to detect at least one IMU signal and at least one EMG signal based on movement of a body part of a user, and wherein the body part includes a forearm of the user and a hand of the user; and a second wearable device configured to: receive, from the first wearable device via a communication interface, the at least one IMU signal and the at least one EMG signal; displaying a virtual cursor on an augmented reality display, wherein the virtual cursor is configured to move based on the at least one IMU signal, and wherein the at least one IMU signal corresponds to a change in orientation of the forearm and a change in position of the forearm; and determining, based on the at least one EMG signal, an input corresponding to a gesture of the hand of the user.The virtual cursor may be implemented as a virtual beam associated with an orientation of the forearm and a direction of the forearm, and the first wearable device may include a bracelet.An object on a virtual space in the augmented reality display may be identified by a ray casting technique using the virtual ray.The device may further include: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the device to: map position information between the IMU sensor and the virtual cursor; select an object on a virtual space associated with the augmented reality display using an EMG-based gesture input; and process a command associated with a target object recognized by the virtual cursor on the augmented reality display.The instructions, when executed by the at least one processor, may cause the device to map three-dimensional coordinates relating to a position and orientation of the forearm to the virtual cursor of a virtual beam shape.The second wearable device may be configured to provide a plurality of different inputs based on a plurality of commands corresponding to a plurality of gestures, each of the plurality of different inputs corresponding to a respective one of the plurality of commands, and wherein at least one of the plurality of gestures is determined based on the at least one EMG signal.The plurality of commands may include a selection command corresponding to a first gesture of the plurality of gestures, an activation command corresponding to a second gesture of the plurality of gestures, a third command corresponding to a third gesture of the plurality of gestures, a menu command corresponding to a fourth gesture of the plurality of gestures, and a speech recognition command corresponding to a fifth gesture of the plurality of gestures.The second wearable device may be configured to identify different gestures among the plurality of gestures via EMG signal processing associated with an artificial intelligence model.The second wearable device may be configured to transmit different commands related to a same gesture or transmit a same command related to different gestures based on a type of an object displayed by the virtual cursor.The second wearable device may be configured to display a plurality of outputs corresponding to a plurality of inputs on a virtual space associated with the augmented reality display.A method performed by an apparatus may include: receiving at least one inertial measurement unit (IMU) signal and at least one electromyography (EMG) signal, wherein the at least one IMU signal is generated based on an inertial measurement unit (IMU) sensor, and wherein the at least one EMG signal is generated based on an EMG sensor; determining, based on the at least one IMU signal, a position of a forearm of a user and an orientation of the forearm; mapping the determined position and orientation of the forearm to position information of a cursor displayed on a virtual space in a display; identifying, based on the at least one EMG signal, a gesture of a hand of the user; and determining, based on the identified gesture, an input associated with an object on the virtual space, the object corresponding to the cursor.generating a plurality of objects and the cursor on the virtual space, wherein generating the cursor comprises implementing, as the cursor, a virtual ray associated with an orientation of the forearm and a direction of the forearm.The method may further include identifying the object on the virtual space by a ray casting technique using the virtual ray.The method may further include processing a command associated with a target object identified by the cursor on the augmented reality implementing display.The mapping may include: determining three-dimensional coordinates with respect to the forearm; and mapping the three-dimensional coordinates to the cursor implemented as the virtual beam.The method may further include determining a plurality of different inputs based on a plurality of commands corresponding to a plurality of gestures, each of the plurality of different inputs corresponding to a respective one of the plurality of commands, and wherein at least one of the plurality of gestures is determined based on the at least one EMG signal.The plurality of commands may include a selection command corresponding to a first gesture of the plurality of gestures, an activation command corresponding to a second gesture of the plurality of gestures, a third command corresponding to a third gesture of the plurality of gestures, a menu command corresponding to a fourth gesture of the plurality of gestures, and a speech recognition command corresponding to a fifth gesture of the plurality of gestures.The method may further include identifying different gestures among the plurality of gestures via EMG signal processing associated with an artificial intelligence model.The method may further include transmitting different commands related to a same gesture or transmitting a same command related to different gestures based on a type of object displayed by the cursor.The method may further include displaying a plurality of outputs corresponding to a plurality of inputs on the virtual space.An IMU and EMG-based augmented reality input device and method according to an example may provide full hands-free input that is not limited to a measurement environment and is not limited to physical controllers.An IMU and EMG-based augmented reality input device and method according to an example may provide high interface extensibility by using an IMU-based ray casting technique and combining EMG-based various gesture input functions.BRIEF DESCRIPTION OF THE DRAWINGSFIG. 1 shows an example of an IMU and EMG-based augmented reality input system according to an example. FIG. 2 shows an example of a block diagram of an IMU and EMG-based augmented reality input device according to an example. FIG. 3 shows an example of a block diagram of an IMU and EMG based augmented reality input device according to an example. FIG. 4 shows an example flowchart of an IMU and EMG based augmented reality input method according to an example. FIG. 5 shows an example of a signal flow diagram for explaining an IMU- and EMG-based augmented reality input method according to an example. FIG. 6 shows an example of an input screen by an IMU and EMG-based augmented reality input method according to an example. FIG. 7 shows an example of a drawing for explaining a computing device according to an example.DETAILED DESCRIPTIONAn example of the disclosure will be described in more detail below with reference to the accompanying drawings, so that a person skilled in the art can easily practice the example. As those skilled in the art would recognize, the described examples may be modified in various ways without departing from the spirit or scope of the present disclosure. In order to clarify the present disclosure, parts unrelated to the description are omitted, and the same elements or equivalents are referred to with the same reference numerals throughout the specification.The term "and / or" is used to include all instances of any combination of multiple elements that are subject matter. For example, "A and / or B" includes all three cases: "A", "B", and "A and B". Using the exemplary term "at least one of: A; B; or C" or "at least one of A, B, or C", the term "at least one A, or at least one B, or at least one C, or any combination of at least one A, at least one B, and at least one C. Further, exemplary terms such as "A, B, and C", "A, B, or C", "at least one of A, B, and C", "at least one of A, B, or C", etc., as used herein, may mean any listed element or all possible combinations of the listed elements. For example, "at least one of A or B" may refer to (1) at least one A; (2) at least one B; or (3) at least one A and at least one B.Moreover, unless expressly stated otherwise, the word "comprise" and variations such as "comprises" or "comprising" will be understood to imply the inclusion of stated elements but not the exclusion of other elements. Terms including an ordinary number such as first and second are used to describe various constituent elements, but the constituent elements are not limited by the terms. The terms are used only to distinguish one component from other components.Moreover, the terms "unit", "part" or "portion", "-er" and "module" in the specification refer to a unit that processes at least one function or operation that can be implemented by hardware, software, or a combination of hardware and software. Throughout the present disclosure, references to components, units, or modules generally refer to elements that can be logically grouped together to perform a function or group of associated functions. Like reference numerals are generally intended to refer to the same or similar components. Components, units, and modules may be implemented in software, hardware, or a combination of software and hardware. The above-described components, units, modules, and / or functions may be implemented and / or performed by one or more processors. For example, the components, units, and / or modules may include processor(s), microprocessor(s), graphics processing unit(s), logic circuitry(s), dedicated circuitry(s), application specific integrated circuit(s), programmable array logic, field programmable gate array(s), controller(s), microcontroller(s), and / or other suitable hardware. The components, units, and / or modules may also include software control module(s) implemented with, for example, a processor or logic circuitry. The components, units, and / or modules may include or otherwise be capable of accessing memory, such as one or more non-transitory computer readable storage media such as random access memory, read-only memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, flash / other memory device(s), data registration(s), database(s), and / or other suitable hardware. One or more storage media may include any or all of the physical memory of computers, processors, or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives, and the like, which may provide nonvolatile memory for software programming at any time.Hereinafter, examples of the present disclosure will be described with reference to the drawings.FIG. 1 shows an example of an IMU and EMG-based augmented reality input system according to an example.Referring to FIG. 1, the IMU and EMG-based augmented reality input system may include an augmented reality module 10, a bracelet 20, and an IMU and EMG-based augmented reality input device 100.Augmented reality (XR) may include virtual reality (VR), augmented reality (AR), and / or mixed reality (MR).The augmented reality module 10 may include a virtual reality module, an augmented reality module, and / or a mixed reality module. The augmented reality module 10 may be one of the virtual reality module, the augmented reality module, or the mixed reality module.Augmented reality module 10 may include hardware and software components that enable implementation and / or augmented reality experience.The augmented reality module 10 may provide a virtual space to a user. The virtual space may be a virtual space that enables virtual reality, augmented reality, or mixed reality experience.The augmented reality module 10 may include a head mount display (HMD), a computer, and / or a controller.The head mount display may be used to provide virtual reality, augmented reality, or mixed reality experience, and may include a high resolution display, a processing unit, and / or various sensors.The controller may be an input device that helps the user interact with the virtual environment. The controller may be implemented as the bracelet 20 and the IMU and EMG-based augmented reality input device 100.The augmented reality module 10 may be connected to the bracelet 20 and the IMU- and EMG-based augmented reality input device 100 via a network.The bracelet 20 may be worn on an arm of the user and may include an inertial measurement unit (IMU) sensor and an electromyography (EMG) sensor.For example, an IMU sensor may include a device that may measure and / or indicate a body specific force, angular rate, and / or magnetic field using a combination of accelerometers, gyroscopes, and / or magnetometers. The IMU sensor can track the movement and orientation of an object in 3D space and provide data about acceleration, rotation, and sometimes direction. IMUs may be useful in applications that require precise motion tracking and stability, such as smart phones, drones, virtual reality systems and / or autonomous vehicles, etc. By incorporating this motion data, IMUs may allow devices to navigate, stabilize, and interact with their environment more effectively.For example, an EMG sensor may be a device used to sense and / or measure the electrical activity produced by muscles during contraction. When a muscle is activated, it generates small electrical signals that can be detected by EMG sensors, which can typically be placed on the surface of the skin above the muscle or, for example, introduced directly into the muscle tissue.By translating muscle activity into electrical signals, EMG sensors can provide a powerful tool for analyzing and / or using human motion in various applications.The bracelet 20 may calculate an angle and / or position of the user's arm by the IMU sensor. The bracelet 20 can recognize various gestures of the user's arm by the EMG sensor. The bracelet 20 can recognize a gesture of the user based on changes in electromyography due to contraction of muscles of the user's arm.The gesture may include various types of movements of the user's arm or hand. For example, the gesture may include the user making a thumb, bending a particular finger, or the like.As described herein, the IMU and EMG-based augmented reality input device 100 may be referred to herein as the augmented reality input device 100.The augmented reality input device 100 may be connected to the augmented reality module 10 and the bracelet 20 via a network.The augmented reality input device 100 may receive sensor signals from the bracelet 20 and provide input to the virtual space provided by the augmented reality module 10.The input may include controlling augmented reality and selecting an object on a virtual space or the like.The augmented reality input device 100 may use a ray casting interaction technique using an IMU signal and simultaneously use an EMG signal-based gesture input technology to provide various types of input.For example, a ray casting interaction technique may include a method used in computer graphics and 3D environments to determine the location and interaction of objects by projecting a virtual beam from a specific point, such as a camera or a user's perspective, in a particular direction. This technique may involve transmitting a straight line or beam into the scene to detect whether it intersects with any objects. When a beam collides with an object, the system may calculate details such as the distance, position, and surface characteristics at the collision point.The ray casting interaction technique may support interaction between a user and a virtual object in a virtual reality (VR) and augmented reality (AR) environment. The ray casting interaction may be used when the user points to or selects a specific point in the virtual space, and may allow interaction with the object by projecting a virtual beam in a direction indicated by the controller or the user's hand.As described herein, the virtual beam may be projected parallel to the arm of the user to which the IMU sensor is attached as a cursor for input to the virtual space.The augmented reality input device 100 may arrange the cursor moving according to a change in the detected angle and the position of the user's arm on the virtual space by the IMU sensor of the bracelet 20.For example, the cursor may be implemented as the virtual beam parallel to an angle and a direction of the user's arm to which the bracelet 20 is attached.The augmented reality input device 100 may recognize an object on the virtual space by a ray casting technique using the virtual ray.The augmented reality input device 100 may provide various forms of input corresponding to a gesture of an arm of the user detected by the EMG sensor of the bracelet 20.For example, the augmented reality input device 100 may provide an input for clicking on a target object recognized by the cursor according to the gesture of the user's arm.FIG. 2 shows an example of a block diagram of the IMU and EMG-based augmented reality input device according to an example.Referring to FIG. 2, the augmented reality input device 100 may include a cursor mapping unit 110, a gesture recognition unit 120, and an augmented reality processing unit 130.The cursor mapping unit 110 may map position information between the IMU sensor and the cursor.The cursor imaging unit 110 may map 3-dimensional coordinates with respect to a position and an angle of the arm received by the IMU sensor to the virtual beam shape cursor.The gesture recognition unit 120 may select the object on the virtual space using an EMG-based gesture input.The gesture recognition unit 120 may provide a plurality of different inputs through a plurality of commands each corresponding to a plurality of gestures.The plurality of commands may include a selection command corresponding to a first gesture, an activation command corresponding to a second gesture, a reset command corresponding to a third gesture, a start command corresponding to the third gesture, a menu command corresponding to a fourth gesture, and / or a voice recognition command corresponding to a fifth gesture.The gesture recognition unit 120 may distinguish and recognize different gestures among the plurality of gestures through EMG signal processing using artificial intelligence.The gesture recognition unit 120 may transmit different commands related to the same gesture or transmit the same command related to different gestures depending on the type of the object displayed by the cursor.The augmented reality processing unit 130 may reflect the target object recognized by the cursor to the augmented reality.The augmented reality processing unit 130 may display a plurality of outputs corresponding to a plurality of inputs and the plurality of inputs on the virtual space.FIG. 3 shows an example of a block diagram of the IMU and EMG-based augmented reality input device according to an example.FIG. 3 shows the bracelet 20, the augmented reality module 10, and the augmented reality input device 100 included in the augmented reality module 10.The augmented reality input device 100 may be implemented to be included in the augmented reality module 10 as the controller of the augmented reality module 10.In FIG. 3, the bracelet 20 may receive a preprocessed IMU signal relating to the position and angle of the user's arm through the IMU sensor 21. The preprocessed IMU signal may appear as 3-dimensional coordinates of XYZ.The bracelet 20 may receive the EMG signal generated according to the gesture of the user's arm by the EMG sensor 22.The IMU signal and the EMG signal may be transmitted to the augmented reality input device 100 through a TCP protocol. The augmented reality input device 100 may be included in a head mount display or a computer server of the augmented reality module 10.The cursor mapping unit 110 may receive the preprocessed IMU signal of 3-dimensional coordinates from the IMU sensor 21 and map it to the cursor.The gesture recognition unit 120 may receive and pre-process the EMG signal from the EMG sensor 22.The gesture recognition unit 120 may classify the preprocessed EMG signal and generate a plurality of different commands. The gesture recognition unit 120 may classify the plurality of commands based on the EMG signals generated for respective recognized gestures.For example, the commands may include a click, an activation, a reset, or the like.The gesture recognition unit 120 may transmit the classified commands to the augmented reality processing unit 130.The augmented reality processing unit 130 may reflect an input to the augmented reality due to a command received from the gesture recognition unit 120.The augmented reality processing unit 130 may activate the cursor implemented as the virtual ray on the virtual space.The augmented reality processing unit 130 may reset the cursor to compensate for the position of the cursor offset due to the accumulated error of the IMU signal.The augmented reality processing unit 130 may select the virtual object on the virtual space. The virtual object may appear as an interface. The augmented reality processing unit 130 may perform a click command with respect to a virtual interface icon implemented on the virtual space.FIG. 4 shows an example flowchart of an IMU and EMG based augmented reality input method according to an example. The IMU and EMG-based augmented reality input method of FIG. 4 may be performed by the augmented reality input device 100 of FIG. 2 or 3.In FIG. 4, at step S 100, the augmented reality module 10 (see FIG. 1 ) may provide the virtual space. The augmented reality module 10 (see FIG. 1 ) may generate a plurality of objects in the virtual space.The augmented reality input device 100 may generate the plurality of objects and the cursor on the virtual space.The augmented reality input device 100 may implement the virtual beam parallel to the angle and the direction of the user's arm as the cursor through the IMU sensor.The augmented reality input device 100 may recognize the object on the virtual space by a ray casting technique using the virtual ray.At step S 200, the augmented reality input device 100 may calculate the position and the angle of the user's arm through the IMU sensor.At step S 300, the augmented reality input device 100 may map the calculated position and angle of the user's arm and position information of the cursor disposed on the virtual space.The augmented reality input device 100 may calculate 3-dimensional coordinates with respect to the arm by the IMU sensor, and may map the calculated 3-dimensional coordinates to the cursor implemented as the virtual beam.At step S 400, the augmented reality input device 100 may recognize the gesture of the user's arm by the EMG sensor.At step S 500, the augmented reality input device 100 may provide input regarding the object on the virtual space based on the recognized gesture.The augmented reality input device 100 may provide the plurality of different inputs through the plurality of commands corresponding to the plurality of gestures, respectively.The plurality of commands may include the selection command corresponding to the first gesture, the activation command corresponding to the second gesture, the reset command corresponding to the third gesture, the start command corresponding to the third gesture, the menu command corresponding to the fourth gesture, and the voice recognition command corresponding to the fifth gesture.The augmented reality input device 100 may distinguish and recognize different gestures among the plurality of gestures through EMG signal processing using artificial intelligence.The augmented reality input device 100 may learn a plurality of gestures and commands corresponding to the EMG signal by machine learning or deep learning, and may generate an artificial intelligence model for distinguishing different gestures.The augmented reality input device 100 may provide various types of inputs without a physical key of the artificial intelligence model for distinguishing different gestures.The artificial intelligence model may perform learning based on the IMU signal and the EMG signal and commands and transmission result data transmitted accordingly. According to the learning, the artificial intelligence model may distinguish and recognize the IMU signal and the EMG signal-based gesture and transmit an appropriate command.For example, the augmented reality input device 100 may transmit different commands related to the same gesture or transmit the same command related to different gestures, depending on the type of the object displayed by the cursor.The augmented reality input device 100 may transmit a first command when the cursor has recognized a first object at the time when the first gesture is input, but when the cursor has recognized a second object at the time when the same first gesture is input, may transmit a second command.Additionally or alternatively, when the cursor has recognized the second object, the augmented reality input device 100 may transmit the same third command even if any of the first gesture is input to the third gesture.The augmented reality input device 100 may reflect the target object recognized by the cursor to the augmented reality. The target object recognized by the cursor may be represented as being recognized in augmented reality.The augmented reality input device 100 may display the plurality of outputs corresponding to the plurality of inputs on the virtual space.When the reset command is input, the augmented reality input device 100 may reset the cursor. When the home command is input, the augmented reality input device 100 may display an initial screen of the augmented reality menu.When the voice recognition command is input due to a specific gesture, the augmented reality input device 100 may be connected to an augmented reality voice recognition system.When the menu command is input, the augmented reality input device 100 may display a menu screen for providing an interface to various functions.FIG. 5 shows an example of a signal flow diagram for explaining an IMU- and EMG-based augmented reality input method according to an example. FIG. 5 shows a signal flow in the IMU and EMG-based augmented reality input method of FIG. 4.In FIG. 5, at step S 100, the augmented reality module 10 may generate the virtual space and provide the virtual space that provides the input of the augmented reality input device 100.At step S 200, the IMU sensor 21 may calculate the position and angle of the user's arm and provide them to the augmented reality input device 100 as 3-dimensional coordinates.At step S 300, the augmented reality input device 100 may map the received 3-dimensional coordinates to a virtual cursor displayed by the augmented reality module 10.At step S 400, the EMG sensor 22 may recognize the gesture of the user's arm and provide the EMG signal to the augmented reality input device 100.At step S 500, the augmented reality input device 100 may classify or analyze the EMG signal using artificial intelligence and may transmit a command for input to the augmented reality module 10.The augmented reality module 10 may display the plurality of outputs corresponding to the plurality of inputs on the virtual space.FIG. 6 illustrates an example input screen through an IMU and EMG-based augmented reality input method according to an example.FIG. 6 shows an example user and an example arm, ARM, of the user wearing an augmented reality module AR and the bracelet 20.In FIG. 6, the user may view a screen where augmented reality is implemented by the augmented reality module AR. A virtual home interface HI including the plurality of objects may be provided on the augmented reality screen.The bracelet 20 can be worn on the arm ARM of the user. The arm ARM of the user and a cursor RAY in the form of a virtual ray of the augmented reality screen appear parallel to each other.The user can move the arm ARM to move the cursor RAY, which is the input means. The cursor RAY may move parallel to the arm of the user according to the IMU signal provided by the bracelet 20.An IMU and EMG-based augmented reality input device is provided to be connected to a bracelet worn on an arm of a user and provided with an inertial measurement unit (IMU) sensor and an electromyography (EMG) sensor and an augmented reality module worn on a face of the user and providing a virtual augmented reality via a network, and configured to arrange a virtual cursor on the augmented reality, wherein the virtual cursor moves according to a change in an angle and a change in position of the arm of the user detected by an IMU sensor, and provides an input corresponding to a gesture of the arm of the user detected by an EMG sensor.The cursor may be implemented as a virtual beam parallel to an angle and a direction of the user's arm equipped with the bracelet.An object on a virtual space can be recognized by a ray casting technique using the virtual ray.The IMU and EMG-based augmented reality input device may include a cursor mapping unit configured to map position information between the IMU sensor and the cursor, a gesture recognition unit configured to select an object on a virtual space using an EMG-based gesture input, and an augmented reality processing unit configured to reflect a target object recognized by the cursor into the augmented reality.The cursor imaging unit may be configured to map 3-dimensional coordinates related to a position and an angle of the arm received by the IMU sensor to the virtual beam shape cursor.The gesture recognition unit may be configured to provide a plurality of different inputs by a plurality of commands each corresponding to a plurality of gestures.The plurality of commands may include a selection command corresponding to a first gesture, an activation command corresponding to a second gesture, a reset command corresponding to a third gesture, a start command corresponding to the third gesture, a menu command corresponding to a fourth gesture, and a voice recognition command corresponding to a fifth gesture.The gesture recognition unit may be configured to distinguish and recognize different gestures among the plurality of gestures through EMG signal processing using artificial intelligence.The gesture recognition unit may be configured to transmit different commands related to the same gesture or transmit the same command related to different gestures, depending on the type of the object displayed by the cursor.The augmented reality processing unit may be configured to display a plurality of outputs corresponding to a plurality of inputs on the virtual space.An IMU and EMG-based augmented reality input method may include computing, by an IMU sensor, a position and an angle of an arm of a user, mapping the computed position and the angle of the arm of the user and position information of a cursor disposed on a virtual space, recognizing, by an EMG sensor, a gesture of the arm of the user, and providing an input regarding an object on the virtual space based on the detected gesture.The IMU and EMG-based augmented reality input method may further include generating a plurality of objects and the cursor on the virtual space, and generating the cursor may include implementing a virtual ray moving parallel to an angle and a direction of the user's arm as the cursor by using the IMU sensor.The IMU and EMG-based augmented reality input method may further include recognizing the object on the virtual space by a ray casting technique using the cursor.The IMU and EMG-based augmented reality input method may further include reflecting a target object recognized by the cursor into the augmented reality.Mapping the position information of the cursor may include computing, by the IMU sensor, 3-dimensional coordinates with respect to the arm and mapping the computed 3-dimensional coordinates to the cursor implemented as the virtual beam.Providing the input related to the object on the virtual space based on the gesture may include providing a plurality of different inputs by a plurality of commands each corresponding to a plurality of gestures.The plurality of commands may include a selection command corresponding to a first gesture, an activation command corresponding to a second gesture, a reset command corresponding to a third gesture, a start command corresponding to the third gesture, a menu command corresponding to a fourth gesture, and a voice recognition command corresponding to a fifth gesture.Detecting the gesture of the user's arm may include distinguishing and recognizing different gestures among the plurality of gestures through EMG signal processing using artificial intelligence.Detecting the gesture of the user's arm may further include transmitting different commands related to the same gesture or transmitting the same command related to different gestures, depending on the type of the object displayed by the cursor.The IMU and EMG-based augmented reality input method may further include displaying a plurality of outputs corresponding to a plurality of inputs on the virtual space.FIG. 7 is an exemplary drawing for explaining a computing device according to an example.Referring to FIG. 7, the IMU and EMG-based augmented reality input device and method according to examples may be implemented using a computing device 900.The computing device 900 may include at least one of a processor 910, a memory 930, the user interface input device 940, the user interface output device 950, and a storage device 960, which communicate via a bus 920. Computing device 900 may also include a network interface 970 electrically connected to a network 90. Network interface 970 may transmit or receive signals with other entities over network 90.The processor 910 may be implemented in various types, such as a microcontroller unit (MCU), an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and the like, and may be any type of semiconductor device capable of executing instructions stored in the memory 930 or the storage device 960. The processor 910 may be configured to implement the functions and methods described above with respect to FIGS. 1-6.The memory 930 and the storage device 960 may include various types of volatile or non-volatile storage media. For example, the memory may include read only memory (ROM) 931 and random access memory (RAM) 932. In this example, memory 930 may be internal or external to processor 910, and memory 930 may be connected to processor 910 by various known means.For example, at least some configurations or functions of the IMU and EMG-based augmented reality input device and method according to an example may be implemented as a program or software executable by the computing device 900, and the program or software may be stored in a computer readable medium.For example, at least some components or functions of the IMU and EMG-based augmented reality input device and method according to the examples may be implemented using hardware or circuitry of the computing device 900 or may be implemented using a separate device that may be electrically connected to the computing device 900.While this disclosure has been described in connection with what is presently considered to be practical examples, it is to be understood that the disclosure is not limited to the disclosed examples, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

An apparatus comprising: a first wearable device comprising an inertial measurement unit (IMU) sensor and an electromyography (EMG) sensor, wherein the first wearable device is configured to detect at least one IMU signal and at least one EMG signal based on movement of a body part of a user, and wherein the body part comprises a forearm of the user and a hand of the user; and a second wearable device configured to: receive, from the first wearable device via a communication interface, the at least one IMU signal and the at least one EMG signal; displaying a virtual cursor on an augmented reality display, wherein the virtual cursor is configured to move based on the at least one IMU signal, and wherein the at least one IMU signal corresponds to a change in orientation of the forearm and a change in position of the forearm; and determining, based on the at least one EMG signal, an input corresponding to a gesture of the hand of the user.The apparatus of claim 1, wherein the virtual cursor is implemented as a virtual beam associated with an orientation of the forearm and a direction of the forearm, and wherein the first wearable device comprises a bracelet.The apparatus of claim 2, wherein an object on a virtual space is identified in the augmented reality display by a ray casting technique using the virtual ray.The apparatus of claim 1, further comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the apparatus to: map position information between the IMU sensor and the virtual cursor; select an object on a virtual space associated with the augmented reality display using an EMG-based gesture input; and process a command associated with a target object recognized by the virtual cursor on the augmented reality display.The apparatus of claim 4, wherein the instructions, when executed by the at least one processor, cause the apparatus to map three-dimensional coordinates with respect to a position and orientation of the forearm to the virtual cursor of a virtual beam shape.The apparatus of claim 1, wherein the second wearable device is configured to provide a plurality of different inputs based on a plurality of commands corresponding to a plurality of gestures, wherein each of the plurality of different inputs corresponds to a respective one of the plurality of commands, and wherein at least one of the plurality of gestures is determined based on the at least one EMG signal.The apparatus of claim 6, wherein the plurality of commands comprises a selection command corresponding to a first gesture of the plurality of gestures, an activation command corresponding to a second gesture of the plurality of gestures, a third command corresponding to a third gesture of the plurality of gestures, a menu command corresponding to a fourth gesture of the plurality of gestures, and a speech recognition command corresponding to a fifth gesture of the plurality of gestures.The apparatus of claim 6, wherein the second wearable device is configured to identify different gestures among the plurality of gestures via EMG signal processing associated with an artificial intelligence model.The apparatus of claim 1, wherein the second wearable device is configured to transmit different commands related to a same gesture or transmit a same command related to different gestures based on a type of an object displayed by the virtual cursor.The apparatus of claim 1, wherein the second wearable device is configured to display a plurality of outputs corresponding to a plurality of inputs on a virtual space associated with the augmented reality display.A method performed by an apparatus, the method comprising: receiving at least one inertial measurement unit (IMU) signal and at least one electromyography (EMG) signal, wherein the at least one IMU signal is generated based on an inertial measurement unit (IMU) sensor, and wherein the at least one EMG signal is generated based on an EMG sensor; determining, based on the at least one IMU signal, a position of a user's forearm and an orientation of the forearm; mapping the determined position and orientation of the forearm to position information of a cursor displayed on a virtual space in a display; identifying, based on the at least one EMG signal, a gesture of a hand of the user; and determining, based on the identified gesture, an input associated with an object on the virtual space, the object corresponding to the cursor.The method of claim 11, further comprising: creating a plurality of objects and the cursor on the virtual space, wherein creating the cursor comprises implementing, as the cursor, a virtual ray associated with an orientation of the forearm and with a direction of the forearm.The method of claim 12, further comprising identifying the object on the virtual space by a ray casting technique using the virtual ray.The method of claim 12, further comprising processing a command associated with a target object identified by the cursor on the augmented reality implementing display.The method of claim 14, wherein the mapping comprises: determining three-dimensional coordinates with respect to the forearm; and mapping the three-dimensional coordinates to the cursor implemented as the virtual beam.The method of claim 11, further comprising determining a plurality of different inputs based on a plurality of commands corresponding to a plurality of gestures, wherein each of the plurality of different inputs corresponds to a respective one of the plurality of commands, and wherein at least one of the plurality of gestures is determined based on the at least one EMG signal.The method of claim 16, wherein the plurality of commands comprises a selection command corresponding to a first gesture of the plurality of gestures, an activation command corresponding to a second gesture of the plurality of gestures, a third command corresponding to a third gesture of the plurality of gestures, a menu command corresponding to a fourth gesture of the plurality of gestures, and a speech recognition command corresponding to a fifth gesture of the plurality of gestures.The method of claim 16, further comprising identifying different gestures among the plurality of gestures via EMG signal processing associated with an artificial intelligence model.The method of claim 11, further comprising: based on a type of an object displayed by the cursor, transmitting different commands related to a same gesture or transmitting a same command related to different gestures.The method of claim 11, further comprising displaying a plurality of outputs corresponding to a plurality of inputs on the virtual space.

Images