Augmented reality device for determining valid touch input by user and operation method thereof

Abstract

Provided are: an augmented reality device which determines a valid touch input to a touch interface so that a user can safely and conveniently use the augmented reality device, and prevents an interaction caused by an invalid touch input from being automatically performed; and an operation method thereof. The augmented reality device determines whether a touch input to the touch interface is a valid input resulting from an intention of the user on the basis of sensing data about a movement of a hand, and can determine whether to perform an interaction according to a result of the determination.

Description

Augmented reality device for determining valid touch input by user and method for operating the same

[0001] The present disclosure relates to an augmented reality device for determining a valid touch input and a method for operating the same. Specifically, the present disclosure discloses an augmented reality device and a method for operating the same, which determines whether a touch input to a touch interface is a valid input based on sensing data acquired through at least one sensor and prevents interaction due to an invalid input.

[0002] Augmented Reality (AR) is a technology that overlays virtual images onto the physical environment or real world objects of the real world. AR devices (e.g., smart glasses) that utilize AR technology are being used in everyday life for purposes such as information retrieval, route guidance, and camera shooting. In particular, smart glasses are worn as fashion items and are mainly used for outdoor activities.

[0003] An augmented reality device includes a touch interface that receives touch or tap input, such as a touchpad, physical key buttons, or switches. When a user's touch input is received, the touch interface performs an interaction corresponding to the touch input. For example, when a touch or tap input is received on the touch interface, the augmented reality device identifies a touch-sensitive area, displays a graphical user interface (GUI), such as a button or menu option corresponding to the identified touch-sensitive area, or provides feedback, such as a vibration or a notification sound, and performs a function or operation corresponding to the touch or tap input.

[0004] Unintentional and unauthorized touch inputs may occur on the touch interface of an AR device. Unauthorized touch inputs may be touch inputs made by someone other than the user of the AR device. Unintentional touch inputs may be touch inputs made accidentally by the user without the user's intention to provide the touch input.

[0005] Recently, technologies for providing security and / or personalized access to AR devices have been developed and widely adopted. However, no solution has been provided to prevent unintentional and unauthorized touch input on the touch interface while the user is wearing the AR device.

[0006] One aspect of the present disclosure provides an augmented reality device that determines whether a touch input is a valid input and performs an interaction based on the determination result. The augmented reality device according to one embodiment of the present disclosure may include at least one sensor, a touch interface configured to receive a touch input, at least one processor including a processing circuit, and a memory that stores one or more instructions. The one or more instructions are individually or collectively executed by the at least one processor, whereby the augmented reality device can sense a movement of a hand through the at least one sensor and determine whether a touch input received through the touch interface is a valid input based on the sensed movement of the hand. By individually or collectively executing one or more of the above commands by the at least one processor, the augmented reality device can determine whether to perform an interaction corresponding to a touch input based on a judgment result regarding a valid input.

[0007] One aspect of the present disclosure provides a method for an augmented reality device to determine a valid touch input and perform an interaction based on the determination result. A method for operating an augmented reality device according to one embodiment of the present disclosure may include a step of obtaining sensing data regarding hand movement using at least one sensor. The method for operating the augmented reality device may include a step of detecting a touch input to a touch interface. The method for operating the augmented reality device may include a step of determining whether a touch input is a valid input based on the obtained sensing data.

[0008] One aspect of the present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may include instructions readable by an augmented reality device, causing the augmented reality device to perform the following operations: acquiring sensing data regarding hand movements using at least one sensor; detecting a touch input to a touch interface; and determining whether a touch input is a valid input based on the sensing data regarding hand movements.

[0009] The present disclosure can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.

[0010] FIG. 1 is a drawing for explaining an operation performed by an augmented reality device according to one embodiment of the present disclosure to determine whether a touch input is a valid input and based on the determination result.

[0011] FIG. 2 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to determine whether a touch input is a valid input.

[0012] FIG. 3 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to perform an interaction based on whether a touch input is a valid input.

[0013] FIG. 4 is a block diagram illustrating components of an augmented reality device according to one embodiment of the present disclosure.

[0014] FIG. 5 is a diagram illustrating a method for an augmented reality device to detect a user's hand-raising motion according to one embodiment of the present disclosure.

[0015] FIG. 6 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to determine whether a touch input is a valid input based on a user's hand raising motion.

[0016] FIG. 7 is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to detect hand joint feature points from an image frame using an artificial intelligence model.

[0017] FIG. 8 is a flowchart illustrating a method for recognizing a hand-raising motion based on temporal and spatial correlation of joint feature points by an augmented reality device according to one embodiment of the present disclosure.

[0018] FIG. 9A is a diagram illustrating the arrangement relationship of a plurality of cameras mounted on an augmented reality device according to one embodiment of the present disclosure.

[0019] FIG. 9b is a diagram illustrating the arrangement relationship of a plurality of cameras mounted on an augmented reality device according to one embodiment of the present disclosure.

[0020] FIG. 10 is a diagram illustrating a method for recognizing a hand-raising gesture using a plurality of image frames acquired through a plurality of cameras by an augmented reality device according to one embodiment of the present disclosure.

[0021] FIG. 11 is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to identify temporal and spatial correlations of joint feature points detected from a plurality of image frames.

[0022] FIG. 12 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to recognize a hand-raising motion based on depth value information of the hand.

[0023] FIG. 13 is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to recognize a hand-raising motion based on depth value information of the hand.

[0024] FIG. 14 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to determine whether a touch input is a valid input based on sensing data received from a wearable device and to perform an interaction based on the determination result.

[0025] FIG. 15 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to determine whether a touch input is a valid input based on sensing data received from a wearable device.

[0026] FIG. 16a is a diagram illustrating an example in which a wearable device is located within an acceptable region.

[0027] Figure 16b is a diagram illustrating an example where a wearable device is outside the allowable area.

[0028] FIG. 17 is a flowchart illustrating a method for determining whether a touch input is a valid input based on EEG (electroencephalogram) signal data acquired using a brainwave sensor by an augmented reality device according to one embodiment of the present disclosure, and performing an interaction based on the determination result.

[0029] Figure 18 is a diagram illustrating an example of EEG signal data indicating changes in brain wave potential when a touch input is detected.

[0030] FIG. 19 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to determine whether a touch input is a valid input based on motion information and to perform an interaction based on the determination result.

[0031] FIG. 20a is a diagram illustrating an example of motion sensing data when a touch input is received.

[0032] FIG. 20b is a diagram illustrating an example of motion sensing data when an adjusting input is received.

[0033] The terms used in the embodiments of this specification have been selected from widely used terms as much as possible, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0034] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0035] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0036] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0037] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0038] In this disclosure, 'Augmented Reality' means displaying a virtual image together within a physical environment space of the real world or displaying a real object and a virtual image together.

[0039] In the present disclosure, an 'augmented reality device' is a device capable of expressing augmented reality, and may be, for example, not only augmented reality glasses in the shape of glasses worn by a user on the face, but also a head-mounted display apparatus (HMD: Head Mounted Display Apparatus) worn on the head, or an augmented reality helmet.

[0040] In one embodiment of the present disclosure, the augmented reality device may be replaced with a 'virtual reality device' (VR device).

[0041] In the present disclosure, a 'valid input' or 'valid input' means an input that has the effect of causing an augmented reality device to perform an interaction including a related function and / or operation based on a received touch input. In the present disclosure, whether a touch input on a touch interface is a valid input can be determined by the augmented reality device. In one embodiment of the present disclosure, a valid input or valid input may include an 'authorized input'.

[0042] In the present disclosure, 'authorized input' may refer to input provided by an authorized user with the intention of causing the augmented reality device to perform an interaction (e.g., a function or operation) corresponding to the input by touching or tapping the touch interface.

[0043] In the present disclosure, functions related to 'artificial intelligence' are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0044] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0045] In the present disclosure, the "artificial intelligence model" may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the computational results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network model may include a deep neural network (DNN), for example, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network, a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks, but is not limited to the examples described above.

[0046] In the present disclosure, 'vision recognition' refers to image signal processing that inputs an image into an artificial intelligence model and, through inference using the artificial intelligence model, recognizes (detection) an object from the input image, classifies the object into a specific category, or segments the object. In one embodiment of the present disclosure, vision recognition may refer to image processing that uses an artificial intelligence model to recognize a user's hand from an image captured by a vision sensor (e.g., a camera), and obtains location information of a plurality of feature points (e.g., joints) included in the hand.

[0047] In the present disclosure, a 'wearable device' is a device that is worn on a part of a user's body and carried while being worn. For example, the wearable device may be at least one of a smartwatch, a ring, a bracelet, an anklet, a necklace, a contact lens, a clothing-integrated device (e.g., electronic clothing), a body-attached device (e.g., a skin pad), or a bio-implantable device (e.g., an implantable circuit), but is not limited thereto.

[0048] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0049] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0050] FIG. 1 is a drawing for explaining an operation performed by an augmented reality device (100) according to one embodiment of the present disclosure to determine whether a touch input (20) is a valid input and based on the determination result.

[0051] The augmented reality device (100) is a device capable of expressing augmented reality, and may be implemented as, for example, a head-mounted display apparatus (HMD) worn by a user on the head, an augmented reality helmet, etc. In FIG. 1, the augmented reality device (100) is illustrated as a head-mounted display apparatus, but this is merely an example for the convenience of explanation and is not limited thereto. For example, the augmented reality device (100) may be configured as augmented reality glasses in the shape of glasses worn on the user's face.

[0052] Referring to FIG. 1, an augmented reality device (100) may include a vision sensor (110) and a touch interface (160). The vision sensor (110) may include a plurality of cameras (110RT, 110RB, 110LT, 110LB). In the embodiment illustrated in FIG. 1, the vision sensor (110) may include an upper left camera (110LT) and a lower left camera (110LB) which are respectively positioned at the upper and lower ends of a frame surrounding a left eye lens of the augmented reality device (100), and an upper right camera (110RT) and a lower right camera (110RB) which are respectively positioned at the upper and lower ends of a frame surrounding a right eye lens. The number and positions of the plurality of cameras (110RT, 110RB, 110LT, 110LB) included in the vision sensor (110) are merely examples and are not limited as illustrated in FIG. 1. In one embodiment of the present disclosure, the augmented reality device (100) may include at least two or more, for example, three, five, six, ..., n, multiple cameras.

[0053] The touch interface (160) is a hardware input device configured to receive a touch input or tap input from a user or an external source. In one embodiment of the present disclosure, the touch interface (160) may include a touchpad, a touchscreen, a physical key button, or a switch. In the embodiment illustrated in FIG. 1, the touch interface (160) may be disposed on one side of the augmented reality device (100), but is not limited thereto.

[0054] Although not illustrated in FIG. 1, the augmented reality device (100) may further include other components in addition to the vision sensor (110) and the touch interface (160). In one embodiment of the present disclosure, the augmented reality device (100) may further include a brainwave sensor (120, see FIG. 4), a motion sensor (130, see FIG. 4), and a communication interface (160, see FIG. 4). The components of the augmented reality device (100) will be described in detail with reference to FIG. 4.

[0055] The augmented reality device (100) obtains sensing data regarding hand movements using at least one sensor including a vision sensor (110) (operation ①).

[0056] The augmented reality device (100) detects a touch input (20) to the touch interface (160) from the user or an external source (operation ②).

[0057] The augmented reality device (100) determines whether the touch input (20) is a valid input based on sensing data regarding hand movement (action ③).

[0058] If the touch input (20) is determined to be a valid input, the augmented reality device (100) performs an interaction (function or action) corresponding to the touch input (20) (action ④-1).

[0059] If the touch input (20) is determined to be an invalid input, that is, an invalid input, the augmented reality device (100) ignores the touch input (20) and does not perform a function or operation (operation ④-2).

[0060] Hereinafter, with reference to FIG. 1 together with FIG. 2 and FIG. 3, a detailed description will be given of the function and / or operation of the augmented reality device (100) for determining whether a touch input (20) is a valid input and performing an interaction based on the determination result.

[0061] FIG. 2 is a flowchart illustrating a method for an augmented reality device (100) according to one embodiment of the present disclosure to determine whether a touch input is a valid input.

[0062] In step S210, the augmented reality device (100) obtains sensing data regarding hand movements using at least one sensor. Referring to FIG. 1, the augmented reality device (100) includes a vision sensor (110, see FIG. 1) and can obtain a plurality of image frames by taking pictures of the user's hand using a plurality of cameras (110RT, 110RB, 110LT, 110LB, see FIG. 1) included in the vision sensor (110). However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the augmented reality device (100) includes an electroencephalogram (EEG) sensor (120, see FIG. 4) and can obtain EEG (electroencephalogram) signal data by sensing potential changes due to brainwaves of the user's head using the electroencephalogram (EEG) sensor (120). In one embodiment of the present disclosure, the augmented reality device (100) further includes a communication interface (170, see FIG. 4) and can receive an Ultra Wide Band (UWB) signal or a Bluetooth signal from a wearable device worn on a user's hand through the communication interface (170). For example, the augmented reality device (100) can receive Angle of Arrival (AoA) information from a UWB signal or Bluetooth Low Energy (BLE) location information from a Bluetooth signal. However, the present invention is not limited thereto.

[0063] In one embodiment of the present disclosure, the augmented reality device (100) includes a motion sensor (130, see FIG. 4) and can obtain motion information regarding vibration or movement of the augmented reality device (100) through the motion sensor (130).

[0064] In one embodiment of the present disclosure, the augmented reality device (100) inputs a plurality of image frames acquired through a plurality of cameras (110RT, 110RB, 110LT, 110LB) included in the vision sensor (110) into an artificial intelligence model, and performs vision recognition using the artificial intelligence model to detect feature points regarding hand joints from the plurality of image frames. The 'artificial intelligence model' according to one embodiment of the present disclosure may be implemented as a deep neural network model trained to recognize an object (e.g., a user's hand) from input image data and recognize feature points (e.g., joints) of the object. In the present disclosure, the 'deep neural network model' may be an end-to-end model trained in a supervised learning manner that applies tens of thousands or hundreds of millions of multiple images as input data and applies feature points of hand joints included in the input data as ground truth values. The deep neural network model can be implemented as, for example, a convolutional neural network (CNN) model, but is not limited thereto. The deep neural network model can also be implemented as, for example, a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks. The augmented reality device (100) can recognize a hand raising motion based on the movement of detected feature points over time.

[0065] The augmented reality device (100) can obtain position information of the user's hand based on sensing data received from an external device, for example, a wearable device worn on the user's hand. In one embodiment of the present disclosure, the augmented reality device (100) can obtain a relative positional relationship between the user's hand and the augmented reality device (100) based on a UWB signal or a Bluetooth signal received from the wearable device. In the present disclosure, the 'relative positional relationship' can include at least one of a distance, a direction, and an orientation between the user's hand wearing the wearable device and the augmented reality device (100).

[0066] The augmented reality device (100) can obtain biometric information about the user's brain waves from EEG signal data acquired through an EEG sensor (120, see FIG. 4). In one embodiment of the present disclosure, the augmented reality device (100) can identify negative feedback of brain wave potentials based on the EEG signal data.

[0067] In step S220, the augmented reality device (100) can detect a touch input to the touch interface (160, see FIG. 1). The touch input to the touch interface (160) may be an input by a user, but is not limited thereto. The touch interface (160) may also receive an unauthorized touch input by an outsider other than an authorized user wearing the augmented reality device (100).

[0068] In step S230, the augmented reality device (100) can determine whether the touch input is a valid input based on the sensing data. In the present disclosure, a 'valid input' means an input that has the effect of causing the augmented reality device (100) to perform an interaction including a function and / or operation corresponding to the touch input. In one embodiment of the present disclosure, a valid input may include an input authorized by a user's intention.

[0069] In an embodiment of recognizing a hand-raising gesture from multiple image frames, the augmented reality device (100) may determine a touch input as an input applied by the user's intention if a touch input is detected within a preset time from the time the hand-raising gesture is recognized.

[0070] In an embodiment of acquiring sensing data from a wearable device and acquiring a relative positional relationship between a user's hand and an augmented reality device (100) based on the sensing data, the augmented reality device (100) can identify whether the user's hand is located within a preset acceptable region based on the relative positional relationship, and determine whether a touch input is a valid input based on the identification result. For example, if the user's hand is located within a region within a preset distance, for example, 5 centimeters (cm), from the augmented reality device (100), the augmented reality device (100) can determine the touch input as an input applied by the user's intention.

[0071] In an embodiment of acquiring EEG signal data from an EEG sensor (120), the augmented reality device (100) can identify negative feedback of the user's brain wave potential, for example, error-related negativity (ERN), from the EEG signal data, and determine whether the touch input is a valid input based on the identification result. For example, if the error-related negativity is identified within a preset time from the time when the touch input is received, the augmented reality device (100) can determine the touch input as an applied input. The 'preset time' may be, for example, 50 milliseconds (ms) or more and 100 ms or less. However, it is not limited thereto.

[0072] The method by which the augmented reality device (100) performs interaction based on the result of determining whether a touch input is a valid input will be described in detail with reference to FIG. 3.

[0073] FIG. 3 is a flowchart illustrating a method in which an augmented reality device (100) according to one embodiment of the present disclosure performs an interaction based on whether a touch input is a valid input.

[0074] In step S310, the augmented reality device (100) determines whether the touch input is a valid input based on the user's intention. The specific method for determining this is the same as that described in step S230 of FIG. 2, so a duplicate description will be omitted.

[0075] If the touch input is determined to be a valid input (step S320), the augmented reality device (100) performs a function or operation corresponding to the touch input. In one embodiment of the present disclosure, if the touch input is determined to be an input applied by the user's intention, the augmented reality device (100) can perform an interaction corresponding to the touch input.

[0076] If the touch input is determined to be an invalid input (step S330), the augmented reality device (100) ignores the touch input and terminates without performing any function or operation. In one embodiment of the present disclosure, if the touch input is determined to be an unintentional and unauthorized input, the augmented reality device (100) may not perform an interaction corresponding to the touch input.

[0077] Unintentional and unauthorized touch inputs may occur on the touch interface (160, see FIG. 1) of the augmented reality device (100). In the present disclosure, an 'unauthorized touch input' may be a touch input by someone other than the user of the augmented reality device (100). For example, when a user is watching a movie through the augmented reality device (100), a family member may make a touch input for a pause or window closing action, or when a navigation application is run through the augmented reality device (100) in a crowded place, a stranger may accidentally make a touch input and terminate the navigation application, etc., and thus, a touch input by an unauthorized outsider may occur. Or, when enjoying a multiplayer virtual reality (VR) game using an augmented reality device (100), an unauthorized external touch input may occur, such as when another player accidentally touches the touch interface (160) while playing with another player or performing a physical activity, causing the game to be temporarily suspended. In the present disclosure, an 'unintentional touch input' may be a touch input that is accidentally provided by a user without the intention of providing a touch input. For example, an unintentional touch input may occur when a user accidentally places a hand on the touch interface (160) of the augmented reality device (100) while trying to fix a hairstyle, or when a user touches the touch interface (160) while trying to approach an object on a shelf.

[0078] Recently, technologies for providing security and / or personalized access to augmented reality devices (100) have been developed and widely distributed. However, no solution has been provided that can prevent unintentional and unauthorized touch input to a touch interface (160) while a user is wearing the augmented reality device (100).

[0079] The present disclosure aims to provide an augmented reality device (100) and an operating method thereof that determines a valid touch input to a touch interface (160) so that a user can use the augmented reality device (100) safely and conveniently, and prevents interaction by unintentional and unauthorized touch input, i.e., invalid input, from being automatically performed.

[0080] The augmented reality device (100) according to the embodiment illustrated and described through FIGS. 1 to 3 obtains sensing data regarding hand movements through a vision sensor, an brainwave sensor, or an external sensor (e.g., UWB, Bluetooth, etc. of a wearable device), and when a touch input is detected by a touch interface (160), determines whether the touch input is a valid input based on the sensing data regarding the hand movements, and if it is determined to be an invalid input according to the determination result, ignores the touch input and may not perform an interaction corresponding to the touch input. The augmented reality device (100) according to one embodiment of the present disclosure can prevent a touch input due to a user's mistake, thereby enhancing comfort and immersion and improving user experience (UX). In addition, the augmented reality device (100) according to one embodiment of the present disclosure can prevent interaction from being automatically performed by unauthorized touch input by an unauthorized outsider, thereby preventing leakage of sensitive data such as personal information, thereby providing a technical effect of strengthening security.

[0081] An augmented reality device (100) according to one embodiment of the present disclosure can detect the surrounding environment and automatically or by user input ignore unintentional and unauthorized touch inputs that are judged as invalid inputs when a crowded crowd environment (e.g., concert, subway, etc.) is detected, thereby implementing personalized automatic authentication.

[0082] FIG. 4 is a block diagram illustrating components of an augmented reality device (100) according to one embodiment of the present disclosure.

[0083] Referring to FIG. 4, the augmented reality device (100) may include a vision sensor (110), an electroencephalographic sensor (120), a motion sensor (130), a processor (140), a memory (150), a touch interface (160), and a communication interface (170). The vision sensor (110), the electroencephalographic sensor (120), the motion sensor (130), the processor (140), the memory (150), the touch interface (160), and the communication interface (170) may be electrically and / or physically connected to each other, respectively. FIG. 4 illustrates components for explaining the operation of the augmented reality device (100), and the components included in the augmented reality device (100) are not limited to those illustrated in FIG. 4. The augmented reality device (100) may not include some of the components illustrated in FIG. 4. In one embodiment of the present disclosure, the augmented reality device (100) may not include the electroencephalographic sensor (120). In one embodiment of the present disclosure, the augmented reality device (100) may not include a motion sensor (130). Furthermore, in one embodiment of the present disclosure, the augmented reality device (100) may not include both a brainwave sensor (120) and a motion sensor (130).

[0084] In one embodiment of the present disclosure, the augmented reality device (100) is implemented as a portable device, in which case the augmented reality device (100) may further include a battery that supplies driving power to a vision sensor (110), an brainwave sensor (120), a motion sensor (130), a processor (140), a touch interface (160), and a communication interface (170).

[0085] A vision sensor (110) is configured to capture a hand image by photographing a real space and a hand within the real space. The vision sensor (110) may include one or more cameras. The camera may include a lens module, an image sensor, and an image processing module. The camera may capture a still image or video of an object by an image sensor (e.g., CMOS or CCD). The video may include a plurality of image frames continuously captured by photographing the object through the camera. The image processing module may encode a still image composed of a single image frame captured by the image sensor or video data composed of a plurality of image frames and transmit the encoded data to the processor (140).

[0086] In one embodiment of the present disclosure, the vision sensor (110) may be implemented in a small form factor so that it can be mounted on a portable augmented reality device (100), and may be implemented as a lightweight RGB camera that consumes low power. However, the present invention is not limited thereto, and the vision sensor (110) may include a depth camera such as a stereo camera, a time of flight (ToF) camera, or an infrared (IR) camera.

[0087] The vision sensor (110) may include two or more cameras. For example, when the augmented reality device (100) is implemented as a head-mounted display or augmented reality glasses, the vision sensor (110) may include a total of six cameras, including a left top camera, a left middle camera, and a left bottom camera, which are respectively positioned at the top, middle, and bottom of a frame surrounding the left eye lens, and a right top camera, a right middle camera, and a right bottom camera, which are respectively positioned at the top, middle, and bottom of a frame surrounding the right eye lens. The positional relationship between the plurality of cameras included in the vision sensor (110) and the augmented reality device (100) will be described in detail with reference to FIGS. 9A and 9B. However, the number and positional relationship of the plurality of cameras are not limited to the above-described examples.

[0088] The vision sensor (110) can obtain multiple image frames including the hand by photographing the user's hand in real space through multiple cameras.

[0089] The brainwave sensor (120) may include an EEG sensor configured to acquire electroencephalogram (EEG) signal data by sensing potential fluctuations caused by brainwaves. In one embodiment of the present disclosure, the brainwave sensor (120) may detect an event-related potential (ERP) component including at least one of a feedback-related negativity (ERP-FRN) component and a feedback-related positivity (ERP-FRP) component by monitoring potential fluctuations caused by brainwaves based on EEG signal data. The brainwave sensor (120) may provide the detected event-related potential component to the processor (140).

[0090] The motion sensor (130) is a sensor configured to sense motion information regarding the movement of the augmented reality device (100). In one embodiment of the present disclosure, the motion sensor (130) can sense vibration or movement of the augmented reality device (100) to obtain motion information when a user input or an unauthorized external input is applied to the augmented reality device (100). The motion sensor (130) can provide the obtained motion information to the processor (140).

[0091] The processor (140) can execute one or more instructions of a program stored in the memory (150). The processor (140) may be configured with hardware components that perform arithmetic, logic, and input / output operations and image processing. Although the processor (140) is illustrated as a single element in FIG. 4, it is not limited thereto. In one embodiment of the present disclosure, the processor (140) may be configured with one or more multiple elements. One or more processors included in the processor (140) may be circuitry such as a system on chip (SoC), an integrated circuit (IC), or the like. For example, the processor (140) may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), a graphics-only processor such as a graphics processing unit (GPU), a vision processing unit (VPU), or an artificial intelligence-only processor such as a neural processing unit (NPU).

[0092] The processor (140) may include various processing circuits and / or multiple processors. For example, the term "processor" as used in this disclosure, including the claims, may include various processing circuits, including at least one processor. One or more processors in at least one processor may be configured to perform various functions described in this disclosure individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0093] The processor (140) can be controlled to process input data according to predefined operating rules or artificial intelligence models. Alternatively, if the processor (140) is a dedicated artificial intelligence processor, the dedicated artificial intelligence processor can be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0094] The memory (150) may be configured as at least one type of storage medium, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk.

[0095] The memory (150) may store instructions related to functions and / or operations for the augmented reality device (100) to obtain sensing data regarding hand movement from at least one of a vision sensor (110), an brainwave sensor (120), and a motion sensor (130), obtain at least one of hand movement information, position information, and biometric information based on the obtained sensing data, and determine whether a touch input to the touch interface (160) is a valid input based on at least one of hand movement information, position information, and biometric information. In one embodiment of the present disclosure, the memory (150) may store at least one of instructions, an algorithm, a data structure, a program code, and an application program that can be read by the processor (140). The instructions, algorithms, data structures, and program codes stored in the memory (150) may be implemented in a programming or scripting language such as, for example, C, C++, Java, assembler, etc.

[0096] The processor (140) may be implemented by executing instructions or program codes stored in the memory (150). Hereinafter, the functions and / or operations performed by the processor (140) by executing instructions or program codes of each of the plurality of modules stored in the memory (150) will be described in detail.

[0097] The processor (140) may obtain sensing data regarding hand movement from at least one of a vision sensor (110), an brain wave sensor (120), and a motion sensor (130), and may obtain at least one of hand motion information, position information, and biometric information based on the obtained sensing data. In one embodiment of the present disclosure, the processor (140) may obtain a plurality of image frames obtained by photographing a hand from a plurality of cameras included in the vision sensor (110). The processor (140) may input the plurality of image frames obtained through the plurality of cameras into an artificial intelligence model, and perform vision recognition using the artificial intelligence model to detect feature points regarding joints of the hand from the plurality of image frames. The 'artificial intelligence model' according to one embodiment of the present disclosure may be implemented as a deep neural network model trained to recognize an object (e.g., a user's hand) from input image data and recognize feature points (e.g., joints) of the object. The artificial intelligence model will be described in detail with reference to FIG. 7.

[0098] The processor (140) can continuously acquire a plurality of image frames from the vision sensor (110) over time, and recognize a hand-raising motion based on the movement of feature points detected from the plurality of continuously acquired image frames over time. A specific embodiment in which the processor (140) detects feature points from the plurality of image frames acquired through the vision sensor (110) and recognizes a hand-raising motion based on the detected feature points will be described in detail with reference to FIGS. 7 to 11.

[0099] In one embodiment of the present disclosure, the vision sensor (110) includes a depth camera configured to acquire a depth value of an object, and the processor (140) can capture a hand over time through the depth camera to acquire a plurality of image frames, and acquire a depth value of the hand from the acquired plurality of image frames. The processor (140) can recognize a change in the acquired depth value over time, and recognize a hand-raising action based on the change in the depth value. A specific embodiment in which the processor (140) acquires the depth value of the hand from the plurality of image frames acquired through the depth camera, and recognizes the hand-raising action based on the change in the depth value over time will be described in detail with reference to FIGS. 12 and 13.

[0100] In one embodiment of the present disclosure, the processor (140) can obtain a UWB signal or a Bluetooth signal from a sensor of a wearable device through a communication interface (170). A wearable device is a device that is worn on a part of a user's body and carried while being worn, and may be, for example, a smart watch worn on the user's hand. However, the present disclosure is not limited thereto, and the wearable device may include, for example, a smart ring, a bracelet, an anklet, a necklace, a contact lens, a clothing-integrated device (e.g., electronic clothing), a body-attached device (e.g., a skin pad), or a bio-implantable device (e.g., an implantable circuit). In one embodiment of the present disclosure, the wearable device includes a UWB communication module or a Bluetooth communication module, and the processor (140) can receive AoA (Angle of Arrival) information of a UWB signal from the wearable device through the communication interface (170) or receive BLE (Bluetooth Low Energy) location information from a Bluetooth signal. The processor (140) may obtain a relative positional relationship between the user's hand and the augmented reality device (100) based on the received AoA information or BLE location information. In one embodiment of the present disclosure, the 'relative positional relationship' may include at least one of a distance, a direction, and an orientation between the hand of the user wearing the wearable device and the augmented reality device (100).

[0101] In one embodiment of the present disclosure, the processor (140) can obtain EEG (electroencephalogram) signal data by sensing potential fluctuations of brain waves of the user's head through the brain wave sensor (120).

[0102] In one embodiment of the present disclosure, the processor (140) may obtain motion information regarding the movement of the augmented reality device (100) through the motion sensor (130). When the touch interface (160) receives an adjusting input from the user to adjust the augmented reality device (100) by changing setting information or options, vibration or movement may occur in the augmented reality device (100). When the user's adjusting input is detected through the touch interface (160), the processor (140) may obtain motion information regarding the vibration or movement occurring in the augmented reality device (100). Motion sensing data regarding the cases where a general touch input is received and the cases where an adjusting input is input will be described in detail with reference to FIGS. 20A and 20B , respectively.

[0103] The processor (140) may determine whether a touch input detected by the touch interface (160) is a valid input based on at least one of hand movement information (e.g., a hand-raising motion), position information, and biometric information (e.g., EEG signal data). In one embodiment of the present disclosure, the processor (140) may determine whether a touch input is an authorized input based on a user's intention based on at least one of hand movement information, position information, and biometric information. In an embodiment that recognizes hand movement information, for example, a hand-raising motion, the processor (140) may determine the touch input as an authorized input based on the user's intention, i.e., a valid input, if a touch input to the touch interface (160) is detected within a preset time from the time at which the hand-raising motion is recognized. In this case, the processor (140) may perform an interaction corresponding to the touch input. For example, the processor may perform a function and / or operation such as executing an application, selecting a menu, or changing an option based on a touch input. If a touch input is detected after a preset time has elapsed from the time a hand-raising motion is detected, the processor (140) may determine the touch input as an unintentional, unauthorized input, i.e., an invalid input. In this case, the processor (140) may ignore the touch input and may not perform a function and / or operation corresponding to the touch input.

[0104] In an embodiment of acquiring sensing data from a wearable device and acquiring position information of a user's hand based on the sensing data, the processor (140) may identify whether the user's hand is located within a preset acceptable region based on the relative positional relationship between the user's hand and the augmented reality device (100), and determine whether a touch input is a valid input based on the identification result. For example, if the user's hand is located within a region within a preset distance, for example, 5 centimeters (cm), from the augmented reality device (100), the processor (140) may determine the touch input as an input applied by the user's intention, i.e., a valid input. In this case, the processor (140) may perform an interaction corresponding to the touch input. For the opposite example, if the user's hand is located within a region outside a preset distance or preset angular range from the augmented reality device (100), the processor (140) may determine the touch input as an unintentional, unauthorized input, i.e., an invalid input. In this case, the processor (140) may ignore the touch input and may not perform a function and / or operation corresponding to the touch input. A specific embodiment in which the augmented reality device (100) determines whether a touch input is an applied input based on sensing data acquired from a wearable device and performs an interaction based on the determination result will be described in detail with reference to FIGS. 14 to 16b.

[0105] In an embodiment of acquiring biometric information, for example, EEG signal data from an EEG sensor (120), the processor (140) may identify negative feedback of the user's brainwave potential, for example, error-related negativity (ERN), from the EEG signal data, and determine whether a touch input is a valid input based on the identification result. In one embodiment of the present disclosure, if an error-related negativity is identified within a preset time from the time at which a touch input is detected by the touch interface (160), the processor (140) may determine the touch input as an applied input. The 'preset time' may be, for example, 50 milliseconds (ms) or more and 100 ms or less. However, the present invention is not limited thereto. If the touch input is determined to be an applied input, the processor (140) may perform an interaction corresponding to the touch input. For example, if an error-related negative potential is not identified even after a preset amount of time has passed since the touch input was detected, the processor (140) may determine the touch input as an unintentional, unauthorized input, i.e., an invalid input. In this case, the processor (140) may ignore the touch input and may not perform a function and / or operation corresponding to the touch input. A specific embodiment in which the augmented reality device (100) determines whether a touch input is an applied input based on biometric information (e.g., EEG signal data) acquired through the brainwave sensor (120) and performs an interaction based on the determination result will be described in detail with reference to FIGS. 17 and 18.

[0106] In an embodiment of acquiring motion information of an augmented reality device (100) from a motion sensor (130), if motion information is acquired by the motion sensor (130) after a touch input is detected, the processor (140) may determine the touch input as an unintentional and unauthorized input. In this case, the processor (140) may ignore the touch input and may not perform a function and / or operation corresponding to the touch input.

[0107] The touch interface (160) is a hardware input device configured to receive a touch input or tap input from a user or an external party. In one embodiment of the present disclosure, the touch interface (160) may include a touchpad, a touchscreen, a physical key button, or a switch. The touch interface (160) may detect a touch input and provide information about the detected touch input to the processor (140).

[0108] The communication interface (170) is a hardware device configured to transmit and receive data with a server or an external device via a wired or wireless communication network. The communication interface (170) may perform data communication with a server or an external device using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, BLE (Bluetooth Low Energy), zigbee, Wi-Fi Direct, infrared data association (IrDA), near field communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication.

[0109] In one embodiment of the present disclosure, the communication interface (170) may include an Ultra Wide Band (UWB) communication module that performs UWB communication. In the present disclosure, "UWB (Ultra Wide Band) communication" refers to a communication method that performs data transmission and reception using an ultra-wideband frequency band between 3.1 GHz and 10.6 GHz. A UWB communication network can transmit and receive data at a maximum speed of 500 Mbps.

[0110] However, the present invention is not limited thereto, and when the augmented reality device (100) is implemented as a device that is worn on a user's body part and carried, such as augmented reality glasses or a head-mounted display device, the communication interface (170) can transmit and receive data with a server or an external device through a network that follows a mobile communication standard, such as a communication method using CDMA, WCDMA, 3G, 4G (LTE), 5G Sub 6, and / or millimeter wave (mmWave).

[0111] In one embodiment of the present disclosure, the communication interface (170) can receive a UWB signal or a Bluetooth signal from a wearable device worn on a user's hand under the control of the processor (140). The communication interface (170) can provide the received UWB signal or Bluetooth signal to the processor (140).

[0112] FIG. 5 is a diagram illustrating a method for an augmented reality device (100) according to one embodiment of the present disclosure to detect a hand-raising motion of a user (10).

[0113] Referring to FIG. 5, a user (10) wearing an augmented reality device (100) may take a steady posture at a first time point (t=N) and then take a hand-raising action at a second time point (t=N+1). The augmented reality device (100) may capture a hand of the user (10) using a vision sensor (110, see FIGS. 1 and 4) including a plurality of cameras to obtain a plurality of image frames, and may detect feature points of joints included in the hand from the obtained plurality of image frames. The augmented reality device (100) may detect feature points of hand joints from a plurality of image frames that are continuously obtained over time through the vision sensor (110), and may recognize a hand-raising action based on movement of the detected feature points over time.

[0114] When a touch input to the touch interface (160) is detected within a preset time from the time point at which the hand-raising motion is recognized, i.e., the second time point, the augmented reality device (100) can determine the touch input as an input applied by the user's intention. In the embodiment illustrated in FIG. 5, when a touch input of touching or tapping the touch interface (160) is received at a third time point (t = N + total time) before the preset time has elapsed from the second time point (t = N), the augmented reality device (100) can determine the touch input as an applied input.

[0115] Conversely, if a touch input that touches or taps the touch interface (160) is received after a preset time has elapsed from the second time point (t=N), the augmented reality device (100) may determine the touch input as an unintentional and unauthorized input. If the touch input is determined to be an unintentional and unauthorized input, the augmented reality device (100) may ignore the touch input and may not perform an interaction corresponding to the touch input.

[0116] The augmented reality device (100) may also recognize a hand-down gesture based on the movement of feature points of the hand joints detected from multiple image frames. In one embodiment of the present disclosure, once the hand-down gesture is recognized, the augmented reality device (100) may determine all detected touch inputs as unauthorized inputs until a hand-raising gesture is recognized again.

[0117] FIG. 6 is a flowchart illustrating a method for an augmented reality device (100) according to one embodiment of the present disclosure to determine whether a touch input is a valid input based on a user's hand-raising motion.

[0118] FIG. 7 is a diagram illustrating an augmented reality device (100) according to an embodiment of the present disclosure that uses an artificial intelligence model (700) to extract hand joint feature points (P1 to P) from a plurality of image frames (710-1 to 710-n). 10 ) is a diagram illustrating a method for detecting.

[0119] Hereinafter, with reference to FIGS. 6 and 7 together, a detailed description will be given of a function and / or operation of an augmented reality device (100) according to one embodiment of the present disclosure for recognizing a hand-raising motion from a plurality of image frames (710-1 to 710-n) and determining whether a touch input is a valid input based on the hand-raising motion.

[0120] Step S610 of FIG. 6 is a step that embodies step S210 illustrated in FIG. 2. In step S610, the augmented reality device (100) acquires a plurality of image frames regarding the hand by continuously photographing the user's hand using a camera. In one embodiment of the present disclosure, the vision sensor (110, see FIGS. 1 and 4) of the augmented reality device (100) may include a plurality of cameras. Referring also to FIG. 7, the augmented reality device (100) may acquire a plurality of image frames (710-1 to 710-n) by continuously photographing the user's hand using a plurality of cameras. In one embodiment of the present disclosure, the plurality of image frames (710-1 to 710-n) may include images regarding not only the user's hand but also at least one body part of a wrist, an elbow, an arm, or a shoulder.

[0121] In step S620 of FIG. 6, the augmented reality device (100) inputs a plurality of image frames into an artificial intelligence model to detect feature points related to joints of a hand. In the present disclosure, a 'joint' refers to a portion where a plurality of bones included in a hand and an arm are connected to each other. In the present disclosure, a 'feature point' may refer to a point within an image that is easily distinguishable from the surrounding background or identifiable. Feature points of a hand joint may include, for example, feature points of a wrist joint, feature points of a palm joint, feature points of a wrist joint, feature points of a forearm, or feature points of an upper arm.

[0122] Referring to FIG. 7, the processor (140, see FIG. 4) of the augmented reality device (100) inputs a plurality of image frames (710-1 to 710-n) to an artificial intelligence model (700), and performs inferencing using the artificial intelligence model (700) to derive feature points (P1 to P) about joints from the plurality of image frames (710-1 to 710-n). 10 ) can be detected. In one embodiment of the present disclosure, the 'artificial intelligence model' may be implemented as a deep neural network model trained to recognize an object (e.g., a user's hand) from input image data and recognize feature points (e.g., joints) of the object. In the present disclosure, the 'deep neural network model' may be a model trained in a supervised learning manner that applies tens of thousands or hundreds of millions of multiple images as input data and applies feature points of hand joints included in the input data as ground truth.

[0123] The artificial intelligence model (700) may include, for example, a 3D feature extractor block, which is an architecture for hand joints, a feature transform layer (FTL) that generates 3D features, and a pose regression block that recognizes a pose based on skeletal features. The pose regression block may be implemented with a known skeleton regressor, for example, Regressor-K. When a plurality of image frames (710-1 to 710-n) are input to the artificial intelligence model (700), feature points (P1 to P) for hand joints are extracted through the 3D feature extractor block. 10 ) are extracted, and the extracted feature points (P1 to P 10 ) is a 3D feature point (z) by the feature transformation layer (FTL). 3D ) is converted to a 3D feature point (z 3D ) is input to the pose regression block and z containing 3D information, temporal context, and skeletal features by the pose regression block. R This can be output. Skeletal Regressor (Regressor-K) is z R When this is input, a 3D hand pose can be predicted. In one embodiment of the present disclosure, the artificial intelligence model (700) may be a deep neural network model that is trained end-to-end to predict a 3D hand pose with respect to a hand joint through a 3D feature extractor block, a feature transformation layer (FTL), a pose regression block, and a skeletal regressor analyzer (Regressor-K) when an image is input.

[0124] Referring again to FIG. 6, in step S630, the augmented reality device (100) can recognize a hand-raising motion based on the movement of the detected feature point over time. Referring also to FIG. 7, the processor (140) of the augmented reality device (100) can recognize a hand-raising motion based on a three-dimensional hand posture output through an artificial intelligence model (700).

[0125] Although not shown in FIG. 6, step S220 shown in FIG. 2 may be performed after step S630 is performed.

[0126] Steps S640 to S660 of FIG. 6 are steps that specify step S230 illustrated in FIG. 2. In step S640, the augmented reality device (100) determines whether a touch input is detected within a preset time from the time when a hand-raising motion is recognized.

[0127] As a result of the determination, if a touch input is detected within a preset time from the time when the hand-raising motion is recognized (step S650), the augmented reality device (100) determines the touch input as a valid input. In one embodiment of the present disclosure, if a touch input is detected within a preset time from the time when the hand-raising motion is recognized, the augmented reality device (100) may determine that the touch input is an authorized input by the user's intention. If the touch input is determined to be an authorized input (step S320), the augmented reality device (100) performs a function or operation corresponding to the touch input.

[0128] As a result of the determination, if a touch input is detected after a preset time has elapsed from the time when the hand-raising motion is recognized (step S660), the augmented reality device (100) determines the touch input as an invalid input. In one embodiment of the present disclosure, if a touch input is detected after a preset time has elapsed from the time when the hand-raising motion is recognized, the augmented reality device (100) may determine the touch input as an unintentional and unauthorized input. If the touch input is determined to be an unintentional and unauthorized input (step S330), the augmented reality device (100) ignores the touch input and terminates without performing a function or operation.

[0129] FIG. 8 is a flowchart illustrating a method for recognizing a hand-raising motion based on temporal and spatial correlation of joint feature points by an augmented reality device (100) according to one embodiment of the present disclosure.

[0130] Steps S810 to S830 illustrated in FIG. 8 are steps that embody step S630 of FIG. 6. After the function and / or operation of step S830 illustrated in FIG. 8 is performed, step S640 of FIG. 6 may be performed.

[0131] Step S810 may be performed after the function and / or operation by step S620 illustrated in FIG. 6 is performed. In step S810, the augmented reality device (100) detects feature points for each part of the hand from each of a plurality of image frames having different view points captured by a plurality of cameras. In one embodiment of the present disclosure, the vision sensor (110, see FIGS. 1 and 4) of the augmented reality device (100) may include a plurality of cameras that are each disposed at different locations and capture images having different view points. The plurality of cameras included in the vision sensor (110) will be described with reference to FIGS. 9A and 9B together.

[0132] FIG. 9A is a diagram illustrating the arrangement relationship of a plurality of cameras (110LT, 110LM, 110LB, 110RT, 110RM, 110RB) mounted on an augmented reality device (100) according to one embodiment of the present disclosure.

[0133] Referring to FIG. 9A, the augmented reality device (100) may be implemented as a head mounted display apparatus (HMD) worn on the user's head. In the embodiment illustrated in FIG. 9A, the head mounted display apparatus may include a left top camera (110LT) and a right top camera (110RT) positioned at the top of a frame surrounding the left eye lens and the right eye lens, a left middle camera (110LM) and a right middle camera (110RM) positioned at the middle of the frame, and a left bottom camera (110LB) and a right bottom camera (110RB) positioned at the bottom of the frame. Although FIG. 9A illustrates that the augmented reality device (100) includes a total of six cameras, this is merely an example and is not limited thereto.

[0134] Since each of the multiple cameras (110LT, 110LM, 110LB, 110RT, 110RM, 110RB) is positioned differently on the augmented reality device (100) and has a different viewpoint, the body parts they capture may not be identical. For example, the upper left camera (110LT) and the upper right camera (110RT) are vision sensors that capture the front view. Therefore, if a hand is not positioned in the front view, the upper left camera (110LT) and the upper right camera (110RT) cannot capture the hand and may obtain an image frame that does not include the hand. The middle left camera (110LM) and the middle right camera (110RM) can capture the upper arm and the hand, thereby obtaining an image frame that includes both the arm and the hand. The lower left camera (110LB) and the lower right camera (110RB) are vision sensors that capture a downward view and can acquire image frames of the user's forearm and palm.

[0135] FIG. 9b is a diagram illustrating the arrangement relationship of a plurality of cameras (110LT, 110LM, 110LB, 110RT, 110RM, 110RB) mounted on an augmented reality device (100) according to one embodiment of the present disclosure.

[0136] Referring to FIG. 9B, the augmented reality device (100) may be implemented as augmented reality glasses in the shape of glasses worn on the user's face. In the embodiment illustrated in FIG. 9B, the augmented reality glasses may include a left top camera (110LT) and a right top camera (110RT) positioned at the top of a frame surrounding the left eye lens and the right eye lens, a left middle camera (110LM) and a right middle camera (110RM) positioned at the middle of the frame, and a left bottom camera (110LB) and a right bottom camera (110RB) positioned at the bottom of the frame. Although FIG. 9B illustrates that the augmented reality device (100) includes a total of six cameras, this is merely an example and is not limited to the illustrated embodiment.

[0137] In the embodiment illustrated in FIG. 9b, the portions (e.g., upper arm, forearm, hand, palm, etc.) photographed by the plurality of cameras (110LT, 110LM, 110LB, 110RT, 110RM, 110RB) are the same as in FIG. 9a, so a duplicate description is omitted.

[0138] Referring back to FIG. 8, the augmented reality device (100) can detect feature points for each part of the hand from each of a plurality of image frames acquired through a plurality of cameras (110LT, 110LM, 110LB, 110RT, 110RM, 110RB, see FIGS. 9A and 9B). For example, the augmented reality device (100) can detect joint feature points included in the upper arm and the hand from image frames acquired through the left middle camera (110LM) and the right middle camera (110RM), and can detect joint feature points of the forearm and the palm from image frames acquired through the left lower camera (110LB) and the right lower camera (110RB).

[0139] In step S820, the augmented reality device (100) combines multiple image frames to identify temporal and spatial correlations between detected feature points. Step S820 will be described in detail with reference to FIGS. 10 and 11 .

[0140] FIG. 10 is a diagram illustrating a method for recognizing a hand-raising gesture using a plurality of image frames acquired through a plurality of cameras (110RM, 110RB) by an augmented reality device (100) according to one embodiment of the present disclosure.

[0141] Referring to FIG. 10, the right-center camera (110RM) continuously captures the user's arm and hand over time to create multiple image frames (i 1-RM Inland i 4-RM ) can be obtained. The lower right camera (110RB) continuously captures the user's arms and hands over time to obtain multiple image frames (i 1-RB Inland i 4-RB ) can be obtained.

[0142] The processor (140, see Fig. 4) of the augmented reality device (100) acquires a plurality of image frames (i) obtained by the right-center camera (110RM). 1-RM Inland i 4-RM ) from the feature points (P) about the joints 1-RM , P 2-RM ) can be detected. For example, the first joint feature point (P 1-RM ) is the feature point of the hand joint, and the second joint feature point (P 2-RM ) may be a feature point of an arm joint. In Fig. 10, a feature point of a hand joint and a feature point of an arm joint are each illustrated as one, but this is for convenience of explanation, and the number of feature points of the present disclosure is not limited as illustrated in Fig. 10. In the same manner, the processor (140) acquires a plurality of image frames (i) acquired by the lower right camera (110RB). 1-RB Inland i 4-RB ) from the feature points (P) about the joints 1-RB , P 2-RB ) can be detected.

[0143] The processor (140) can recognize a part of a hand by comparing the feature points detected from each image frame acquired through multiple different cameras and identifying the corresponding feature points. For example, the processor (140) can recognize a first image frame (i) acquired by the right-middle camera (110RM) at a first time point (t1). 1-RM ) detected from the first joint feature point (P 1-RM ) and the first image frame (i) acquired by the lower right camera (110RB) 1-RB ) detected from the first joint feature point (P 1-RB ) by comparing the first joint feature point (P 1-RM , P 1-RB ) can recognize that it is a hand part. Similarly, the processor (140) can recognize the first image frame (i) acquired by the right-middle camera (110RM) at time point (t1). 1-RM ) detected from the second joint feature point (P2-RM ) and the first image frame (i) acquired by the lower right camera (110RB) 1-RB ) detected from the second joint feature point (P 2-RB ) by comparing the second joint feature point (P 2-RM , P 2-RB ) can be recognized as the arm part.

[0144] The processor (140) can identify spatial correlations between feature points based on parts of a hand recognized through a plurality of image frames. In one embodiment of the present disclosure, the processor (140) can obtain spatial correlations based on information about the positional relationship between feature points recognized from a plurality of image frames acquired by each of different cameras (110RM, 110RB) and the arrangement relationship of the cameras (110RM, 110RB) on the augmented reality device (100).

[0145] Although only the right-middle camera (110RM) and the right-bottom camera (110RB) are illustrated in FIG. 10, this is for convenience of explanation and the present disclosure is not limited thereto. Similarly to the aforementioned method, a method of acquiring multiple image frames, detecting joint features from the multiple image frames, and acquiring spatial correlations between the detected joint features can be applied to the left-middle camera and the left-bottom camera.

[0146] The processor (140) can identify temporal correlations between feature points through a plurality of image frames acquired over time. FIG. 11 illustrates an augmented reality device (100) according to an embodiment of the present disclosure, wherein the augmented reality device (100) acquires a plurality of image frames (i -RM , i -RB ) detected joint features (P 1-RM , P 2-RM , P 1-RM , P 2-RB) is a diagram illustrating a method for identifying temporal and spatial correlations. Referring to FIG. 11, the processor (140) acquires an image frame (i) obtained through the right-center camera (110RM). -RM ) detected from the first joint feature point (P 1-RM ) and the corresponding feature point (P 1-RB ) image frame (i) acquired through the lower right camera (110RB) -RB ) and detects feature points (P 1-RM, P 1-RB ) can calculate the time difference between them. Similarly, the processor (140) can calculate the image frame (i) obtained through the right-center camera (110RM). -RM ) detected from the second joint feature point (P 2-RM ) and the image frame (i) acquired through the lower right camera (110RB) -RB ) from the second joint feature point (P 2-RB ) and detect feature points (P 2-RM, P 2-RB ) can calculate the time difference between the feature points. The processor (140) can identify the temporal correlation between the feature points based on the calculated time difference.

[0147] Referring again to FIG. 8, in step S830, the augmented reality device (100) recognizes a hand-raising motion based on the identified temporal and spatial correlations. Referring also to FIG. 10, the processor (140) of the augmented reality device (100) recognizes feature points (P) that change over time. 1-RM, P 2-RM, P 1-RB , P 2-RB ) location and feature points (P 1-RM, P 2-RM, P 1-RB , P 2-RB ) can recognize the hand-raising action based on the temporal and spatial correlation between them.

[0148] FIG. 12 is a flowchart illustrating a method for recognizing a hand-raising motion based on depth value information of a hand by an augmented reality device (100) according to one embodiment of the present disclosure.

[0149] FIG. 13 is a diagram illustrating a method for an augmented reality device (100) according to one embodiment of the present disclosure to recognize a hand-raising motion based on depth value information of the hand.

[0150] Hereinafter, with reference to FIGS. 12 and 13, a detailed description will be given of a function and / or operation of an augmented reality device (100) according to one embodiment of the present disclosure for recognizing a hand-raising motion based on depth value information of the hand.

[0151] Step S1210 of FIG. 12 is a step that embodies step S210 illustrated in FIG. 2. In step S1210, the augmented reality device (100) continuously captures a user's hand using a depth camera to acquire multiple image frames of the hand. In one embodiment of the present disclosure, the vision sensor (110, see FIGS. 1 and 4) may include a depth camera that acquires a depth value of an object. The depth camera may include, but is not limited to, at least one of a stereo camera, a time of flight (ToF) camera, and an infrared (IR) camera, for example.

[0152] In step S1220, the augmented reality device (100) obtains a depth value of the hand from the acquired plurality of image frames. In one embodiment of the present disclosure, the augmented reality device (100) may capture a user's body using a depth camera and obtain depth value information according to a body part. Referring to the embodiment illustrated in FIG. 13, a user (10) wearing the augmented reality device (100) may take a steady posture at a first time point (t=N) and may perform a hand-raising action at a second time point (t=N+1). The processor (140, see FIG. 4) of the augmented reality device (100) may capture the user's (10) body using a depth camera and obtain a depth value for the body part. For example, the depth value of the hand part among the body parts of the user (10) may be 66, and the depth values ​​of the arm part may be 38, 44, 50, 56, and 62. The depth value illustrated in Fig. 13 is a relative value indicating a depth value calculated based on the position of the depth camera, and the size of the value is proportional to the distance from the augmented reality device (100). For example, the depth value of the upper chest of the user's (10) body part is 5 to 8, and the value increases as it goes toward the stomach area.

[0153] In step S1230, the augmented reality device (100) recognizes a change in the depth value over time. Referring to the embodiment illustrated in FIG. 13, at a second time point (t=N+1), the user (10) makes a hand-raising motion, and accordingly, the depth value of the hand may be changed. For example, at a first time point (t=N), the depth value of the hand is 66, and the depth values ​​of the arm part are 38, 44, 50, 56, and 62. However, due to the hand-raising motion, the depth values ​​of the hand at the second time point (t=N+1) are changed to 30, 32, and 35, and the depth values ​​of the arm part are also changed to 33, 35, 36, and 40. The processor (140) of the augmented reality device (100) may recognize that the depth values ​​of the hand and a portion of the arm are changed.

[0154] In step S1240, the augmented reality device (100) recognizes a hand-raising action based on a change in the depth value. Referring to the embodiment illustrated in FIG. 13, the processor (140) can recognize a change in the depth values ​​of the hand and arm, and recognize a hand-raising action of the user (10) based on the change in the depth value. For example, if the depth value of the hand decreases below a preset value, the processor (140) can recognize that a hand-raising action has been taken by the user (10).

[0155] After the function and / or operation by step S1240 is performed, step S230 illustrated in FIG. 2 may be performed. Referring also to FIG. 13, when a touch input on the touch interface (160) is detected, the depth value of the hand may be changed to 0. The processor (140) of the augmented reality device (100) may detect a touch input when the depth value of the hand is changed to 0. If a touch input is detected within a preset time from the time when the hand-raising motion is recognized, the processor (140) may determine the touch input as an authorized input by the user's intention. If the touch input is determined to be an authorized input, the processor (140) may perform an interaction corresponding to the touch input.

[0156] If a touch input is detected after a preset time has elapsed from the time a hand-raising motion is recognized, the processor (140) may determine the touch input as an unintentional and unauthorized input. If the touch input is determined to be an unintentional and unauthorized input, the processor (140) ignores the touch input and terminates without performing a function or operation.

[0157] FIG. 14 is a flowchart illustrating a method in which an augmented reality device (100) according to one embodiment of the present disclosure determines whether a touch input is a valid input based on sensing data received from a wearable device, and performs an interaction based on the determination result.

[0158] FIG. 15 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to determine whether a touch input is a valid input based on sensing data received from a wearable device (200).

[0159] Hereinafter, with reference to FIGS. 14 and 15, a detailed description will be given of a function and / or operation of an augmented reality device (100) according to one embodiment of the present disclosure for determining whether a touch input is a valid input based on sensing data received from a wearable device (200) and performing an interaction based on the determination result.

[0160] Steps S1410 and S1420 illustrated in FIG. 14 are steps that embody step S210 illustrated in FIG. 2.

[0161] In step S1410, the augmented reality device (100) receives sensing data from a sensor included in a wearable device worn on the user's hand. In the present disclosure, a 'wearable device' is a device worn on a part of the user's body and carried in a worn state, and may be, for example, a smart watch worn on the user's hand. Referring to the embodiment illustrated in FIG. 15, the wearable device (200) may be a smart watch worn on the wrist of the user (10). However, the wearable device (200) is not limited thereto, and may include, for example, a smart ring, a bracelet, an anklet, a necklace, a contact lens, a clothing-integrated device (e.g., electronic clothing), a body-attached device (e.g., a skin pad), or a bio-implantable device (e.g., an implantable circuit).

[0162] In one embodiment of the present disclosure, the wearable device (200) may include a UWB communication module or a Bluetooth communication module. The augmented reality device (100) includes a communication interface (170, see FIG. 4), and the processor (140, see FIG. 4) may receive a UWB signal or a Bluetooth signal from the wearable device (200) through the communication interface (170).

[0163] In step S1410, the augmented reality device (100) obtains a relative positional relationship between the user's hand and the augmented reality device (100) based on the received sensing data. In one embodiment of the present disclosure, the 'relative positional relationship' may include at least one of a distance, a direction, and an orientation between the user's hand and the augmented reality device (100).

[0164] In one embodiment of the present disclosure, the communication interface (170) of the augmented reality device (100) may include an UWB communication module. The 'UWB (Ultra Wide Band) communication module' is a communication module that performs data transmission and reception using an ultra-wideband frequency band between 3.1 GHz and 10.6 GHz. The UWB communication module can transmit and receive data at a maximum speed of 500 Mbps. In one embodiment of the present disclosure, the UWB communication module can receive location information from a wearable device using an ultra-wideband frequency. For example, the processor (140, see FIG. 4) of the augmented reality device (100) may perform ranging using either a single-sided two-way ranging (SS-TWR) or a double-sided two-way ranging (DS-TWR). In one embodiment of the present disclosure, the processor (140) may transmit a ranging request message (Poll message) to the wearable device using a plurality of UWB antenna elements included in the UWB communication module, and receive a response message (Response message) from the wearable device in response to the ranging request signal. The processor (140) may obtain location information of the wearable device through a Time of Arrival (TOA) or Time Difference of Arrival (TDOA) method that utilizes the time difference between the ranging request message and the response message. Referring also to the embodiment illustrated in FIG. 15, the processor (140) may obtain ranging information (Ranging), which is information about the relative distance between the wearable device (200) worn on the hand of the user (10) and the augmented reality device (100), and AoA information (Angle of Arrival), which is direction information of the wearable device.

[0165] In one embodiment of the present disclosure, the communication interface (170) of the augmented reality device (100) includes a Bluetooth communication module and can establish (enable) pairing with a wearable device through the Bluetooth communication module. The processor (140) of the augmented reality device (100) can receive a Bluetooth signal from the wearable device through the communication interface (170) and obtain BLE (Bluetooth Low Energy) location information from the received Bluetooth signal. Based on the obtained BLE location information, the processor (140) can obtain a relative location relationship between the user's hand and the augmented reality device (100).

[0166] Although not shown in FIG. 14, step S220 shown in FIG. 2 may be performed after step S1420 is performed.

[0167] Steps S1430 to S1460 of FIG. 14 are steps that specify step S230 illustrated in FIG. 2. In step S1430, the augmented reality device (100) identifies whether the user's hand is positioned within a preset acceptable region based on the relative positional relationship. Referring also to the embodiment illustrated in FIG. 15, the 'acceptable region (1500)' refers to a position of a hand where a touch input can be recognized as an input authorized by the user's intention, and may represent, for example, an area within a preset distance or angle range from the augmented reality device (100). The acceptable region (1500) may be preset as a user input or a default value at the time of product shipment. The acceptable region (1500) may refer to, for example, an area within 5 centimeters (cm) based on the augmented reality device (100). However, the present invention is not limited thereto.

[0168] The processor (140) of the augmented reality device (100) can determine whether the user's hand is in an acceptable position based on at least one of a relative positional relationship, that is, a distance, a direction, and a bearing between the user's hand and the augmented reality device (100). Referring to the embodiment illustrated in FIG. 15, a user (10) wearing the augmented reality device (100) may assume a steady posture at a first time point (t=N) and then perform a hand-raising action at a second time point (t=N+1). At the first time point (t=N), the distance between the augmented reality device (100) and the wearable device (200) may be greater than a threshold distance, and the angle may not be equal to the threshold angle. Even if the user (10) performs the hand-raising action at the second time point (t=N+1), the position of the user's hand identified by the wearable device (200) may be outside the acceptable area (1500). At a third point in time (t = N + total time), a touch input to the touch interface (160) is detected, and accordingly, the distance between the hand of the user (10) wearing the wearable device (200) and the augmented reality device (100) is changed to less than a threshold distance, and the angle may also be reduced to less than a threshold angle. Based on the relative positional relationship between the augmented reality device (100) and the wearable device (200) at the third point in time (t = N + total time), the processor (140) may identify that the user's hand is located within the allowable area (1500).

[0169] In step S1440, the augmented reality device (100) determines whether the user's hand is within an acceptable area.

[0170] If it is determined that the hand is located within the acceptable region (step S1450), the augmented reality device (100) determines the touch input as a valid input. In one embodiment of the present disclosure, if it is determined that the hand is located within the acceptable region, the augmented reality device (100) may determine that the touch input is an authorized input based on the user's intention. FIG. 16A is a diagram illustrating an example in which a wearable device (200) is located within an acceptable region. Referring to FIG. 16A together, the distance between the hand of the user (10) wearing the wearable device (200) and the augmented reality device (100) may be less than a threshold distance, and the angle between the wearable device (200) and the augmented reality device (100) may be less than a threshold angle. In this case, the processor (140) may determine the touch input to the touch interface (160) as an authorized input.

[0171] If the touch input is determined to be a valid input (step S320), the augmented reality device (100) performs a function or operation corresponding to the touch input.

[0172] If it is determined that the hand is outside the allowable area (step S1460), the augmented reality device (100) determines the touch input as an invalid input. In one embodiment of the present disclosure, if it is determined that the hand is outside the allowable area, the augmented reality device (100) may determine that the touch input is an unintentional and unauthorized input. FIG. 16B is a diagram illustrating an example in which the wearable device (200) is outside the allowable area. Referring also to FIG. 16B, the distance between the wearable device (200) worn on the hand and the augmented reality device (100) may exceed the size of the threshold distance due to the motion of the user (10) raising the hand above the head. In addition, the angle between the wearable device (200) and the augmented reality device (100) may not be equal to the threshold angle. In this case, the processor (140) may determine that the touch input to the touch interface (160) is an unauthorized input unrelated to the user's intention.

[0173] If the touch input is determined to be an invalid input (step S330), the augmented reality device (100) ignores the touch input and terminates without performing a function or operation.

[0174] FIG. 17 is a flowchart illustrating a method for determining whether a touch input is an input applied by a user's intention based on EEG (electroencephalogram) signal data acquired using a brainwave sensor by an augmented reality device (100) according to one embodiment of the present disclosure, and performing an interaction based on the determination result.

[0175] Step S1710 of FIG. 17 is a step that embodies step S210 illustrated in FIG. 2. In step S1710, the augmented reality device (100) senses potential fluctuations due to brainwaves using an EEG (electroencephalogram) sensor to obtain EEG signal data. In one embodiment of the present disclosure, the augmented reality device (100) includes an EEG sensor (120, see FIG. 4), and the EEG sensor (120) may include an EEG sensor configured to sense potential fluctuations due to brainwaves to obtain EEG (electroencephalogram) signal data. The processor (140, see FIG. 4) of the augmented reality device (100) may sense potential fluctuations due to brainwaves of the user's head using the EEG sensor (120) to obtain EEG signal data.

[0176] Steps S1720 to S1750 are steps that embody step S230 illustrated in FIG. 2.

[0177] In step S1720, the augmented reality device (100) identifies negative feedback of brain wave potential based on EEG signal data. In one embodiment of the present disclosure, the processor (140) monitors potential fluctuations caused by brain waves of the user's head based on EEG signal data acquired through the brain wave sensor (120), thereby detecting an event-related potential (ERP) component including at least one of a feedback-related negativity (ERP-FRN) component and a feedback-related positivity (ERP-FRP) component from the EEG signal data. Here, the 'feedback-related negativity' is a feedback indicating an error or anomaly regarding a specific event (e.g., a stimulus or user input), and may include, for example, an error-related negativity (ERN). In one embodiment of the present disclosure, the processor (140) can monitor EEG signal data acquired over time to identify error-related negative potentials (ERNs).

[0178] Although not shown in FIG. 17, step S220 shown in FIG. 2 may be performed after step S1720 is performed.

[0179] In step S1730, the augmented reality device (100) determines whether an error-related negative potential (ERN) is identified within a preset time period from the time a touch input is received. In the present disclosure, the "preset time period" may be, for example, 50 milliseconds (ms) or more and 100 ms or less from the time a touch input is detected. However, the present disclosure is not limited thereto.

[0180] In one embodiment of the present disclosure, when a user's touch input is received through a touch interface (160, see FIG. 4), the processor (140) of the augmented reality device (100) may not perform an interaction corresponding to the touch input for a preset period of time, for example, 50 ms to 100 ms, and may not output a graphical user interface (UI) related to the interaction. After the user provides a touch input to the touch interface, the user expects a change in the output. However, if the expected result (e.g., a change in the output) and the actual result (e.g., no interaction is performed for the preset period of time) do not match, the user becomes confused, and in this case, negative feedback may be detected in the electroencephalographic potential signal. The negative feedback may include, for example, an error-related negative potential (ERN). If the touch input is an unintended input, the user's brain does not expect a change in the current output. In this case, negative feedback is not identified in the EEG signal data.

[0181] An embodiment in which negative feedback, for example, an error related negative potential (ERN), is identified is described with reference to FIG. 18.

[0182] Fig. 18 is a diagram illustrating an example of EEG signal data (1800) indicating a change in brain wave potential when a touch input is detected. Referring to the EEG signal data (1800) illustrated in Fig. 18, a correct response negativity (CRN) (1810) signal does not fluctuate significantly for a preset period of time from a time point (t0) when a touch input is received from a user, whereas an error-related negativity (ERN) (1820) signal increases significantly in value. Between a time point (t1) when a preset period of time, for example, 100 ms, has elapsed from a time point (t0) when a touch input is received, the error-related negativity (1820) may fluctuate more and have a larger value than the negativity-related response (1810). In the present disclosure, 'error-related negative potential (1820)' is signal data indicating a change in brain wave potential that occurs when a user provides a touch input to a touch interface and then expects an interaction to be performed or a related graphic UI to be output, but the interaction or graphic UI is not output.

[0183] Referring again to FIG. 17, if an error-related negative potential (ERN) is identified within a preset time from the time point at which a touch input is detected (step S1740), the augmented reality device (100) determines the touch input as a valid input. Referring also to FIG. 18, since an error-related negative potential (1820) is identified within a preset time from the time point at which a touch input is received (t0), the processor (140) can determine the touch input as an input applied by the user's intention.

[0184] If it is determined to be a valid input (step S320), the augmented reality device (100) performs a function or operation corresponding to the touch input.

[0185] If an error related negative potential (ERN) is identified or no error related negative potential (ERN) is identified after a preset time has elapsed from the time when a touch input is detected (step S1750), the augmented reality device (100) determines the touch input as an invalid input. In one embodiment of the present disclosure, if an error related negative potential (ERN) is identified or no error related negative potential (ERN) is identified after a preset time has elapsed from the time when a touch input is detected, the augmented reality device (100) may determine that the touch input is an unintentional and unauthorized input. If the touch input is determined to be an invalid input (step S330), the augmented reality device (100) ignores the touch input and terminates without performing a function or operation.

[0186] FIG. 19 is a flowchart illustrating a method in which an augmented reality device (100) according to one embodiment of the present disclosure determines whether a touch input is an input authorized by a user's intention based on motion information, and performs an interaction based on the determination result.

[0187] Steps S1910 and S1920 of FIG. 19 are steps that embody step S210 illustrated in FIG. 2.

[0188] In step S1910, the augmented reality device (100) acquires motion information of the augmented reality device using a motion sensor (130, see FIG. 4). In the present disclosure, the 'motion sensor (130)' is a sensor configured to sense motion information regarding the movement of the device. In one embodiment of the present disclosure, when a user's input or an unauthorized external person's input is applied to the augmented reality device (100), the processor (140, see FIG. 4) can sense the vibration or movement of the augmented reality device (100) through the motion sensor (130) to acquire motion information.

[0189] In step S1920, the augmented reality device (100) acquires vibration or movement information of the augmented reality device by the user's adjusting input based on motion information. In the present disclosure, 'adjusting input' refers to a user input that changes setting information or options of the augmented reality device (100). The adjusting input may be different in strength and pattern from a touch input that touches or taps the touch interface (160). In one embodiment of the present disclosure, the processor (140) of the augmented reality device (100) detects the type of motion happing of sensing data acquired from the motion sensor (130), and determines whether the input received from the user is an adjusting input or a touch input based on the type of motion happing. The determination method will be described with reference to FIGS. 20A and 20B .

[0190] FIG. 20a is a diagram illustrating an example of motion sensing data (2000a) when a touch input is received.

[0191] Referring to the motion sensing data (2000a) illustrated in FIG. 20a, when a touch input is received from a user, vibration having an impulse pattern is detected for a relatively short period of time.

[0192] FIG. 20b is a diagram illustrating an example of motion sensing data (2000b) when an adjusting input is received.

[0193] Referring to the motion sensing data (2000b) illustrated in FIG. 20b, unlike when a touch input is received, the sensing data has a pattern that is continuously detected over a relatively long period of time, and the magnitude of the vibration also tends to be larger than the vibration caused by the touch input.

[0194] Referring again to FIG. 19, although not shown in the drawing, step S220 illustrated in FIG. 2 may be performed after step S1920 is performed.

[0195] Steps S1930 to S1950 of FIG. 19 are steps that embody step S230 illustrated in FIG. 2. In step S1930, the augmented reality device (100) determines whether vibration or movement information is acquired after a touch input is detected.

[0196] If vibration or movement information is not acquired after a touch input is detected (step S1940), the augmented reality device (100) determines that a touch input, not a user's adjustment input, has been received and determines the touch input as a valid input. If the touch input is determined to be a valid input (step S320), the augmented reality device (100) performs a function or operation corresponding to the touch input.

[0197] If vibration or movement information is acquired after a touch input is detected (step S1940), the augmented reality device (100) determines that a user's adjustment input has been received and determines the touch input as an invalid input. In one embodiment of the present disclosure, if vibration or movement information is acquired after a touch input is detected, the augmented reality device (100) determines that a user's adjustment input has been received and may determine that the touch input is an unintentional and unauthorized input. If the touch input is determined to be an invalid input (step S330), the augmented reality device (100) ignores the touch input and terminates without performing a function or operation.

[0198] In the embodiments illustrated in FIGS. 19, 20a, and 20b, the augmented reality device (100) may acquire motion sensing data (2000a, 2000b) acquired through the motion sensor (130), analyze the pattern of the motion sensing data (2000a, 2000b), and if it is determined that an adjusting input has been input, determine that the touch input is an unintentional and unauthorized input. Through this, the augmented reality device (100) according to one embodiment of the present disclosure may prevent unintentional interactions from being automatically performed by not performing an interaction corresponding to a related input when a user provides an input such as changing setting information or adjusting an option.

[0199] The present disclosure provides an augmented reality device (100) that determines whether a touch input is an authorized input by a user's intention and performs an interaction based on the determination result. The augmented reality device (100) according to one embodiment of the present disclosure may include at least one sensor, a touch interface (160) configured to receive a touch input, at least one processor (140) including a processing circuit, and a memory (150) that stores one or more instructions. The one or more instructions are individually or collectively executed by the at least one processor (140), whereby the augmented reality device (100) can sense a movement of a hand through the at least one sensor and determine whether a touch input received through the touch interface (160) is a valid input based on the sensed movement of the hand. By individually or collectively executing one or more of the above commands by the at least one processor (140), the augmented reality device (100) can determine whether to perform an interaction corresponding to a touch input based on the judgment result regarding a valid input.

[0200] In one embodiment of the present disclosure, the one or more commands are individually or collectively executed by the at least one processor (140), so that the augmented reality device (100) can ignore the touch input without performing a function or operation corresponding to the touch input when the touch input is determined to be an invalid input.

[0201] In one embodiment of the present disclosure, the at least one sensor may include a vision sensor (110) configured as a camera. By individually or collectively executing the one or more commands by the at least one processor (140), the augmented reality device (100) may acquire a plurality of image frames regarding the hand by continuously photographing the user's hand using the camera, input the acquired plurality of image frames into an artificial intelligence model, and perform vision recognition by the artificial intelligence model to detect feature points regarding joints of the hand from the plurality of image frames. By individually or collectively executing the one or more commands by the at least one processor (140), the augmented reality device (100) may recognize a hand-raising motion based on movement of the detected feature points over time. By individually or collectively executing one or more of the above commands by the at least one processor (140), the augmented reality device (100) can determine the touch input as a valid input when the touch input is detected within a preset time from the time when the hand-raising motion is recognized.

[0202] In one embodiment of the present disclosure, the camera may be disposed at different locations on the augmented reality device (100) and may include a plurality of cameras having different views. By individually or collectively executing the one or more commands by the at least one processor (140), the augmented reality device (100) may detect feature points for each part of the hand from each of the plurality of image frames having different view points captured by each of the plurality of cameras. By individually or collectively executing the one or more commands by the at least one processor (140), the augmented reality device (100) may combine the plurality of image frames to identify temporal and spatial correlations between feature points for each part detected from each of the plurality of image frames. By individually or collectively executing one or more of the above commands by the at least one processor (140), the augmented reality device (100) can recognize a hand-raising motion based on the temporal and spatial correlation between feature points.

[0203] In one embodiment of the present disclosure, the at least one sensor may include a vision sensor (110) configured as a depth camera that acquires a depth value of an object. The one or more commands are individually or collectively executed by the at least one processor (140), so that the augmented reality device (100) can acquire a depth value of a hand from a plurality of image frames continuously acquired through the depth camera, and recognize a change in the acquired depth value over time. The one or more commands are individually or collectively executed by the at least one processor (140), so that the augmented reality device (100) can recognize a hand-raising action based on a change in the depth value.

[0204] In one embodiment of the present disclosure, the augmented reality device (100) may further include a communication interface (170) that performs data communication with an external device. The communication interface (170) may receive the sensing data from a sensor included in a wearable device (200) worn on a user's hand. As the one or more commands are individually or collectively executed by the at least one processor (140), the augmented reality device (100) may obtain relative position relationship information including information regarding at least one of a distance, a direction, and an orientation between the user's hand and the augmented reality device based on the sensing data received from the wearable device (200). By individually or collectively executing the one or more commands mentioned above by the at least one processor (140), the augmented reality device (100) can identify whether the user's hand is located within a preset acceptable region based on the acquired relative position relationship information, and can determine whether the touch input is a valid input based on the identification result.

[0205] In one embodiment of the present disclosure, the wearable device (200) may include at least one of an Ultra Wide Band (UWB) communication module and a Bluetooth communication module. The sensing data may include at least one of AoA (Angle of Arrival) information and BLE (Bluetooth Low Energy) location information based on a UWB signal received from the wearable device (200).

[0206] In one embodiment of the present disclosure, the at least one sensor may include a brainwave sensor (120) configured to sense potential fluctuations due to brainwaves to obtain electroencephalogram (EEG) signal data. The one or more commands may be individually or collectively executed by the at least one processor (140), so that the augmented reality device (100) may sense potential fluctuations due to brainwaves of the user's head using the brainwave sensor (120), thereby obtaining EEG signal data, and identify negative feedback of brainwave potentials based on the obtained EEG signal data. The one or more commands may be individually or collectively executed by the at least one processor (140), so that the augmented reality device (100) may determine whether a touch input is a valid input based on the identification result.

[0207] In one embodiment of the present disclosure, the one or more commands are individually or collectively executed by the at least one processor (140), so that the augmented reality device (100) can identify an error-related negativity (ERN) by monitoring a potential characteristic change of EEG signal data for a preset time from the time at which a touch input is received. The one or more commands are individually or collectively executed by the at least one processor (140), so that the augmented reality device (100) can determine a touch input as a valid input when an error-related negativity is identified.

[0208] In one embodiment of the present disclosure, the at least one sensor may further include a motion sensor (130) that detects movement of the augmented reality device (100). By individually or collectively executing the one or more commands by the at least one processor (140), the augmented reality device (100) may obtain motion information including vibration or movement of the augmented reality device (100) by an adjusting input of a user using the motion sensor (130). By individually or collectively executing the one or more commands by the at least one processor (140), the augmented reality device (100) may determine a touch input as an invalid input when motion information is obtained after a touch input is detected.

[0209] The present disclosure provides a method for an augmented reality device (100) to determine a valid touch input and perform an interaction based on the determination result. An operating method of an augmented reality device (100) according to an embodiment of the present disclosure may include a step (S210) of obtaining sensing data regarding hand movement using at least one sensor. The operating method of the augmented reality device (100) may include a step (S220) of detecting a touch input to a touch interface (160). The operating method of the augmented reality device (100) may include a step (S230) of determining whether a touch input is a valid input based on the obtained sensing data.

[0210] In one embodiment of the present disclosure, the method of operating the augmented reality device (100) may further include a step of ignoring the touch input without performing a function or operation corresponding to the touch input, if the touch input is determined to be an invalid input as a result of the determination regarding the valid input.

[0211] In one embodiment of the present disclosure, the at least one sensor may include a vision sensor (110) configured as a camera. The step of obtaining the sensing data (S210) may include a step of obtaining a plurality of image frames regarding the hand by continuously photographing the user's hand using the camera (S610). The operating method of the augmented reality device (100) may further include a step of inputting the obtained plurality of image frames into an artificial intelligence model and detecting feature points regarding joints of the hand from the plurality of image frames through inferencing using the artificial intelligence model (S620); and a step of recognizing a hand-raising motion based on movement of the detected feature points over time (S630). The step of determining whether the touch input is a valid input (S230) may include a step of determining the touch input as a valid input (S650) if the touch input is detected within a preset time from the time at which the hand-raising motion is recognized.

[0212] In one embodiment of the present disclosure, the camera may be disposed at different locations on the augmented reality device (100) and may include a plurality of cameras having different views. The step of recognizing the hand-raising motion (S630) may include a step of detecting a feature point for each part of the hand from each of a plurality of image frames having different view points captured by each of the plurality of cameras (S810); and a step of identifying temporal and spatial correlations between feature points for each part detected from each of the plurality of image frames by combining the plurality of image frames (S820). The step of recognizing the hand-raising motion (S630) may include a step of recognizing the hand-raising motion based on the temporal and spatial correlations between feature points (S830).

[0213] In one embodiment of the present disclosure, the at least one sensor may include a vision sensor (110) configured as a depth camera that acquires a depth value of an object. The method of operating the augmented reality device (100) may further include a step (S1220) of acquiring a depth value of a hand from a plurality of image frames continuously acquired through the depth camera; and a step (S1230) of recognizing a change in the acquired depth value over time. The step (S220) of acquiring at least one of hand motion information, position information, and biometric information may include a step (S1240) of recognizing a hand-raising motion based on a change in the depth value.

[0214] In one embodiment of the present disclosure, the step of obtaining the sensing data (S210) may include the step of receiving the sensing data from a sensor included in a wearable device (200) worn on a user's hand (S1410). The step of obtaining at least one of hand movement information, position information, and biometric information (S220) may include the step of obtaining relative position relationship information including information regarding at least one of a distance, a direction, and an orientation between the user's hand and the augmented reality device based on the received sensing data (S1420). The step of determining whether the touch input is a valid input (S230) may include the step of identifying whether the user's hand is located within a preset acceptable region based on the acquired relative position relationship information (S1430); and the step of determining whether the touch input is a valid input based on the identification result (S1440).

[0215] In one embodiment of the present disclosure, the at least one sensor may include an EEG sensor configured to acquire electroencephalogram (EEG) signal data by sensing potential fluctuations due to brainwaves. The step of acquiring the sensing data (S210) may include a step of acquiring EEG signal data (S1710) by sensing potential fluctuations due to brainwaves of the user's head using the EEG sensor. The step of determining whether the touch input is a valid input (S230) may include a step of identifying negative feedback of brainwave potentials based on the acquired EEG signal data (S1720); and a step of determining whether the touch input is a valid input based on the identification result (S1730).

[0216] In one embodiment of the present disclosure, the step of identifying negative feedback of the EEG potential (S1720) may include a step of identifying an error-related negativity (ERN) by monitoring a potential characteristic change of EEG signal data for a preset period of time from the time at which the touch input is received. The step of determining whether the touch input is a valid input (S230) may include a step of determining the touch input as a valid input if an error-related negativity is identified.

[0217] In one embodiment of the present disclosure, the step (S210) of acquiring the sensing data may include a step of acquiring motion information regarding the movement of the augmented reality device (100) through a motion sensor (130) and a step (S1910) of acquiring motion information regarding vibration or movement of the augmented reality device by an adjusting input for adjusting the augmented reality device by a user using the motion sensor (130). The step (S230) of determining whether the touch input is a valid input may include a step of determining the touch input as an invalid input when motion information is acquired after the touch input is detected.

[0218] The present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an augmented reality device (100), such that the augmented reality device (100) performs the following operations: acquiring sensing data regarding hand movements using at least one sensor; detecting a touch input to a touch interface (160); and determining whether a touch input is a valid input based on the sensing data regarding hand movements.

[0219] The program executed by the augmented reality device (100) described in the present disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0220] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.

[0221] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The media may be readable by a computer, stored in a memory, and executed by a processor.

[0222] A computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium does not contain signals and is tangible, but does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0223] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0224] A computer program product may include a software program, or a computer-readable storage medium storing the software program. For example, the computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by the manufacturer of the augmented reality device (100) or through an electronic marketplace (e.g., the Samsung Galaxy Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a server of the manufacturer of the augmented reality device (100), a server of an electronic marketplace, or a relay server that temporarily stores the software program.

[0225] The computer program product may include a storage medium of the server or the storage medium of the augmented reality device (100) in a system comprising an augmented reality device (100) and / or a server. Alternatively, if there is a third device (e.g., a wearable device) that is communicatively connected to the augmented reality device (100), the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the augmented reality device (100) to the third device, or from the third device to an electronic device.

[0226] In this case, either the augmented reality device (100) or the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, at least one of the augmented reality device (100) and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0227] For example, the augmented reality device (100) may execute a computer program product stored in a memory (150, see FIG. 4) to control another electronic device that is in communication with the augmented reality device (100) to perform a method according to the disclosed embodiments.

[0228] As another example, a third device may execute a computer program product to control an electronic device in communication with the third device to perform a method according to the disclosed embodiment.

[0229] When a third device executes a computer program product, the third device may download the computer program product from the augmented reality device (100) and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the disclosed embodiments.

[0230] Although the embodiments described above have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

1. At least one sensor; A touch interface (160) configured to receive touch input; At least one processor (140) comprising a processing circuit; and Memory (150) storing one or more instructions Including, The augmented reality device (100) is configured such that the one or more of the above commands are individually or collectively executed by the at least one processor (140). Sense the movement of the hand through at least one sensor above, Based on the sensed hand movement, it is determined whether the touch input received through the touch interface (160) is a valid input, An augmented reality device (100) that determines whether to perform an interaction corresponding to the touch input based on the judgment result regarding the valid input.

2. In paragraph 1, The above at least one sensor comprises a vision sensor (110) comprising a camera, The augmented reality device (100) is configured such that the one or more of the above commands are individually or collectively executed by the at least one processor (140). By continuously photographing the user's hand using the above camera, multiple image frames of the hand are acquired, The above-mentioned acquired multiple image frames are input into an artificial intelligence model, and vision recognition is performed by the artificial intelligence model to detect feature points related to hand joints from the above-mentioned multiple image frames. Recognizes a hand-raising motion based on the movement of the detected feature points over time, An augmented reality device (100) that determines the touch input as a valid input when the touch input is detected within a preset time from the time the hand-raising motion is recognized.

3. In paragraph 1, The above at least one sensor comprises a vision sensor (110) comprising a depth camera that acquires a depth value of an object, The augmented reality device (100) is configured such that the one or more of the above commands are individually or collectively executed by the at least one processor (140). Obtaining the depth value of the hand from multiple image frames continuously acquired through the depth camera, Recognize the change in the acquired depth value over time, An augmented reality device (100) that recognizes a hand-raising motion based on a change in the depth value.

4. In paragraph 1, A communication interface (170) that performs data communication with an external device; Including more, The above communication interface (170) receives sensing data from a sensor included in a wearable device (200) worn on the user's hand, The augmented reality device (100) is configured such that the one or more of the above commands are individually or collectively executed by the at least one processor (140). Based on the received sensing data, relative position relationship information including information about at least one of distance, direction, and orientation between the user's hand and the augmented reality device is obtained, An augmented reality device (100) that identifies whether the user's hand is located within a preset acceptable region based on the acquired relative position relationship information and determines whether the touch input is a valid input based on the identification result.

5. In paragraph 1, The above at least one sensor is a brainwave sensor (120) configured to sense potential fluctuations caused by brainwaves to obtain EEG (electroencephalogram) signal data; Including, The augmented reality device (100) is configured such that the one or more of the above commands are individually or collectively executed by the at least one processor (140). By sensing the potential change due to brain waves of the user's head using the above brain wave sensor (120), EEG signal data is acquired, An augmented reality device (100) that identifies negative feedback of brain wave potential based on the acquired EEG signal data and determines whether the touch input is a valid input based on the identification result.

6. In paragraph 5, The augmented reality device (100) is configured such that the one or more of the above commands are individually or collectively executed by the at least one processor (140). By monitoring the potential characteristic fluctuations of the EEG signal data for a preset period of time from the time the touch input is received, error-related negativity (ERN) is identified, An augmented reality device (100) that determines the touch input as a valid input when the above error-related negative potential is identified.

7. In any one of clauses 1 to 6, At least one of the above sensors is a motion sensor (130) that detects movement of the augmented reality device (100); Including, The augmented reality device (100) is configured such that the one or more of the above commands are individually or collectively executed by the at least one processor (140). By using the above motion sensor (130), motion information including vibration or movement of the augmented reality device (100) is obtained by adjusting input of the user, An augmented reality device (100) that determines the touch input as an invalid input when the motion information is acquired after the touch input is detected.

8. In the operating method of the augmented reality device (100), A step (S210) of obtaining sensing data regarding hand movements using at least one sensor; Step (S220) of detecting a touch input to the touch interface (160) of the augmented reality device (100); and A step (S230) of determining whether the touch input is a valid input based on the acquired sensing data; A method comprising:

9. In paragraph 8, The above at least one sensor comprises a vision sensor (110) comprising a camera, The step (S210) of acquiring the above sensing data is as follows: A step (S610) of obtaining multiple image frames of a hand by continuously photographing the user's hand using the above camera; Including, The operating method of the above augmented reality device (100) is: A step (S620) of inputting the acquired plurality of image frames into an artificial intelligence model and detecting feature points related to hand joints from the plurality of image frames through inferencing using the artificial intelligence model; and A step (S630) of recognizing a hand-raising motion based on the movement of the detected feature points over time; Including more, The step (S230) of determining whether the above touch input is a valid input is: A step (S650) of determining the touch input as a valid input when the touch input is detected within a preset time from the time when the hand-raising motion is recognized; A method comprising:

10. In paragraph 8, The above at least one sensor comprises a vision sensor (110) comprising a depth camera that acquires a depth value of an object, The operating method of the above augmented reality device (100) is: A step (S1220) of obtaining a depth value of a hand from a plurality of image frames continuously acquired through the depth camera; Step (S1230) of recognizing changes in the acquired depth value over time; and Step (S1240) of recognizing a hand-raising motion based on a change in the above depth value; A method further comprising:

11. In paragraph 8, The step (S210) of acquiring the above sensing data is as follows: Step (S1410) of receiving the sensing data from a sensor included in a wearable device (200) worn on the user's hand; and A step (S1420) of obtaining relative position relationship information including information about at least one of distance, direction, and orientation between a user's hand and the augmented reality device based on the received sensing data; Including, The step (S230) of determining whether the above touch input is a valid input is: A step (S1430) of identifying whether the user's hand is located within a preset acceptable region based on the acquired relative position relationship information; and Step (S1440) of determining whether the touch input is a valid input based on the identification result; A method comprising:

12. In paragraph 8, The at least one sensor is an EEG sensor configured to sense potential fluctuations caused by brainwaves to obtain EEG (electroencephalogram) signal data; Including, The step (S210) of acquiring the above sensing data is as follows: A step (S1710) of acquiring EEG signal data by sensing potential fluctuations caused by brain waves of the user's head using the above EEG sensor; Including, The step (S230) of determining whether the above touch input is a valid input is: A step (S1720) of identifying negative feedback of brain wave potential based on the acquired EEG signal data; and Based on the identification result, a step of determining whether the touch input is a valid input (S1730); A method comprising:

13. In paragraph 12, The step (S1720) of identifying the negative feedback of the above brain wave potential is: A step of identifying an error-related negativity (ERN) by monitoring a change in potential characteristics of the EEG signal data for a preset period of time from the time at which the touch input is received; Including, The step (S230) of determining whether the above touch input is a valid input is: A step of determining the touch input as a valid input when the above error-related negative potential is identified; A method comprising:

14. In any one of paragraphs 8 to 13, The step (S210) of acquiring the above sensing data is as follows: A step of obtaining motion information about the movement of the augmented reality device (100) through a motion sensor (130); and A step (S1910) of obtaining motion information regarding vibration or movement of the augmented reality device by an adjusting input for adjusting the augmented reality device by the user using the motion sensor (130); Including, The step (S230) of determining whether the above touch input is a valid input is: A step of determining the touch input as an invalid input when the motion information is acquired after the touch input is detected; A method comprising:

15. In a computer program product including a computer-readable storage medium, The above storage medium, An act of obtaining sensing data regarding hand movements using at least one sensor; An operation of detecting a touch input to a touch interface (160) of the augmented reality device (100); and An operation of determining whether the touch input is a valid input based on the acquired sensing data; A computer program product including instructions executed by an augmented reality device (100) to perform an augmented reality device (100).

Images