Augmented reality device for determining valid touch input by a user and operating method thereof

Abstract

An augmented reality device and its operating method are provided. The augmented reality device determines valid touch input to a touch interface, enabling users to use the device safely and conveniently, and preventing interactions caused by invalid touch input from being automatically executed. The augmented reality device determines whether touch input to the touch interface is valid input caused by user intent based on sensing data about hand movement, and can determine whether to execute an interaction based on the determination result.

Description

Technical Field

[0001] This disclosure relates to an augmented reality device and its operating method for determining valid touch input. More specifically, this disclosure relates to an augmented reality device and its operating method that determines whether a touch input to a touch interface is valid based on sensing data obtained through at least one sensor, and prevents interaction caused by invalid input. Background Technology

[0002] Augmented reality (AR) is a technology used to display virtual images by overlaying virtual images onto the physical environment or real-world objects. AR devices using AR technology (such as smart glasses) are being effectively used in daily life, for example, for searching information, providing directions, and taking photos. In particular, smart glasses are worn as fashion items and are primarily used for outdoor activities.

[0003] AR devices include a touch interface (such as a touchpad, physical button, switch, etc.) for receiving touch or tap input. When receiving touch input from a user, the touch interface performs an interaction corresponding to the touch input. For example, in response to a touch or tap input being received on the touch interface, the AR device identifies the area where the touch was detected, displays a graphical user interface (GUI) (such as a button, menu option, etc.) corresponding to the identified area, or provides feedback (such as vibration or notification sound), and performs a function or operation corresponding to the touch or tap input.

[0004] Unintentional and unauthorized touch input can occur on the touch interface of an AR device. Unauthorized touch input can be from someone other than the user of the AR device. Unintentional touch input can be accidentally provided without any intent from the user.

[0005] Recently, technologies for providing secure and / or personalized access to AR devices have been developed and widely adopted. However, when users wear AR devices, there is no solution to prevent unintentional and unauthorized touch input to the touch interface. Summary of the Invention

[0006] Solution to the problem One aspect of this disclosure provides an augmented reality (AR) device that determines whether a touch input is valid and performs an interaction based on the determination result. According to embodiments of this disclosure, the AR device may include at least one sensor, a touch interface configured to receive touch input, at least one processor including processing circuitry, and a memory storing one or more instructions. The one or more instructions may be executed individually or jointly by the at least one processor to cause the AR device to: detect hand movement via the at least one sensor and determine, based on the detected hand movement, whether the touch input received via the touch interface is valid. The one or more instructions may be executed individually or jointly by the at least one processor to cause the AR device to: determine, based on the determination of the valid input, whether to perform an interaction corresponding to the touch input.

[0007] One aspect of this disclosure provides a method by which an AR device determines valid touch input and performs an interaction based on the determined result. According to embodiments of this disclosure, the operation method of the AR device may include: acquiring sensing data about hand movement using at least one sensor. The operation method of the AR device may include: detecting touch input to a touch interface. The operation method of the AR device may include: determining whether the touch input is valid based on the acquired sensing data.

[0008] Another aspect of this disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an AR device to cause the AR device to perform the following operations: acquiring sensing data about hand movement using at least one sensor; detecting touch input to a touch interface; and determining whether the touch input is a valid input based on the sensing data about hand movement. Attached Figure Description

[0009] This disclosure will be readily understood through the combination of the following detailed description and the accompanying drawings, and reference numerals refer to structural elements.

[0010] Figure 1 This is a diagram illustrating an augmented reality (AR) device that determines whether a touch input is a valid input and performs an operation based on the determination result, according to embodiments of the present disclosure.

[0011] Figure 2 This is a flowchart illustrating a method for determining whether a touch input is a valid input via its AR device according to an embodiment of the present disclosure.

[0012] Figure 3 This is a flowchart illustrating a method for performing an interaction via its AR device based on whether the touch input is a valid input, according to an embodiment of the present disclosure.

[0013] Figure 4 This is a block diagram illustrating the components of an AR device according to an embodiment of the present disclosure.

[0014] Figure 5 This illustrates how an AR device according to an embodiment of the present disclosure detects a user's hand-raising motion.

[0015] Figure 6 This is a flowchart illustrating a method, according to an embodiment of the present disclosure, for determining whether a touch input is valid based on a user's hand-raising motion using its AR device.

[0016] Figure 7 A method for detecting feature points of hand joints from image frames using an AI model via its AR device, according to embodiments of the present disclosure, is illustrated.

[0017] Figure 8 This is a flowchart illustrating a method for identifying hand-raising movements based on the temporal and spatial correlation of joint feature points using an AR device according to an embodiment of the present disclosure.

[0018] Figure 9a An arrangement of multiple cameras mounted on an AR device according to an embodiment of the present disclosure is shown.

[0019] Figure 9b An arrangement of multiple cameras mounted on an AR device according to an embodiment of the present disclosure is shown.

[0020] Figure 10 This illustration shows how an AR device according to an embodiment of the present disclosure can identify a hand-raising motion by using multiple image frames obtained via multiple cameras.

[0021] Figure 11 This illustrates how an AR device according to embodiments of the present disclosure identifies the temporal and spatial correlations of joint feature points detected from multiple image frames.

[0022] Figure 12 This is a flowchart illustrating a method for identifying hand-raising movements based on hand depth value information using an AR device according to an embodiment of the present disclosure.

[0023] Figure 13 This illustrates how an AR device according to an embodiment of the present disclosure can recognize a hand-raising motion based on hand depth value information.

[0024] Figure 14 This is a flowchart illustrating a method, according to an embodiment of the present disclosure, for determining whether a touch input is valid based on sensing data received from a wearable device using its AR device and performing an interaction based on the determination result.

[0025] Figure 15This illustrates how an AR device according to an embodiment of the present disclosure determines whether a touch input is a valid input based on sensing data received from a wearable device.

[0026] Figure 16a An example of a wearable device located in an acceptable area is shown.

[0027] Figure 16b Examples of wearable devices outside of acceptable areas are shown.

[0028] Figure 17 This is a flowchart illustrating a method, according to an embodiment of the present disclosure, for determining whether a touch input is valid based on electroencephalogram (EEG) signal data obtained by using an electroencephalogram sensor and performing an interaction based on the determined result using an AR device.

[0029] Figure 18 An example of EEG signal data representing the potential fluctuations of brain waves when touch input is detected is shown.

[0030] Figure 19 This is a flowchart illustrating a method, according to an embodiment of the present disclosure, for determining whether a touch input is valid based on motion information using its AR device and performing an interaction based on the determined result.

[0031] Figure 20a This shows an example of motion sensing data when a touch input is received.

[0032] Figure 20b This shows an example of motion sensing data when an adjustment input is received. Detailed Implementation

[0033] In consideration of the principles of this disclosure, terms are selected from commonly used terms that are currently widely available; however, these terms may depend on the intent of those skilled in the art, judicial precedent, the emergence of new technologies, etc. Some terms used herein were chosen at the applicant's discretion, and in such cases, these terms will be described in detail later in conjunction with embodiments of this disclosure. Therefore, terms should be defined based on their meaning and description throughout this disclosure.

[0034] As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. All terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0035] The terms “comprising” or “including” are inclusive or open-ended and do not exclude additional, unlisted elements or method steps. As used herein, the terms “unit,” “module,” “block,” etc., each refer to a unit for performing at least one function or operation and may be implemented in hardware, software, or a combination thereof.

[0036] In this disclosure, the expression “configured as” as used herein may be used interchangeably with “suitable for,” “capable of,” “designed for,” “suitable for,” “manufactured as,” or “capable”, depending on the given context. The expression “configured as” may not necessarily mean “specifically designed for” in hardware terms. For example, in some cases, the expression “system configured to do something” may refer to “an entity capable of cooperating with another device or component to do something.” For example, “processor configured to perform functions A, B, and C” may refer to a dedicated processor (e.g., an embedded processor for performing functions A, B, and C) or a general-purpose processor (e.g., a central processing unit (CPU) or application processor that performs functions A, B, and C by executing one or more software programs stored in memory).

[0037] When the terms “connection” or “coupling” are used, a component may be directly connected to or coupled to another component. However, unless otherwise defined, it will be understood that a component may be indirectly connected to or coupled to another component via a new component.

[0038] In this disclosure, augmented reality (AR) refers to displaying virtual images or both real objects and virtual images in a physical environment in the real world.

[0039] In this disclosure, an AR device is a device capable of representing augmented reality, which may be, for example, not only AR glasses shaped like glasses worn on a user's face, but also head-mounted display devices (HMDs) or AR helmets worn on the head.

[0040] In embodiments of this disclosure, the AR device may be replaced by a virtual reality (VR) device.

[0041] In this disclosure, valid input refers to input that has the effect of causing the AR device to perform an interaction, including associated functions and / or operations, based on the received touch input. In this disclosure, the AR device can determine whether touch input to a touch interface is valid input. In embodiments of this disclosure, valid input may include authorized input.

[0042] In this disclosure, authorized input may refer to input intentionally provided by an authorized user of the AR device to enable the AR device to perform an interaction (e.g., a function or operation) corresponding to input via a touch or tap touch interface.

[0043] The artificial intelligence (AI) related functions in this disclosure are operated via a processor and memory. The processor may be configured with one or more processors. The one or more processors may include general-purpose processors (such as CPUs, application processors, digital signal processors (DSPs), etc.), dedicated graphics processors (such as GPUs and vision processing units (VPUs)), or dedicated AI processors (such as NPUs). The one or more processors control the processing of input data according to predefined operating rules or AI models stored in memory. When the one or more processors are dedicated AI processors, the dedicated AI processors may be designed in a hardware architecture specifically designed to process a particular AI model.

[0044] The characteristic of a predefined operating rule or AI model is that it is created through learning. Specifically, an AI model created through learning refers to a predefined operating rule or AI model built to perform desired features (or objects) when a basic AI model is trained using a learning algorithm with a large amount of training data. This learning can be performed by the device itself, in which AI is performed according to this disclosure, or by a separate server and / or system. Examples of learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0045] In this disclosure, the AI ​​model may consist of multiple neural network layers. Each of the multiple neural network layers may have multiple weight values, and neural network operations are performed through operations between the results of operations of the previous layer and the multiple weight values. The multiple weight values ​​possessed by the multiple neural network layers can be optimized through the learning results of the AI ​​model. For example, multiple weight values ​​can be updated to reduce or minimize the loss or cost values ​​obtained by the AI ​​model during the training process. The artificial neural network model may include, but is not limited to, deep neural networks (DNNs), such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), or deep Q-networks.

[0046] In this disclosure, "visual recognition" refers to image signal processing that inputs an image into an AI model and detects objects from the input image, classifies objects into specific categories, or performs object segmentation from an input image using reasoning from an AI model. In embodiments of this disclosure, visual recognition may refer to image processing that uses an AI model to identify a user's hand from an image captured by a visual sensor (e.g., a camera) and obtain positional information including multiple feature points (e.g., joints) on the hand.

[0047] In this disclosure, a wearable device is a device worn on a part of a user's body and carried by the user. For example, a wearable device may be at least one of, but is not limited to, a smartwatch, a ring, a bracelet, an anklet, a necklace, a contact lens, a garment integration device (e.g., electronic clothing), a body attachment device (e.g., a skin patch), or a bio-implantable device (e.g., an implantable circuit).

[0048] Embodiments of this disclosure will now be described in detail with reference to the accompanying drawings to enable them to be readily practiced by those skilled in the art. However, embodiments of this disclosure may be implemented in many different forms and are not limited to those discussed herein.

[0049] Embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0050] Figure 1 This is a diagram illustrating an augmented reality (AR) device that determines whether a touch input 20 is a valid input and performs an operation based on the determination result according to embodiments of the present disclosure.

[0051] AR device 100 is a device capable of representing augmented reality, which can be implemented as a head-mounted display (HMD) or AR helmet worn on a user's head. Although for ease of illustration, AR device 100 is described as... Figure 1 The device is shown as an HMD, but it is not limited to this. For example, the AR device 100 can be configured as AR glasses in the form of glasses worn on the user's face.

[0052] Reference Figure 1 The AR device 100 may include a vision sensor 110 and a touch interface 160. The vision sensor 110 may include multiple cameras 110RT, 110RB, 110LT, and 110LB. Figure 1 In the illustrated embodiment, the vision sensor 110 may include an upper left camera 110LT and a lower left camera 110LB disposed at the upper and lower ends of the frame surrounding the left eye lens of the AR device 100, and an upper right camera 110RT and a lower right camera 110RB disposed at the upper and lower ends of the frame surrounding the right eye lens. The number and positions of the plurality of cameras 110RT, 110RB, 110LT, and 110LB included in the vision sensor 110 are merely examples and are not limited to this embodiment. Figure 1 The quantity and location are shown. In embodiments of this disclosure, the AR device 100 may include at least two (e.g., three, five, six, ..., n) cameras.

[0053] Touch interface 160 is a hardware input device configured to receive touch or tap input from a user or from an external source. In embodiments of this disclosure, touch interface 160 may include a touchpad, a touchscreen, a physical key button, or a switch. Figure 1 In the embodiment shown, the touch interface 160 may be located on one side of the AR device 100, but is not limited thereto.

[0054] Despite Figure 1 Not shown, but in addition to the visual sensor 110 and touch interface 160, the AR device 100 may also include other components. In embodiments of this disclosure, the AR device 100 may also include an electroencephalogram (EEG) sensor 120 (see [link to documentation]). Figure 4 ), motion sensor 130 (see Figure 4 ) and communication interface 160 (see Figure 4 (will be combined) Figure 4 The components of AR device 100 are described in detail.

[0055] In operation ①, the AR device 100 obtains sensing data about hand movement by using at least one sensor, including a vision sensor 110.

[0056] In operation ②, the AR device 100 detects touch input 20 from the user or from the outside to the touch interface 160.

[0057] In operation ③, the AR device 100 determines whether the touch input 20 is a valid input based on sensing data about hand movement.

[0058] When touch input 20 is determined to be a valid input, in operation ④-1, AR device 100 performs the interaction (function or operation) corresponding to touch input 20.

[0059] When touch input 20 is determined to be invalid, in operation ④-2, AR device 100 ignores touch input 20 and does not perform any function or operation.

[0060] Now we will combine Figure 2 and Figure 3 Reference Figure 1 The AR device 100 is described in detail for determining whether the touch input 20 is a valid input and performing interactive functions and / or operations based on the determination result.

[0061] Figure 2 This is a flowchart illustrating a method for determining whether a touch input is a valid input using its AR device 100 according to an embodiment of the present disclosure.

[0062] In operation S210, the AR device 100 acquires sensing data about hand movement using at least one sensor. See also... Figure 1 AR device 100 may include visual sensor 110 (see Figure 1), and by utilizing multiple cameras 110RT, 110RB, 110LT and 110LB included in the vision sensor 110 (see Figure 1 The AR device 100 captures multiple image frames by photographing the user's hand. However, it is not limited to this, and in embodiments of this disclosure, the AR device 100 may include an electroencephalogram (EEG) sensor 120 (see [link to documentation]). Figure 4 The AR device 100 obtains electroencephalogram (EEG) signal data by sensing potential fluctuations in brain waves from the user's head using an EEG sensor 120. In embodiments of this disclosure, the AR device 100 may also include a communication interface 170 (see...). Figure 4 The AR device 100 receives ultra-wideband (UWB) signals or Bluetooth signals from a wearable device worn on the user's hand via communication interface 170. For example, the AR device 100 may receive angle of arrival (AoA) information from a UWB signal or Bluetooth Low Energy (BLE) location information from a Bluetooth signal. However, it is not limited to these.

[0063] In embodiments of this disclosure, the AR device 100 may include a motion sensor 130 (see [link to documentation]). Figure 4 Furthermore, motion information about the vibration or movement of the AR device 100 is obtained through the motion sensor 130.

[0064] In embodiments of this disclosure, AR device 100 can input multiple image frames obtained through multiple cameras 110RT, 110RB, 110LT, and 110LB included in visual sensor 110 into an AI model, and perform visual recognition using the AI ​​model to detect feature points of hand joints from the multiple image frames. In embodiments of this disclosure, the AI ​​model can be implemented as a deep neural network (DNN) model trained to identify objects (e.g., a user's hand) and feature points (e.g., joints) of the object from input image data. In this disclosure, the DNN model can be an end-to-end model trained using supervised learning methods, applying tens of thousands or hundreds of millions of images as input data and applying feature points of hand joints included in the input data as ground truth. The DNN model can be implemented using, for example, a convolutional neural network (CNN) model, but is not limited to this. The DNN model can be implemented using, for example, a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network. AR device 100 can identify hand-raising movements based on the movement of detected feature points over time.

[0065] AR device 100 can obtain the position information of the user's hand based on sensing data received from an external device (e.g., a wearable device worn on the user's hand). In embodiments of this disclosure, AR device 100 can obtain the relative positional relationship between the user's hand and AR device 100 based on UWB signals or Bluetooth signals received from the wearable device. In this disclosure, the relative positional relationship may include at least one of distance, direction, and orientation between the user's hand wearing the wearable device and AR device 100.

[0066] AR device 100 can be transmitted via brainwave sensor 120 (see...) Figure 4 The obtained EEG signal data provides biometric information about the user's brainwaves. In embodiments of this disclosure, the AR device 100 can identify negative feedback of brainwave potentials based on the EEG signal data.

[0067] In operation S220, the AR device 100 can detect a touch interface 160 (see...). Figure 1 Touch input to touch interface 160 can be input from the user, but is not limited to this. Touch interface 160 can receive unauthorized touch input from other persons other than the authorized user of the AR device 100.

[0068] In operation S230, the AR device 100 may determine whether the touch input is a valid input based on sensing data. In this disclosure, a valid input refers to input that has the effect of causing the AR device 100 to perform an interaction including a function and / or operation corresponding to the touch input. In embodiments of this disclosure, valid input may include authorized input based on user intent.

[0069] In an embodiment that identifies a hand-raising motion from multiple image frames, when touch input is detected within a preset time period from the point in time when the hand-raising motion is identified, the AR device 100 can determine the touch input as an authorized input by the user's intent.

[0070] In embodiments where sensing data is obtained from a wearable device and the relative positional relationship between the user's hand and the AR device 100 is determined based on the sensing data, the AR device 100 can identify whether the user's hand is located within a preset acceptable area based on the relative positional relationship, and determine whether the touch input is valid based on the identification result. For example, when the user's hand is located within a preset distance (e.g., 5 centimeters) from the AR device 100, the AR device 100 can determine the touch input as an authorized input based on the user's intent.

[0071] In embodiments where EEG signal data is obtained from the brainwave sensor 120, the AR device 100 can identify negative feedback of the user's brainwave potentials (e.g., error-related negativity (ERN)) from the EEG signal data and determine whether the touch input is valid based on the identification result. For example, when ERN is identified within a preset time period from the time the touch input is received, the AR device 100 can determine the touch input as authorized input. The preset time period can be, for example, at least 50 ms to 100 ms. However, it is not limited to this.

[0072] Reference Figure 3 The method of performing interaction through its AR device 100 based on the result of determining whether the touch input is a valid input is described in detail.

[0073] Figure 3 This is a flowchart illustrating a method by which an AR device 100 performs an interaction based on whether a touch input is a valid input, according to an embodiment of the present disclosure.

[0074] In operation S310, the AR device 100 determines whether the touch input is a valid input based on user intent. The specific method of determination is... Figure 2 The operation is the same as that described in S230, so redundant descriptions are omitted.

[0075] In operation S320, when the touch input is determined to be valid, the AR device 100 executes the function or operation corresponding to the touch input. In embodiments of this disclosure, when the touch input is determined to be authorized input by the user's intent, the AR device 100 may execute the interaction corresponding to the touch input.

[0076] In operation S330, when the touch input is determined to be invalid, the AR device 100 ignores the touch input and terminates the function or operation instead of performing it. In embodiments of this disclosure, when the touch input is determined to be unintentional and unauthorized, the AR device 100 may not perform the interaction corresponding to the touch input.

[0077] Unintentional and unauthorized input may occur at the touch interface 160 of the AR device 100 (see...) Figure 1In this disclosure, unauthorized touch input can be touch input from someone other than the user of AR device 100. For example, unauthorized touch input by an outsider may occur, such as when a user is watching a movie through AR device 100 and a family member makes a touch input to pause or close a window, or when a navigation application is running through AR device 100 in a crowded place and a stranger accidentally triggers a touch input to terminate the navigation application. Furthermore, when enjoying a multiplayer virtual reality (VR) game using AR device 100, unauthorized external touch input may occur, such as when another player accidentally touches touch interface 160 during gameplay or physical activity, causing the game to be temporarily stopped. In this disclosure, unintentional touch input can be touch input that is accidentally provided without the user's intention to provide the touch input. For example, unintentional touch input may occur, such as when a user mistakenly places his / her hand on touch interface 160 of AR device 100 while trying to change his / her hairstyle, or when a user touches touch interface 160 while reaching for an object on a shelf.

[0078] Recently, technologies for providing secure and / or personalized access to AR device 100 have been developed and become widespread. However, when a user wears AR device 100, there is no solution to prevent unintentional and unauthorized touch input to touch interface 160.

[0079] This disclosure provides an AR device 100 and a method of operating thereof, which determines valid touch input to a touch interface 160 and prevents interactions based on unintentional and unauthorized touch input (i.e. invalid input) from being automatically executed, so that users can use the AR device 100 safely and conveniently.

[0080] In passing Figures 1 to 3 In the embodiments shown and described, the AR device 100 can obtain sensing data about hand movement via a visual sensor, an electroencephalogram (EEG) sensor, or an external sensor (e.g., UWB, Bluetooth, etc. of a wearable device). When touch input is detected by the touch interface 160, it determines whether the touch input is valid based on the sensing data about hand movement. If the touch input is determined to be invalid based on the determination result, the touch input is ignored and no interaction corresponding to the touch input is performed. The AR device 100 according to embodiments of this disclosure can prevent touch input provided by the user erroneously, enhancing comfort and immersion and improving user experience (UX). Furthermore, the AR device 100 according to embodiments of this disclosure can prevent the automatic execution of interactions caused by unauthorized touch input from unauthorized outsiders, thereby preventing the leakage of sensitive data (such as personal information), resulting in a technical effect of enhanced security.

[0081] The AR device 100 according to embodiments of the present disclosure can detect the surrounding environment and, when a crowded environment (e.g., in a concert or subway) is detected, automatically or by user input ignores unintentional and unauthorized touch inputs that are determined to be invalid inputs, thereby achieving personalized automatic authentication.

[0082] Figure 4 This is a block diagram illustrating the components of an AR device 100 according to an embodiment of the present disclosure.

[0083] Reference Figure 4 The AR device 100 may include a visual sensor 110, an electroencephalogram (EEG) sensor 120, a motion sensor 130, a processor 140, a memory 150, a touch interface 160, and a communication interface 170. The visual sensor 110, EEG sensor 120, motion sensor 130, processor 140, memory 150, touch interface 160, and communication interface 170 may be electrically and / or physically connected to each other. Figure 4 The diagram shows components used to describe the operation of AR device 100, but the components included in AR device 100 are not limited to those shown. Figure 4 The components shown. AR device 100 may not include Figure 4 Some of the components shown are described. In embodiments of this disclosure, AR device 100 may not include EEG sensor 120. In embodiments of this disclosure, AR device 100 may not include motion sensor 130. Furthermore, in embodiments of this disclosure, AR device 100 may not include either EEG sensor 120 or motion sensor 130.

[0084] In embodiments of this disclosure, the AR device 100 may be implemented as a portable device, in which case the AR device 100 may further include a battery for supplying operating power to the visual sensor 110, the electroencephalogram sensor 120, the motion sensor 130, the processor 140, the touch interface 160, and the communication interface 170.

[0085] The vision sensor 110 is configured to acquire images of a hand by capturing images of a hand in real space and in real space. The vision sensor 110 may include one or more cameras. Each camera may include a lens module, an image sensor, and an image processing module. The camera may acquire still images or videos of an object via an image sensor (e.g., CMOS or CCD). The video may include multiple image frames acquired sequentially by capturing the object via the camera. The image processing module may encode still images having a single image frame or video data consisting of multiple image frames acquired by the image sensor and send it to the processor 140.

[0086] In embodiments of this disclosure, the vision sensor 110 may be implemented as a small shape element to be mounted on a portable AR device 100, and may be implemented as a lightweight RGB camera that consumes low power. However, it is not limited thereto, and the vision sensor 110 may include a depth camera, such as a stereo camera, a time-of-flight (ToF) camera, or an infrared (IR) camera.

[0087] The vision sensor 110 may include two or more cameras. For example, when the AR device 100 is implemented as an HMD or AR glasses, the vision sensor 110 may include a total of six cameras, including an upper left camera, a middle left camera, and a lower left camera respectively positioned at the top, middle, and bottom of the frame surrounding the left eye lens, and an upper right camera, a middle right camera, and a lower right camera respectively positioned at the top, middle, and bottom of the frame surrounding the right eye lens. Figure 9a and Figure 9b The locations of the multiple cameras included in the visual sensor 110 set on the AR device 100 are described in detail. However, the number and location of the multiple cameras are not limited to the examples above.

[0088] The visual sensor 110 can acquire multiple image frames, including the user's hand, by capturing images of the hand in real space via multiple cameras.

[0089] The brainwave sensor 120 may include an EEG sensor configured to acquire EEG signal data by sensing potential fluctuations in brainwaves. In embodiments of this disclosure, the brainwave sensor 120 may detect event-related potential (ERP) components by monitoring potential fluctuations in brainwaves based on EEG signal data. These ERP components include at least one of feedback-related negative (ERP-FRN) and feedback-related positive (ERP-FRP) components. The brainwave sensor 120 may provide the detected ERP components to the processor 140.

[0090] Motion sensor 130 is a sensor configured to sense motion information relating to movement of AR device 100. In embodiments of this disclosure, when user input or unauthorized input from a third party is applied to AR device 100, motion sensor 130 can obtain motion information by sensing vibration or movement of AR device 100. Motion sensor 130 can provide the obtained motion information to processor 140.

[0091] Processor 140 can execute one or more instructions of a program stored in memory 150. Processor 140 may include hardware components for performing arithmetic, logic, and input / output operations, as well as image processing. Figure 4The element is shown as a component, but is not limited thereto. In embodiments of this disclosure, processor 140 may be configured with one or more components. The one or more processors included in processor 140 may be circuits such as system-on-a-chip (SoC), integrated circuit (IC), etc. For example, processor 140 may be a general-purpose processor (such as a central processing unit (CPU), application processor (AP), digital signal processor (DSP), etc.), a dedicated graphics processor (such as a graphics processing unit (GPU), vision processing unit (VPU), etc.), or a dedicated artificial intelligence (AI) processor (such as a neural processing unit (NPU)).

[0092] Processor 140 may include various processing circuitry and / or multiple processors. For example, the term "processor" as used in this disclosure, including the claims, may include various processing circuitry that includes at least one processor. As described in this disclosure, one or more of the at least one processor may be configured to perform various functions individually and / or collectively in a distributed manner. As used herein, a processor, at least one processor, or one or more processors may be configured to perform various functions. However, these terms cover, but are not limited to, situations where one processor performs some functions while other processors(one or more) perform some other functions, and situations where a single processor performs all functions. Furthermore, at least one processor may include a combination of processors performing the various functions disclosed in a distributed manner. At least one processor may execute program instructions to implement or perform various functions.

[0093] Processor 140 can control the processing of input data according to predefined operating rules or AI models. When processor 140 is a dedicated AI processor, it can be designed as a hardware architecture specifically for processing a particular AI model.

[0094] The memory 150 may include, for example, at least one type of storage medium, including flash memory, hard disk, multimedia card micro-memory, card-type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), or optical disk.

[0095] The memory 150 may include instructions related to function and / or operation, wherein the AR device 100 obtains sensing data about hand movement from at least one of a visual sensor 110, an electroencephalogram (EEG) sensor 120, and a motion sensor 130, obtains at least one of hand movement information, location information, and biometric information based on the obtained sensing data, and determines whether a touch input to the touch interface 160 is a valid input based on at least one of the hand movement information, location information, and biometric information. In embodiments of this disclosure, the memory 150 may store at least one of algorithms, data structures, program code, applications, and instructions readable by the processor 140. The instructions, algorithms, data structures, and program code stored in the memory 150 may be implemented, for example, in a programming or scripting language such as C, C++, Java, assembler, etc.

[0096] The processor 140 can be implemented by executing instructions or program code stored in memory 150. The functions and / or operations performed when the processor 140 executes instructions or program code from each of the plurality of modules stored in memory 150 will now be described in detail.

[0097] Processor 140 can acquire sensing data about hand movement from at least one of vision sensor 110, electroencephalogram (EEG) sensor 120, and motion sensor 130, and obtain at least one of hand movement information, location information, and biometric information based on the acquired sensing data. In embodiments of this disclosure, processor 140 can acquire multiple image frames obtained by capturing images of the hand using multiple cameras included in vision sensor 110. Processor 140 can input the multiple image frames acquired by the multiple cameras into an AI model and perform visual recognition using the AI ​​model to detect feature points of hand joints from the multiple image frames. In embodiments of this disclosure, the AI ​​model can be implemented as a DNN model trained to identify objects (e.g., a user's hand) and feature points (e.g., joints) of the object from input image data. Figure 7 Describe the AI ​​model in detail.

[0098] The processor 140 can continuously acquire multiple image frames from the vision sensor 110 over time, and identify the hand-raising motion based on the movement of feature points detected from the multiple image frames acquired over time. This will be combined with... Figures 7 to 11 A specific embodiment of the processor 140 detecting feature points from multiple image frames obtained by the vision sensor 110 and identifying the hand-raising motion based on the detected feature points is described in detail.

[0099] In embodiments of this disclosure, the visual sensor 110 may include a depth camera configured to acquire depth values ​​of an object, and the processor 140 may acquire multiple image frames and obtain hand depth values ​​(or hand depth values) from the acquired multiple image frames by capturing images of a hand over time via the depth camera. The processor 140 may detect changes in the acquired depth values ​​over time and identify hand-raising movements based on these changes in depth values. Figure 12 and Figure 13 A specific embodiment of the processor 140 obtaining hand depth values ​​from multiple image frames acquired by a depth camera and identifying hand raising movements based on changes in the depth values ​​over time is described in detail.

[0100] In embodiments of this disclosure, the processor 140 can obtain UWB signals or Bluetooth signals from the sensors of the wearable device via the communication interface 170. The wearable device can be a device worn on a part of the user's body and carried by the user, such as a smartwatch worn around the user's hand. However, it is not limited to this, and for example, the wearable device can include a smart ring, bracelet, anklet, necklace, contact lens, clothing integration device (e.g., electronic clothing), body attachment device (e.g., skin patch), or bio-implantable device (e.g., implantable circuitry). In embodiments of this disclosure, the wearable device can include a UWB communication module or a Bluetooth communication module, and the processor 140 can receive Angle of Arrival (AoA) information from the UWB signal or BLE location information from the Bluetooth signal from the wearable device via the communication interface 170. The processor 140 can obtain the relative positional relationship between the user's hand and the AR device 100 based on the received AoA information or BLE location information. In embodiments of this disclosure, the relative positional relationship can include at least one of the distance, direction, and orientation between the user's hand wearing the wearable device and the AR device 100.

[0101] In embodiments of this disclosure, the processor 140 can obtain EEG signal data by sensing potential fluctuations in brain waves from the user's head via the EEG sensor 120.

[0102] In embodiments of this disclosure, the processor 140 can obtain motion information about the movement of the AR device 100 via the motion sensor 130. When the touch interface 160 receives adjustment input from the user to adjust the AR device 100 by changing settings or options, vibration or movement may occur within the AR device 100. When adjustment input from the user is detected via the touch interface 160, the processor 140 can obtain motion information about the vibration or movement occurring within the AR device 100. Figure 20a and Figure 20b The motion sensing data for each of the following scenarios is described in detail: general touch input and adjustment input.

[0103] Processor 140 may determine whether a touch input detected by touch interface 160 is a valid input based on at least one of hand movement (e.g., hand-raising motion) information, location information, and biometric information (e.g., EEG signal data). In embodiments of this disclosure, processor 140 may determine whether the touch input is an authorized input intended by the user based on at least one of hand movement information, location information, and biometric information. In embodiments where hand movement information (e.g., hand-raising motion) is identified, when touch input to touch interface 160 is detected within a preset time period from the time the hand-raising motion is identified, processor 140 may determine the touch input as an input indicating user intent (i.e., valid input). In this case, processor 140 may perform an interaction corresponding to the touch input. For example, functions and / or operations such as application execution, menu selection, or option change caused by the touch input may be performed. When touch input is detected after a preset time period has elapsed from the time the hand-raising motion is detected, processor 140 may determine the touch input as unintentional and unauthorized input (i.e., invalid input). In this case, the processor 140 may ignore the touch input and may not perform the functions and / or operations corresponding to the touch input.

[0104] In embodiments where sensing data is obtained from a wearable device and the user's hand position information is obtained based on the sensing data, the processor 140 can identify whether the user's hand is located within a preset acceptable area based on the relative positional relationship between the user's hand and the AR device 100, and determine whether the touch input is valid based on the identification result. For example, when the user's hand is located within a preset distance (e.g., 5 cm) from the AR device 100, the processor 140 can determine the touch input as authorized input (i.e., valid input) by the user's intent. In this case, the processor 140 can perform the interaction corresponding to the touch input. On the other hand, for example, when the user's hand is located outside a preset distance or preset angle range from the AR device 100, the processor 140 can determine the touch input as unintentional and unauthorized input (i.e., invalid input). In this case, the processor 140 can ignore the touch input and may not perform the function and / or operation corresponding to the touch input. Figures 14 to 16b A detailed description of a specific embodiment of the AR device 100 determining whether a touch input is authorized based on sensing data obtained from a wearable device and performing an interaction based on the determination result.

[0105] In embodiments where biometric information (e.g., EEG signal data) is obtained from the brainwave sensor 120, the processor 140 can identify negative feedback of the user's brainwave potentials (e.g., error-related negative waves (ERN)) from the EEG signal data and determine whether the touch input is valid based on the identification result. In embodiments of this disclosure, when an ERN is identified within a preset time period from the time the touch input is detected by the touch interface 160, the processor 140 can determine the touch input as an authorized input. The preset time period can be, for example, at least 50 ms to 100 ms. However, it is not limited to this. When the touch input is determined to be an authorized input, the processor 140 can perform the interaction corresponding to the touch input. On the other hand, for example, when no ERN is identified even after the preset time period from the time the touch input is detected, the processor 140 can determine the touch input as unintentional and unauthorized input (i.e., invalid input). In this case, the processor 140 can ignore the touch input and may not perform the function and / or operation corresponding to the touch input. Figure 17 and Figure 18 The following describes in detail a specific embodiment of the AR device 100 that determines whether a touch input is authorized based on biometric information (e.g., EEG signal data) obtained through an electroencephalogram (EEG) sensor 120 and performs an interaction based on the determination result.

[0106] In an embodiment where motion information of the AR device 100 is obtained from the motion sensor 130, when motion information is obtained via the motion sensor 130 after a touch input is detected, the processor 140 may determine the touch input as unintentional and unauthorized. In this case, the processor 140 may ignore the touch input and may not perform the function and / or operation corresponding to the touch input.

[0107] Touch interface 160 is a hardware input device configured to receive touch or tap input from a user or a third party. In embodiments of this disclosure, touch interface 160 may include a touchpad, touchscreen, physical key button, or switch. Touch interface 160 may detect touch input and provide information about the detected touch input to processor 140.

[0108] Communication interface 170 is a hardware device configured to send data to or receive data from a server or external device via a wired or wireless communication network. Communication interface 170 can perform data communication with a server or external electronic device using at least one of the following data communication schemes: wired local area network (LAN), wireless LAN, Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), Zigbee, Wi-Fi Direct, Infrared Data Association (IrDA), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), Global Microwave Access Interoperability (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and radio frequency (RF) communication.

[0109] In embodiments of this disclosure, the communication interface 170 may include a UWB communication module for performing UWB communication. In this disclosure, UWB communication refers to a communication scheme that uses an ultra-wideband frequency band between 3.1 GHz and 10.6 GHz to perform data transmission and reception. UWB communication networks can transmit or receive data at rates up to 500 Mbps.

[0110] However, it is not limited to this, and when the AR device 100 is implemented as a portable device (such as AR glasses or HMD) to be worn on a part of the user's body, the communication interface 170 can perform data transmission and reception through a network and server or external device that conforms to mobile communication standards (such as CDMA, WCDMA, 3G, 4G (LTE), 5G Sub-6) and / or uses a millimeter wave (mmWave) communication scheme.

[0111] In embodiments of this disclosure, the communication interface 170 can receive UWB signals or Bluetooth signals from a wearable device worn around the user's hand under the control of the processor 140. The communication interface 170 can provide the received UWB signals or Bluetooth signals to the processor 140.

[0112] Figure 5 This illustrates how an AR device 100, according to an embodiment of the present disclosure, detects the hand-raising motion of a user 10.

[0113] Reference Figure 5 The user 10 wearing the AR device 100 can maintain a stable posture at a first time point (t=N) and then perform a hand-raising motion at a second time point (t=N+1). The AR device 100 can utilize a visual sensor 110 including multiple cameras (see... Figure 1 and Figure 4The AR device 100 captures multiple image frames of the user's hand using a visual sensor 110, and detects feature points of the joints in the hand from these multiple image frames. The AR device 100 can detect feature points of the hand joints from multiple image frames acquired continuously over time by the visual sensor 110, and recognize hand-raising movements based on the movement of the detected feature points over time.

[0114] When touch input to the touch interface 160 is detected within a preset time period from the point in time when the hand-raising movement is detected (i.e., the second time point), the AR device 100 can determine the touch input as authorized input by the user's intent. Figure 5 In the embodiment shown, when a touch input or tap on the touch interface 160 is received at a third time point (t=N+total time) before a preset time period has elapsed from the second time point (t=N), the AR device 100 can determine the touch input as an authorized input.

[0115] On the other hand, when a touch input or tap on the touch interface 160 is received after a preset time period from the second time point (t=N), the AR device 100 can determine the touch input as unintentional and unauthorized. When the touch input is determined to be unintentional and unauthorized, the AR device 100 can ignore the touch input and may not perform the interaction corresponding to the touch input.

[0116] AR device 100 can identify downward hand movement based on the movement of feature points of hand joints detected from multiple image frames. In embodiments of this disclosure, when downward hand movement is identified, AR device 100 can determine all detected touch inputs as unauthorized inputs until a hand-raising movement is identified again later.

[0117] Figure 6 This is a flowchart illustrating a method by which an AR device 100 determines whether a touch input is a valid input based on a user's hand-raising motion, according to an embodiment of the present disclosure.

[0118] Figure 7 This illustrates how an AR device 100 according to an embodiment of the present disclosure detects hand joint features P1 to P2 from multiple image frames 710-1 to 710-n using an AI model 700. 10 .

[0119] Reference Figure 6 and Figure 7 The present disclosure will describe in detail the functions and / or operations of the AR device 100 according to embodiments of the present disclosure, which identify hand-raising movements from a plurality of image frames 710-1 to 710-n and determine whether touch input is a valid input based on the hand-raising movements.

[0120] Figure 6 The operation of S610 is Figure 2The detailed operation of operation S210 is shown below. In operation S610, the AR device 100 obtains multiple image frames of the user's hand by continuously capturing images of the hand with a camera. In embodiments of this disclosure, the visual sensor 110 of the AR device 100 (see...) Figure 1 and Figure 4 (This may include multiple cameras. See also...) Figure 7 The AR device 100 can obtain multiple image frames 710-1 to 710-n by continuously capturing images of the user's hand using multiple cameras. In embodiments of this disclosure, the multiple image frames 710-1 to 710-n may include not only images of the user's hand, but also images of at least one of the user's wrist, elbow, arm, or shoulder.

[0121] exist Figure 6 In operation S620, the AR device 100 detects feature points of hand joints by inputting multiple image frames into an AI model. In this disclosure, a joint refers to a portion of the hand and arm where multiple bones connect to each other. In this disclosure, a feature point can refer to a point in an image that is easily identifiable or distinguishable from the surrounding background. Feature points of hand joints may include, for example, feature points of the wrist joint, feature points of the palm joint, feature points of the forearm, or feature points of the upper arm.

[0122] Also refer to Figure 7 The processor 140 of the AR device 100 (see Figure 4 Multiple image frames 710-1 to 710-n can be input into AI model 700, and inference can be performed using AI model 700 to detect feature points P1 to P2 of the joints from the multiple image frames 710-1 to 710-n. 10 In embodiments of this disclosure, the AI ​​model may be implemented as a DNN model, which is trained to identify objects (e.g., a user's hand) and feature points (e.g., joints) of the objects from input image data. In this disclosure, the DNN model may be a model trained using a supervised learning method, applying tens of thousands or hundreds of millions of images as input data and using feature points of hand joints included in the input data as a baseline.

[0123] AI model 700 may include, for example, a 3D feature extractor block as the architecture of a hand joint, a feature transform layer (FTL) for generating 3D features, and a pose regression block for skeletal-based feature recognition of pose. The pose regression block may be implemented as a known skeleton regressor, such as regressor-K. When multiple image frames 710-1 to 710-n are input into AI model 700, feature points P1 to P2 of the hand joint can be extracted via the 3D feature extractor block. 10 The extracted feature points P1 to P2 can be transferred via FTL. 10 Transformed into 3D feature points z 3D The transformed 3D feature points z3D The input is fed into the pose regression block, and the pose regression block can output z-axis data including 3D information, temporal context, and skeleton features. R The skeleton regressor (regressor-K) can receive z... R The AI ​​model 700 can predict 3D hand poses when an image is input. In embodiments of this disclosure, the AI ​​model 700 may be an end-to-end DNN model trained to predict 3D hand poses of hand joints using a 3D feature extractor block, FTL, pose regression block, and skeleton regressor (regressor-K) when an image is input.

[0124] Refer again Figure 6 In operation S630, the AR device 100 can identify a hand-raising motion based on the movement of detected feature points over time. See also... Figure 7 The processor 140 of the AR device 100 can recognize the raising hand movement based on the 3D hand gesture output by the AI ​​model 700.

[0125] Although not in Figure 6 As shown in the diagram, but after operating the S630, it can be... Figure 2 Operation S220 is shown in the figure.

[0126] Figure 6 Operations S640 to S660 are Figure 2 The detailed operation of operation S230 is shown below. In operation S640, the AR device 100 determines whether touch input is detected within a preset time period from the point at which the hand-raising movement is recognized.

[0127] In operation S650, when touch input is detected within a preset time period from the time point at which the hand-raising movement is detected, the AR device 100 determines the touch input as valid input. In embodiments of this disclosure, when touch input is detected within a preset time period from the time point at which the hand-raising movement is detected, the AR device 100 may determine the touch input as authorized input by the user's intent. In operation S320, when the touch input is determined to be authorized input, the AR device 100 executes the function or operation corresponding to the touch input.

[0128] In operation S660, when touch input is detected after a preset time period has elapsed since the time of recognizing the hand-raising movement, the AR device 100 determines the touch input as invalid. In embodiments of this disclosure, when touch input is detected after a preset time period has elapsed since the time of recognizing the hand-raising movement, the AR device 100 may determine the touch input as unintentional and unauthorized. In operation S330, when the touch input is determined to be unintentional and unauthorized, the AR device 100 ignores the touch input and terminates the function or operation instead of performing any action.

[0129] Figure 8 This is a flowchart illustrating a method for identifying hand-raising movements based on the temporal and spatial correlation of feature points of a joint using its AR device 100, according to an embodiment of the present disclosure.

[0130] Figure 8 The operations S810 to S830 shown are Figure 6 Detailed operation of S630. Figure 8 The operation shown in S830 can be followed by Figure 6 Operation S640.

[0131] Can be executed Figure 6 Following the function and / or operation of S620, operation S810 is executed. In operation S810, the AR device 100 detects feature points for each part of the hand from each of multiple image frames captured by multiple cameras with different viewpoints. In embodiments of this disclosure, the visual sensor 110 of the AR device 100 (see...) Figure 1 and Figure 4 This may include multiple cameras positioned at different locations to capture images from different viewpoints. (Refer to...) Figure 9a and Figure 9b The description includes multiple cameras in the vision sensor 110.

[0132] Figure 9a The arrangement of a plurality of cameras 110LT, 110LM, 110LB, 110RT, 110RM and 110RB mounted on an AR device 100 according to an embodiment of the present disclosure is shown.

[0133] Reference Figure 9a The AR device 100 can be implemented as a head-mounted display (HMD) that will be worn on the user's head. Figure 9a In the illustrated embodiment, the HMD may include an upper left camera 110LT and an upper right camera 110RT positioned at the upper end of a frame surrounding the left and right eye lenses, a middle left camera 110LM and a middle right camera 110RM positioned in the middle of the frame, and a lower left camera 110LB and a lower right camera 110RB positioned at the lower end of the frame. Although in Figure 9a The AR device 100 is shown as including a total of six cameras, but this is just an example and is not limited to this.

[0134] Each of the multiple cameras 110LT, 110LM, 110LB, 110RT, 110RM, and 110RB is positioned at different locations on the AR device 100 and has a different viewpoint, so the body parts captured may differ. For example, since the upper left camera 110LT and the upper right camera 110RT are visual sensors that capture a forward view, when the hand is not placed within the forward view, the upper left camera 110LT and the upper right camera 110RT may not capture the hand, but may instead obtain image frames excluding the hand. The middle left camera 110LM and the middle right camera 110RM can capture image frames including both the arm and hand by capturing the upper arm and hand. The lower left camera 110LB and the lower right camera 110RB are visual sensors that capture a downward view and can obtain image frames about the user's forearm and palm.

[0135] Figure 9b The arrangement of a plurality of cameras 110LT, 110LM, 110LB, 110RT, 110RM and 110RB mounted on an AR device 100 according to an embodiment of the present disclosure is shown.

[0136] Reference Figure 9b The AR device 100 can be implemented as AR glasses in the form of glasses to be worn on the user's face. Figure 9b In the illustrated embodiment, the AR glasses may include an upper left camera 110LT and an upper right camera 110RT positioned at the upper end of a frame surrounding the left and right eye lenses, a middle left camera 110LM and a middle right camera 110RM positioned in the middle of the frame, and a lower left camera 110LB and a lower right camera 110RB positioned at the lower end of the frame. Although in Figure 9b The AR device 100 is shown as including a total of six cameras, but this is just an example and is not limited to this.

[0137] exist Figure 9b In the illustrated embodiment, portions (e.g., upper arm, forearm, hand, and palm) captured by multiple cameras 110LT, 110LM, 110LB, 110RT, 110RM, and 110RB are combined with... Figure 9a Those descriptions that are identical or redundant will be omitted.

[0138] Refer again Figure 8 The AR device 100 can be powered by multiple cameras 110LT, 110LM, 110LB, 110RT, 110RM, and 110RB (see...). Figure 9a and Figure 9bThe AR device 100 detects feature points for each part of the hand in each of the multiple image frames obtained. For example, the AR device 100 can detect joint feature points including those in the upper arm and hand from image frames obtained by the left center camera 110LM and the right center camera 110RM, and detect joint feature points in the forearm and palm from image frames obtained by the lower left camera 110LB and the lower right camera 110RB.

[0139] In operation S820, the AR device 100 combines multiple image frames and identifies the temporal and spatial correlations between detected feature points. Figure 10 and Figure 11 Detailed description of operation S820.

[0140] Figure 10 This illustration shows how an AR device 100 according to an embodiment of the present disclosure identifies a hand-raising motion by using multiple image frames obtained via multiple cameras 110RM and 110RB.

[0141] Reference Figure 10 The right-center camera 110RM can obtain multiple image frames by continuously capturing images of the user's arm and hand over time. 1-RM to i 4-RM The lower right camera, 110RB, can acquire multiple image frames by continuously capturing images of the user's arm and hand over time. 1-RB to i 4-RB .

[0142] The processor 140 of the AR device 100 (see...) Figure 4 Multiple image frames can be obtained from the right-center camera 110RM. 1-RM to i 4-RM Detection of feature points P related to joints 1-RM and P 2-RM For example, the first joint feature point P 1-RM It can be a feature point of the hand joint, and the second joint feature point P 2-RM These could be feature points of the arm joints. Although in Figure 10 The feature points of the hand joint and the feature points of the arm joint are shown in singular numbers, but this is only for illustrative purposes, and the number of feature points in this disclosure is not limited to such a number. Figure 10 The number shown. Similarly, the processor 140 can obtain multiple image frames i from the lower right camera 110RB. 1-RB to i 4-RB Detection of feature points P related to joints 1-RB and P 2-RB .

[0143] Processor 140 can identify a part of the hand by comparing feature points detected from various image frames acquired by multiple different cameras. For example, processor 140 can identify a part of the hand by comparing feature points detected from various image frames acquired by the right-center camera 110RM at a first time point t1. 1-RM The first joint feature point P detected in the middle 1-RM The first image frame i obtained by the lower right camera 110RB 1-RB The first joint feature point P detected in the middle 1-RB By comparison, the first joint feature point P was identified. 1-RM and P 1-RB The hand-pointing part. Similarly, the processor 140 can obtain the first image frame i from the right-center camera 110RM at time t1. 1-RM The second joint feature point P detected in the middle 2-RM The first image frame i obtained by the lower right camera 110RB 1-RB The second joint feature point P detected in the middle 2-RB By comparison, the feature point P of the second joint was identified. 2-RM and P 2-RB Indicates the arm portion.

[0144] The processor 140 can identify spatial correlations between feature points based on hand portions identified through multiple image frames. In embodiments of this disclosure, the processor 140 can obtain spatial correlations based on positional relationships between feature points identified from multiple image frames obtained by different cameras 110RM and 110RB, respectively, and information about the arrangement of cameras 110RM and 110RB on the AR device 100.

[0145] Although only the right center camera 110RM and the right bottom camera 110RB are shown for ease of illustration, this disclosure is not limited thereto. The method of acquiring multiple image frames, detecting joint feature points from the multiple image frames, and obtaining the spatial correlation between the detected joint feature points can also be applied to the left center camera and the left bottom camera.

[0146] Processor 140 can identify the temporal correlation between feature points by acquiring multiple image frames over time. Figure 11 This illustrates how an AR device 100 according to an embodiment of the present disclosure identifies images from multiple image frames. -RM and i -RB Joint feature point P detected in 1-RM P 2-RM P 1-RM and P 2-RB The temporal and spatial correlation. Also refer to... Figure 11 The processor 140 can obtain image frames from the lower right camera 110RB. -RBDetection and image frames obtained from the right-center camera 110RM -RM The first joint feature point P detected 1-RM Corresponding feature point P 1-RB And calculate feature point P 1-RM With P 1-RB The time difference between them. Similarly, the processor 140 can obtain image frames i from the right center camera 110RM. -RM Detecting the second joint feature point P 2-RM and image frames obtained from the lower right camera 110RB -RB Detecting the second joint feature point P 2-RB And calculate feature point P 2-RM With P 2-RB The processor 140 can identify the temporal correlation between feature points based on the calculated time difference.

[0147] Refer again Figure 8 In operation S830, AR device 100 identifies the hand-raising motion based on the recognized temporal and spatial correlation. (See also...) Figure 10 The processor 140 of the AR device 100 can base its data on feature points P that change over time. 1-RM P 2-RM P 1-RB and P 2-RB Location and feature point P 1-RM P 2-RM P 1-RB and P 2-RB The temporal and spatial correlation between them is used to identify the arm raising motion.

[0148] Figure 12 This is a flowchart illustrating a method for recognizing hand-raising movements using its AR device 100 based on hand depth value information (or hand depth value information) according to an embodiment of the present disclosure.

[0149] Figure 13 This illustrates how an AR device 100 according to an embodiment of the present disclosure identifies a hand-raising motion based on hand depth value information.

[0150] Now refer to Figure 12 and Figure 13 The AR device 100 according to embodiments of the present disclosure is described in detail for recognizing hand-raising movements based on hand depth information.

[0151] Figure 12 Operation S1210 is Figure 2 The detailed operation of operation S210 is shown below. In operation S1210, the AR device 100 obtains multiple image frames of the user's hand by continuously capturing images of the hand using a depth camera. In embodiments of this disclosure, the visual sensor 110 (see...) Figure 1 and Figure 4 The depth camera may include a depth camera that obtains the depth value of an object. The depth camera may include, but is not limited to, at least one of, a stereo camera, a ToF camera, and an IR camera.

[0152] In operation S1220, the AR device 100 obtains hand depth values ​​from multiple acquired image frames. In embodiments of this disclosure, the AR device 100 may utilize a depth camera to photograph the user's body and obtain depth value information for each body part. See also... Figure 13 In the embodiment shown, the user 10 wearing the AR device 100 can maintain a stable posture at a first time point (t=N) and then perform a hand-raising motion at a second time point (t=N+1). The processor 140 of the AR device 100 (see...) Figure 4 The depth values ​​of the body parts of user 10 can be obtained by using a depth camera to photograph the body. For example, in the body parts of user 10, the hand part can have a depth value of 66, and the arm part can have depth values ​​of 38, 44, 50, 56 and 62. Figure 13 The depth values ​​shown are relative values ​​representing depth values ​​calculated based on the position of the depth camera, and the magnitude of the value is proportional to the distance to the AR device 100. For example, in the body portion of user 10, the upper chest has a depth value ranging from 5 to 8, and the depth value increases towards the abdomen.

[0153] In operation S1230, the AR device 100 identifies changes in depth values ​​over time. (See also...) Figure 13 In the embodiment shown, at a second time point (t=N+1), user 10 can perform a hand-raising motion, and therefore, the depth value of the hand can change. For example, at a first time point (t=N), the depth value of the hand is 66, and the depth values ​​of the arm portion are 38, 44, 50, 56, and 62, and at the second time point (t=N+1), the hand-raising motion causes the depth value of the hand to change to 30, 32, or 35, and the depth value of the arm portion also changes to 33, 35, 36, and 40. The processor 140 of the AR device 100 can recognize the change in depth value of the hand and a portion of the arm.

[0154] In operation S1240, the AR device 100 recognizes the hand-raising motion based on changes in depth values. (See also...) Figure 13 In the embodiment shown, the processor 140 can recognize changes in the depth values ​​of the hand and arm, and recognize the user 10's hand-raising movement based on the changes in the depth values. For example, when the depth value of the hand is reduced to below a preset value, the processor 140 can recognize that the user 10 has performed a hand-raising movement.

[0155] After operating the function and / or operation of S1240, it can be Figure 2 Operation S230. Also refer to... Figure 13 When touch input is detected on the touch interface 160, the hand depth value can be changed to "0". When the hand depth value changes to "0", the processor 140 of the AR device 100 can detect the touch input. When touch input is detected within a preset time period from the time point when the hand movement is detected, the processor 140 can determine the touch input as authorized input by the user's intention. When the touch input is determined to be authorized input, the processor 140 can execute the interaction corresponding to the touch input.

[0156] When a preset time period elapses after the detected hand movement, the processor 140 may determine that the touch input is unintentional and unauthorized. If the touch input is determined to be unintentional and unauthorized, the processor 140 ignores the touch input and terminates the function or operation instead of executing it.

[0157] Figure 14 This is a flowchart illustrating a method by which an AR device 100, according to an embodiment of the present disclosure, determines whether a touch input is a valid input based on sensing data received from a wearable device and performs an interaction based on the determination result.

[0158] Figure 15 This illustrates how an AR device 100 according to an embodiment of the present disclosure determines whether a touch input is a valid input based on sensing data received from a wearable device 200.

[0159] Now refer to Figure 14 and Figure 15 The present disclosure describes in detail the functions and / or operations of an AR device 100 according to an embodiment of the present disclosure for determining whether a touch input is a valid input based on sensing data received from a wearable device 200 and performing an interaction based on the determination result.

[0160] Figure 14 Operations S1410 and S1420 shown are Figure 2 The detailed operation of operation S210 shown is described below.

[0161] In operation S1410, the AR device 100 receives sensing data from sensors included in a wearable device worn on the user's hand. In this disclosure, the wearable device can be a device worn on a part of the user's body and carried by the user, for example, a smartwatch worn around the user's hand. See also... Figure 15 In the embodiment shown, wearable device 200 may be a smartwatch worn around the wrist of user 10. However, it is not limited to this, and wearable device 200 may include, for example, a smart ring, bracelet, anklet, necklace, contact lens, clothing integration device (e.g., electronic clothing), body attachment device (e.g., skin patch) or bio-implantable device (e.g., implantable circuitry).

[0162] In embodiments of this disclosure, the wearable device 200 may include a UWB communication module or a Bluetooth communication module. The AR device 100 may include a communication interface 170 (see...). Figure 4 ), and processor 140 (see Figure 4 It can receive UWB signals or Bluetooth signals from wearable device 200 through communication interface 170.

[0163] In operation S1410, the AR device 100 obtains the relative positional relationship between the user's hand and the AR device 100 based on the received sensing data. In embodiments of this disclosure, the relative positional relationship may include at least one of the distance, direction, and orientation between the user's hand and the AR device 100.

[0164] In embodiments of this disclosure, the communication interface 170 of the AR device 100 may include a UWB communication module. The UWB communication module is a communication module that performs data transmission and reception using an ultra-wideband frequency band between 3.1 GHz and 10.6 GHz. The UWB communication module can transmit or receive data at rates up to 500 Mbps. In embodiments of this disclosure, the UWB communication module can receive location information from the wearable device using ultra-wideband frequencies. For example, the processor 140 of the AR device 100 (see...) Figure 4 Ranging can be performed using either single-sided two-way ranging (SS-TWR) or double-sided two-way ranging (DS-TWR). In embodiments of this disclosure, processor 140 can use a plurality of UWB antenna elements included in the UWB communication module to send ranging request messages or polling messages to the wearable device and receive response messages from the wearable device in response to the ranging request signal. Processor 140 can obtain the location information of the wearable device by using the time of arrival (ToA) or time difference of arrival (TDoA) method of the time difference between the ranging request message and the response message. See also... Figure 15 In the embodiment shown, the processor 140 can obtain ranging information about the relative distance between the wearable device 200 worn on the hand of the user 10 and the AR device 100, as well as AoA information as orientation information of the wearable device.

[0165] In embodiments of this disclosure, the communication interface 170 of the AR device 100 may include a Bluetooth communication module, and pairing with a wearable device can be established through the Bluetooth communication module. The processor 140 of the AR device 100 may receive Bluetooth signals from the wearable device through the communication interface 170, and obtain BLE location information from the received Bluetooth signals. The processor 140 may obtain the relative positional relationship between the user's hand and the AR device 100 based on the obtained BLE location information.

[0166] Although not in Figure 14 As shown in the diagram, however, after operating S1420, it can be... Figure 2 Operation S220 is shown in the figure.

[0167] Figure 14 Operations S1430 to S1460 are Figure 2 The detailed operation of operation S230 shown is as follows. In operation S1430, the AR device 100 identifies whether the user's hand is within a preset acceptable area based on relative positional relationships. See also... Figure 15 In the embodiment shown, the acceptable area 1500 refers to the location where a hand touch input can be recognized as authorized input by the user's intent, and can indicate an area within a preset distance or angle range from the AR device 100, for example. The acceptable area 1500 can be preset by the user input or preset as a default value when the product is released. For example, the acceptable area 1500 can refer to an area within 5 centimeters (cm) of the AR device 100. However, it is not limited to this.

[0168] The processor 140 of the AR device 100 can determine whether the user's hand is placed in an acceptable position based on a relative positional relationship (i.e., at least one of the distance, orientation, and orientation between the user's hand and the AR device 100). See also... Figure 15 In the embodiment shown, a user 10 wearing the AR device 100 can maintain a stable posture at a first time point (t=N) and then perform a hand-raising motion at a second time point (t=N+1). At the first time point (t=N), the distance between the AR device 100 and the wearable device 200 may be greater than a threshold distance, and the angle may not be equal to a threshold angle. At the second time point (t=N+1), even when the user 10 performs the hand-raising motion, the user 10's hand, as recognized by the wearable device 200, may be outside the acceptable area 1500. At the third time point (t=N+total time), touch input to the touch interface 160 can be detected, and therefore, the distance between the user 10's hand and the AR device 100 can change to less than a threshold distance, and the angle can also decrease to less than a threshold angle. Based on the relative positional relationship between the AR device 100 and the wearable device 200 at the third time point (t=N+total time), the processor 140 can recognize that the user's hand is within the acceptable area 1500.

[0169] In operation S1440, AR device 100 determines whether the user's hand is in an acceptable area.

[0170] In operation S1450, when a hand is determined to be within an acceptable area, the AR device 100 determines the touch input as valid input. In embodiments of this disclosure, when a hand is determined to be within an acceptable area, the AR device 100 may determine that the touch input is authorized input by the user's intent. Figure 16a An example of wearable device 200 located in an acceptable area is shown. See also... Figure 16a If the distance between the user 10 wearing the wearable device 200 and the AR device 100 is less than a threshold distance, and the angle between the wearable device 200 and the AR device 100 is less than a threshold angle, then the processor 140 can determine the touch input to the touch interface 160 as authorized input.

[0171] In operation S320, when the touch input is determined to be a valid input, the AR device 100 executes the function or operation corresponding to the touch input.

[0172] In operation S1460, when a hand is determined to be outside the acceptable area, the AR device 100 determines the touch input as invalid input. In embodiments of this disclosure, when a hand is determined to be outside the acceptable area, the AR device 100 may determine that the touch input is unintentional and unauthorized. Figure 16b An example of wearable device 200 outside the acceptable area is shown. See also... Figure 16b Because of the hand-raising motion performed by user 10, the distance between the wearable device 200 worn on the hand and the AR device 100 may exceed a threshold distance. Furthermore, the angle between the wearable device 200 and the AR device 100 may not be equal to a threshold angle. In this case, processor 140 may determine touch input to touch interface 160 as unauthorized input unrelated to the user's intent.

[0173] In operation S330, when the touch input is determined to be invalid, the AR device 100 ignores the touch input and terminates the function or operation instead of performing it.

[0174] Figure 17 This is a flowchart illustrating a method by which an AR device 100, according to an embodiment of the present disclosure, determines whether a touch input is an authorized input of user intent based on electroencephalogram (EEG) signal data obtained by using an electroencephalogram (EEG) sensor and performs an interaction based on the determined result.

[0175] Figure 17 The operation of S1710 is Figure 2 The detailed operation of operation S210 is shown below. In operation S1710, the AR device 100 obtains EEG signal data by sensing the potential fluctuations of brain waves using an EEG sensor. In embodiments of this disclosure, the AR device 100 may include an EEG sensor 120 (see [link to documentation]). Figure 4 The brainwave sensor 120 may include an EEG sensor configured to acquire EEG signal data by sensing potential fluctuations in brainwaves. The processor 140 of the AR device 100 (see...) Figure 4EEG signal data can be obtained by using the EEG sensor 120 to sense the potential fluctuations of brain waves from the user's head.

[0176] Operations from S1720 to S1750 are Figure 2 The detailed operation of operation S230 shown is described below.

[0177] In embodiments of this disclosure, the AR device 100 identifies negative feedback of brainwave potentials based on EEG signal data. In embodiments of this disclosure, based on EEG signal data obtained by the brainwave sensor 120, the processor 140 can detect event-related potential (ERP) components from the EEG signal data by monitoring potential fluctuations of brainwaves from the user's head. The ERP components include at least one of feedback-related negative wave (ERP-FRN) components and feedback-related positive wave (ERP-FRP) components. The feedback-related negative wave component is feedback indicating error or abnormality regarding a specific event (e.g., a stimulus or user input) and may include, for example, error-related negative waves (ERN). In embodiments of this disclosure, the processor 140 can identify error-related negative waves (ERN) by monitoring EEG signal data acquired over time.

[0178] Although not in Figure 17 As shown in the diagram, however, after operating S1720, it can be... Figure 2 Operation S220 is shown in the figure.

[0179] In operation S1730, the AR device 100 determines whether an error-related negative signal (ERN) is identified within a preset time period from the time the touch input is received. In this disclosure, the preset time period may be, for example, at least 50 ms to 100 ms from the time the touch input is detected. However, it is not limited to this.

[0180] In embodiments of this disclosure, when using the touch interface 160 (see...) Figure 4 When the AR device 100 receives touch input from a user, its processor 140 may refrain from performing the interaction corresponding to the touch input and from outputting the graphical user interface (GUI) associated with the interaction for a preset time period (e.g., at least 50 ms to 10 ms). After providing touch input to the touch interface, the user expects a change in output. However, when the expected result (e.g., a change in output) does not match the actual result (e.g., no interaction was performed during the preset time period), the user may become confused. In this case, negative feedback can be detected in the EEG potential signal. Negative feedback may include, for example, error-related negative waves (ERN). When the touch input is unintentional, the user's brain does not expect a change in the current output. In this case, negative feedback is not identified from the EEG signal data.

[0181] Reference Figure 18 Describe an embodiment for identifying negative feedback (e.g., error-related negative waves (ERN)).

[0182] Figure 18 This shows an example of EEG signal data 1800 representing fluctuations in brainwave potentials when touch input is detected. (See reference...) Figure 18 The EEG signal data 1800 shown exhibits that, during a preset time period from the time point t0 when the user receives touch input, the Correct Response Negative (CRN) signal 1810 shows no significant fluctuations, while the Error Related Negative (ERN) signal 1820 shows a significant increase in value. Between the time point t0 when the touch input is received and a first time point t1 before the preset time period has elapsed (e.g., 100 ms), the Error Related Negative 1820 may have a larger fluctuation range and a larger value than the Correct Response Negative 1810. In this disclosure, the Error Related Negative 1820 refers to the signal data of fluctuations in brainwave potentials that occur when, after the user provides touch input to the touch interface, although the user expects an interaction to be performed or a related graphical UI to be output, no interaction or graphical UI is output.

[0183] Refer again Figure 17 In operation S1740, when an error-related negative signal (ERN) is identified within a preset time period from the time the touch input is detected, the AR device 100 determines the touch input as a valid input. See also... Figure 18 When an error-related negative wave 1820 is identified within a preset time period starting from the time point t0 from when the touch input is received, the processor 140 can determine the touch input as an authorized input by the user's intent.

[0184] In operation S320, when the touch input is determined to be a valid input, the AR device 100 executes the function or operation corresponding to the touch input.

[0185] In operation S1750, when an Error-Related Negative Wave (ERN) is identified or not identified after a preset time period has elapsed since the touch input was detected, the AR device 100 determines the touch input as invalid. In embodiments of this disclosure, when an Error-Related Negative Wave (ERN) is identified or not identified after a preset time period has elapsed since the touch input was detected, the AR device 100 can determine that the touch input is unintentional and unauthorized. In operation S330, when the touch input is determined to be invalid, the AR device 100 ignores the touch input and terminates the function or operation instead of performing any action.

[0186] Figure 19This is a flowchart illustrating a method by which an AR device 100, according to an embodiment of the present disclosure, determines whether a touch input is an authorized input based on user intent based on motion information and performs an interaction based on the determined result.

[0187] Figure 19 Operations S1910 and S1920 are Figure 2 The detailed operation of operation S210 shown is described below.

[0188] In operation S1910, the AR device 100 uses the motion sensor 130 (see...) Figure 4 The motion sensor 130 is configured to sense motion information about the movement of the AR device. In embodiments of this disclosure, when user input or unauthorized input from a third party is applied to the AR device 100, the processor 140 (see [link to disclosure]) obtains motion information about the AR device. Figure 4 Motion information can be obtained by sensing the vibration or movement of the AR device 100 via the motion sensor 130.

[0189] In operation S1920, AR device 100 receives vibrations or movements of the AR device caused by user adjustment input based on motion information. In this disclosure, adjustment input refers to user input that changes settings or options of AR device 100. Adjustment input may differ in intensity and pattern from touch input via touch or tapping the touch interface 160. In embodiments of this disclosure, processor 140 of AR device 100 may detect the type of motion event in sensing data obtained from motion sensor 130 and determine whether the input received from the user is adjustment input or touch input based on the motion event type. (See also...) Figure 20a and Figure 20b Describe the determined method.

[0190] Figure 20a An example of motion sensing data 2000a is shown when a touch input is received.

[0191] Reference Figure 20a The motion sensing data 2000a shown in the figure detects vibrations with a pulse pattern over a relatively short period of time when a touch input is received from a user.

[0192] Figure 20b An example of motion sensing data 2000b is shown when an adjustment input is received.

[0193] Reference Figure 20b The motion sensing data 2000b shown here, unlike the case of receiving touch input, tends to have patterns that are continuously detected over a relatively long period of time and have a larger vibration amplitude than the vibration caused by touch input.

[0194] Refer again Figure 19 Although not shown, it can be achieved after operation S1920. Figure 2 Operation S220 is shown in the figure.

[0195] Figure 19 Operations S1930 to S1950 are Figure 2 The detailed operation of operation S230 is shown below. In operation S1930, the AR device 100 determines whether vibration or movement information is obtained after detecting touch input.

[0196] If no vibration or movement information is received in operation S1940 after detecting touch input, the AR device 100 determines that touch input was received from the user rather than adjustment input and identifies the touch input as valid input. In operation S320, when the touch input is identified as valid input, the AR device 100 executes the function or operation corresponding to the touch input.

[0197] When vibration or movement information is obtained in operation S1940 after detecting touch input, the AR device 100 determines that an adjustment input has been received from the user and identifies the touch input as invalid. In embodiments of this disclosure, when vibration or movement information is obtained after detecting touch input, the AR device 100 may determine that an adjustment input has been received from the user and determine that the touch input is unintentional and unauthorized. In operation S330, when the touch input is determined to be invalid, the AR device 100 ignores the touch input and terminates the function or operation instead of performing any action.

[0198] exist Figure 19 , Figure 20a and Figure 20b In the illustrated embodiment, the AR device 100 can obtain motion sensing data 2000a and 2000b obtained through the motion sensor 130, analyze the patterns of the motion sensing data 2000a and 2000b, and determine that the touch input is unintentional and unauthorized when it is determined that adjustment input has been received. Therefore, the AR device 100 according to embodiments of this disclosure can prevent unintentional interactions from being automatically executed by not performing the interaction corresponding to the input when the user provides input such as changing settings information or adjusting options.

[0199] This disclosure provides an AR device 100 that determines whether a touch input is an authorized input intended by a user and performs an interaction based on the determination result. According to embodiments of this disclosure, the AR device 100 may include at least one sensor, a touch interface 160 configured to receive touch input, at least one processor 140 including processing circuitry, and a memory 150 storing one or more instructions. One or more instructions may be executed individually or jointly by the at least one processor 140 to cause the AR device 100 to detect hand movement via the at least one sensor and determine, based on the detected hand movement, whether the touch input received via the touch interface 160 is a valid input. One or more instructions may be executed individually or jointly by the at least one processor 140 to cause the AR device 100 to determine, based on the determination of the valid input, whether to perform an interaction corresponding to the touch input.

[0200] In embodiments of this disclosure, one or more instructions may be executed individually or jointly by at least one processor 140 to cause the AR device 100 to ignore the touch input and not perform the function or operation corresponding to the touch input when the touch input is determined to be invalid as a result.

[0201] In embodiments of this disclosure, at least one sensor may include a vision sensor 110 configured as a camera. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to acquire multiple image frames of the user's hand by continuously capturing images of the hand with the camera, input the acquired multiple image frames into an AI model, and detect feature points of the hand's joints from the multiple image frames by performing visual recognition via the AI ​​model. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to recognize a hand-raising movement based on the movement of the detected feature points over time. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to determine that a touch input is valid when it detects touch input within a preset time period from the point at which the hand-raising operation is recognized.

[0202] In embodiments of this disclosure, the cameras may include multiple cameras positioned at different locations on the AR device 100 and having different viewpoints. One or more instructions may be executed individually or jointly by at least one processor 140 to cause the AR device 100 to detect feature points of each part of the hand from multiple image frames captured separately by the multiple cameras from different viewpoints. One or more instructions may be executed individually or jointly by at least one processor 140 to cause the AR device 100 to identify the temporal and spatial correlations between feature points of each part detected from the multiple image frames by combining the multiple image frames. One or more instructions may be executed individually or jointly by at least one processor 140 to cause the AR device 100 to recognize a hand-raising motion based on the temporal and spatial correlations between feature points.

[0203] In embodiments of this disclosure, at least one sensor may include a vision sensor 110 of a depth camera configured to acquire depth values ​​of an object. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to acquire depth values ​​of a hand from a plurality of image frames acquired sequentially by the depth camera and to identify changes in the acquired depth values ​​over time. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to identify a hand-raising motion based on changes in the depth values.

[0204] In embodiments of this disclosure, the AR device 100 may further include a communication interface 170 configured to perform data communication with an external device. The communication interface 170 may receive sensing data from sensors included in the wearable device 200 worn on the user's hand. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to obtain relative positional information, including at least one of the information regarding distance, direction, and orientation between the user's hand and the AR device, based on the sensing data received from the wearable device 200. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to identify whether the user's hand is located within an acceptable area based on the obtained relative positional information, and to determine whether a touch input is a valid input based on the identification result.

[0205] In embodiments of this disclosure, the wearable device 200 may include at least one of a UWB communication module and a Bluetooth communication module. Sensing data may include at least one of BLE location information and AoA information from the UWB signal received from the wearable device 200.

[0206] In embodiments of this disclosure, at least one sensor may include a brainwave sensor 120 configured to acquire EEG signal data by sensing potential fluctuations in brainwaves. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to acquire EEG signal data by utilizing the brainwave sensor 120 to detect potential fluctuations in brainwaves from the user's head, and to identify negative feedback of brainwave potentials based on the acquired EEG signal data. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to determine whether touch input is a valid input based on the identification result.

[0207] In embodiments of this disclosure, one or more instructions may be executed individually or jointly by at least one processor 140 to cause the AR device 100 to identify error-related negative waves (ERNs) by monitoring fluctuations in the potential characteristics of EEG signal data during a preset time period from the time the touch input is received. One or more instructions may be executed individually or jointly by at least one processor 140 to cause the AR device 100 to determine that the touch input is a valid input when an error-related negative wave is identified.

[0208] In embodiments of this disclosure, at least one sensor may further include a motion sensor 130 for detecting movement of the AR device 100. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to obtain motion information, including vibrations or movements of the AR device 100 caused by adjustment input from a user, using the motion sensor 130. One or more instructions may be executed individually or collectively by at least one processor 140 to cause the AR device 100 to determine a touch input as invalid when it obtains motion information after detecting a touch input.

[0209] This disclosure provides a method for determining valid touch input using its AR device 100 and performing an interaction based on the determined result. According to embodiments of this disclosure, the operation method of the AR device 100 may include: in operation S210, acquiring sensing data about hand movement using at least one sensor. The operation method of the AR device 100 may include: in operation S220, detecting touch input to a touch interface 160. The operation method of the AR device 100 may include: in operation S230, determining whether the touch input is valid based on the acquired sensing data.

[0210] In embodiments of this disclosure, the operation method of the AR device 100 may include: when a touch input is determined to be invalid as a result of a determination of valid input, ignoring the touch input and not performing the function or operation corresponding to the touch input.

[0211] In embodiments of this disclosure, at least one sensor may include a vision sensor 110 configured as a camera. Obtaining sensing data in operation S210 may include, in operation S610, acquiring multiple image frames of the user's hand by continuously capturing images of the hand. The operation method of the AR device 100 may include, in operation S620, inputting the acquired multiple image frames into an AI model and detecting feature points of hand joints from the multiple image frames using inference from the AI ​​model; and in operation S630, identifying a hand-raising motion based on the movement of the detected feature points over time. Determining whether a touch input is a valid input in operation S230 may include, in operation S650, determining the touch input as a valid input when a touch input is detected within a preset time period from the point at which the hand-raising motion is identified.

[0212] In embodiments of this disclosure, the cameras may include multiple cameras positioned at different locations on the AR device 100 and having different viewpoints. Identifying the hand-raising motion in operation S630 may include: in operation S810, detecting feature points of each part of the hand from each of multiple image frames captured by the multiple cameras from different viewpoints; and in operation S820, identifying the temporal and spatial correlations between the feature points of each part detected from the multiple image frames by combining the multiple image frames. Identifying the hand-raising motion in operation S630 may include identifying the hand-raising motion based on the temporal and spatial correlations between features.

[0213] In embodiments of this disclosure, at least one sensor may include a vision sensor 110 of a depth camera configured to acquire depth values ​​of an object. The operation method of the AR device 100 may further include: in operation S1220, acquiring hand depth values ​​from a plurality of image frames successively acquired by the depth camera; and in operation S1230, identifying changes in the acquired depth values ​​over time. Acquiring at least one of hand movement information, position information, and biometric information in operation S220 may include: in operation S1240, identifying a hand-raising motion based on changes in the depth values.

[0214] In embodiments of this disclosure, obtaining sensing data in operation S210 may include: in operation S1410, receiving sensing data from sensors included in the wearable device 200 worn on the user's hand. Obtaining at least one of hand movement information, location information, and biometric information in operation S220 may include: in operation S1420, obtaining relative positional relationship information including at least one of the distance, direction, and orientation between the user's hand and the AR device based on the received sensing data. Determining whether the touch input is a valid input in operation S230 may include: in operation S1430, identifying whether the user's hand is located in a preset acceptable area based on the obtained relative positional relationship information; and in operation S1440, determining whether the touch input is a valid input based on the identification result.

[0215] In embodiments of this disclosure, at least one sensor may include an EEG sensor configured to acquire EEG signal data by detecting potential fluctuations in brainwaves. Acquiring the sensed data in operation S210 may include, in operation S1710, acquiring the EEG signal data by sensing potential fluctuations in brainwaves in the user's head using the EEG sensor. Determining whether the touch input is a valid input in operation S230 may include, in operation S1720, identifying negative feedback of brainwave potentials based on the acquired EEG signal data; and in operation S1730, determining whether the touch input is a valid input based on the identification result.

[0216] In embodiments of this disclosure, identifying negative feedback of EEG potentials in operation S1720 may include identifying error-related negative waves (ERNs) by monitoring fluctuations in the potential characteristics of EEG signal data over a preset time period starting from the time point when the touch input is received. Determining whether the touch input is a valid input in operation S230 may include determining the touch input as a valid input when an error-related negative wave is identified.

[0217] In embodiments of this disclosure, obtaining sensing data in operation S210 may include: in operation S1910, obtaining motion information about the movement of the AR device 100 via the motion sensor 130, and obtaining motion information about the vibration or movement of the AR device caused by adjustment input from the user adjusting the AR device via the motion sensor 130. Determining whether a touch input is a valid input in operation S230 may include: determining the touch input as invalid when motion information is obtained after the touch input is detected.

[0218] This disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an AR device 100 to cause the AR device 100 to perform: acquiring sensing data about hand movement using at least one sensor; detecting touch input to a touch interface 160; and determining whether the touch input is a valid input based on the sensing data about hand movement.

[0219] The program executed by the AR device 100 as described in this disclosure can be implemented using hardware elements, software elements, and / or combinations thereof. The program can be executed by any system capable of executing computer-readable instructions.

[0220] Software may include computer programs, code, instructions, or one or more combinations thereof, and may configure processing means to operate as needed or to instruct processing means independently or jointly.

[0221] Software can be implemented using a computer program that includes instructions stored in a computer-readable recording (or storage) medium. Examples of computer-readable recording media include magnetic storage media (e.g., read-only memory (ROM), floppy disks, hard disks, etc.) and optical recording media (e.g., optical disc ROM (CD-ROM) or digital versatile optical disc (DVD)). Computer-readable recording media can also be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. The medium can be read by a computer, stored in memory, and executed by a processor.

[0222] Computer-readable storage media may be provided in the form of non-transitory storage media. The term "non-transitory" means only that the storage medium is tangible and does not include signals, but does not help to distinguish any data stored in the storage medium as semi-permanent or temporary. For example, a non-transitory storage medium may include a buffer for temporarily storing data.

[0223] Furthermore, programs according to embodiments of this disclosure may be provided in a computer program product. The computer program product may be a commercial product that can be traded between a seller and a buyer.

[0224] Computer program products may include software programs and computer-readable storage media on which the software programs are stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application), which is electronically distributed by the manufacturer of AR device 100 or by an electronic marketplace (e.g., Samsung Galaxy store®). For electronic distribution, at least a portion of the software program may be stored in the storage medium or arbitrarily generated. In this case, the storage medium may be one of the manufacturer's server, the electronic marketplace's server, or a relay server temporarily storing the software program.

[0225] The computer program product in a system including AR device 100 and / or a server may include the storage medium of the server or the storage medium of AR device 100. Optionally, when a third device (e.g., a wearable device) communicatively connected to AR device 100 is present, the computer program product may include the storage medium of the third device. In another example, the computer program product may include the software program itself transmitted from AR device 100 to a third device or from the third device to an electronic device.

[0226] In this configuration, an executable computer program product from either the AR device 100 or the third device performs the method according to an embodiment of the present disclosure. Optionally, at least one executable computer program product from the AR device 100 and the third device performs the method according to an embodiment of the present disclosure in a distributed manner.

[0227] For example, AR device 100 can execute functions stored in memory 150 (see...). Figure 4 The computer program product in the present disclosure is used to control another electronic device communicatively connected to the AR device 100 to perform the method according to the embodiments of the present disclosure.

[0228] In another example, a third device may execute a computer program product to control an electronic device communicatively connected to the third device to perform a method according to an embodiment of the present disclosure.

[0229] When a third device executes a computer program product, the third device may download the computer program product from AR device 100 and execute the downloaded computer program product. Optionally, the third device may execute a pre-loaded computer program product to perform a method according to an embodiment of this disclosure.

[0230] Although this disclosure has been described with reference to some embodiments and accompanying drawings as described above, it will be apparent to those skilled in the art that various modifications and changes can be made to the embodiments. For example, the methods described above may be performed in a different order, and / or the components described above (such as computer systems or modules) may be combined in a different form than described above, and / or replaced or substituted by other components or their equivalents to obtain appropriate results.

Claims

1. An augmented reality (AR) device (100), comprising: At least one sensor; The touch interface (160) is configured to receive touch input; At least one processor (140) includes processing circuitry; as well as Memory (150) stores one or more instructions. The one or more instructions are executed individually or jointly by the at least one processor (140) to cause the AR device (100) to perform the following operations: Hand movement is sensed by at least one of the sensors. The determination of whether a touch input received through the touch interface (160) is a valid input is based on hand movement, and Based on the results of determining valid input, it is determined whether to perform the interaction corresponding to the touch input.

2. The AR device (100) according to claim 1, wherein: The at least one sensor includes a vision sensor (110) configured as a camera, and The one or more instructions are executed individually or jointly by the at least one processor (140) to cause the AR device (100) to perform the following operations: Multiple image frames of the user's hand are obtained by continuously capturing images of the hand with a camera. The obtained image frames are input into an artificial intelligence (AI) model, and feature points of the hand joints are detected from the image frames by performing visual recognition via the AI ​​model. The arm-raising motion is identified based on the movement of detected feature points over time. When touch input is detected within a preset time period starting from the time the hand movement is identified, the touch input is determined to be valid input.

3. The AR device (100) according to claim 1, wherein: The at least one sensor includes a vision sensor (110) of a depth camera configured to acquire depth values ​​of an object, and The one or more instructions are executed individually or jointly by the at least one processor (140) to cause the AR device (100) to perform the following operations: Hand depth values ​​are obtained from multiple image frames acquired sequentially by a depth camera. The change of the depth value obtained by identification over time, and The movement of raising an arm is identified based on changes in depth values.

4. The AR device (100) according to claim 1 further includes: The communication interface (170) is configured to perform data communication with an external device, wherein: The communication interface (170) receives sensing data from sensors included in the wearable device (200) worn on the user's hand, and The one or more instructions are executed individually or jointly by the at least one processor (140) to cause the AR device (100) to perform the following operations: Relative positional information is obtained based on the received sensing data. This relative positional information includes information about at least one of the distance, direction, and orientation between the user's hand and the AR device. Based on the obtained relative positional information, it identifies whether the user's hand is located in a preset acceptable area, and determines whether the touch input is valid based on the identification result.

5. The AR device (100) according to claim 1, wherein: The at least one sensor includes a brainwave sensor (120) configured to acquire electroencephalogram (EEG) signal data by detecting potential fluctuations in brainwaves, and The one or more instructions are executed individually or jointly by the at least one processor (140) to cause the AR device (100) to perform the following operations: EEG signal data is obtained by sensing the potential fluctuations of brain waves from the user's head using an electroencephalogram (120), and The system identifies negative feedback of brainwave potentials based on the obtained EEG signal data, and determines whether touch input is valid based on the identification results.

6. The AR device (100) according to claim 5, wherein, The one or more instructions are executed individually or jointly by the at least one processor (140) to cause the AR device (100) to perform the following operations: Error-related negative waves (ERNs) are identified by monitoring the fluctuations in the potential characteristics of EEG signal data over a preset time period starting from the moment touch input is received. When the ERN is identified, the touch input is determined to be valid input.

7. The AR device (100) according to any one of claims 1 to 6, wherein: The at least one sensor detects the movement of the AR device (100), and The one or more instructions are executed individually or jointly by the at least one processor (140) to cause the AR device (100) to perform the following operations: Motion information, including vibrations or movements of the AR device (100) caused by adjustment input from the user, is obtained by using a motion sensor (130). When motion information is obtained after detecting touch input, the touch input is determined to be invalid.

8. A method of operating an AR device (100), the method comprising: Sensing data about hand movement is obtained by using at least one sensor (S210). Detect touch input to the touch interface (160) of the AR device (100) (S220). as well as Based on the obtained sensing data, determine whether the touch input is a valid input (S230).

9. The operating method according to claim 8, wherein: The at least one sensor includes a vision sensor (110) configured as a camera, and Obtaining sensing data (S210) includes: obtaining multiple image frames of the user's hand by continuously capturing images of the hand with a camera (S610). The operation method of the AR device (100) further includes: The obtained multiple image frames are input into an artificial intelligence (AI) model, and feature points of the hand joints are detected from the multiple image frames using inference from the AI ​​model (S620); and The arm-raising motion is identified based on the movement of detected feature points over time, and Determining whether a touch input is a valid input (S230) includes: when a touch input is detected within a preset time period from the time point at which the hand lift movement is detected, determining the touch input as a valid input (S650).

10. The operating method according to claim 8, wherein: The at least one sensor includes a vision sensor (110) of a depth camera configured to acquire depth values ​​of an object, and The operation method of the AR device (100) further includes: The hand depth value is obtained from multiple image frames acquired sequentially by a depth camera (S1220). The depth value obtained by identification changes over time (S1230); and The hand-raising motion is identified based on the change in depth value (S1240).

11. The operating method according to claim 8, wherein: Acquire sensing data (S210), including: Receive sensing data from sensors included in a wearable device (200) worn on the user's hand (S1410); and Relative positional relationship information is obtained based on the received sensing data (S1420), the relative positional relationship information including information about at least one of the distance, direction, and orientation between the user's hand and the AR device, and Determining whether the touch input is a valid input (S230) includes: Based on the obtained relative positional relationship information, it is identified whether the user's hand is located within a preset acceptable area (S1430); and Based on the recognition result, determine whether the touch input is a valid input (S1440).

12. The operating method according to claim 8, wherein: The at least one sensor includes an EEG sensor configured to acquire electroencephalogram (EEG) signal data by sensing potential fluctuations in brain waves. Obtaining sensing data (S210) includes: obtaining EEG signal data by sensing the potential fluctuations of brain waves from the user's head using an EEG sensor (S1710), and Determining whether the touch input is a valid input (S230) includes: Negative feedback based on the acquisition of EEG signal data to identify brainwave potentials (S1720); and Based on the recognition result, determine whether the touch input is a valid input (S1730).

13. The operating method according to claim 12, wherein: The negative feedback for identifying brainwave potentials (S1720) includes: Error-related negative waves (ERNs) are identified by monitoring the fluctuations in the potential characteristics of EEG signal data over a preset time period starting from the moment touch input is received. Determining whether the touch input is a valid input (S230) includes: when the ERN is identified, determining the touch input as a valid input.

14. The operating method according to any one of claims 8 to 13, wherein: Acquire sensing data (S210), including: Motion information about the movement of the AR device (100) is obtained via the motion sensor (130); and Motion information about the vibration or movement of the AR device caused by adjustment input from the user for adjusting the AR device is obtained through the motion sensor (130) (S1910), and Determining whether a touch input is a valid input (S230) includes: when motion information is obtained after a touch input is detected, determining the touch input as an invalid input.

15. A computer program product, comprising a computer-readable storage medium, in, The storage medium includes instructions that are executed by the augmented reality (AR) device (100) to cause the AR device (100) to perform the following operations: Sensing data about hand movement is obtained by using at least one sensor; Detect touch input on the touch interface (160) of the AR device (100); as well as The determination of whether a touch input is valid is based on the obtained sensing data.

Images