Augmented reality device and operation method thereof

The augmented reality device addresses the challenge of intuitive object interaction by using image analysis to identify grip positions and rotations, allowing users to control devices with minimal movement and enhancing user experience through precise virtual guides.

WO2024248486A1PCT designated stage expired Publication Date: 2024-12-05SAMSUNG ELECTRONICS CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/007349
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-11-29
Filing Date
2024-05-29
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Current augmented reality devices lack efficient methods for users to intuitively interact with virtual environments by identifying grip positions and rotations of objects held in the hand, leading to cumbersome control mechanisms and unnecessary hand movements.

Method used

An augmented reality device equipped with a camera and processor that identifies objects and grip positions through image analysis, allowing users to perform corresponding operations by recognizing features and rotations, and providing virtual guides for precise control.

Benefits of technology

Enables users to control electronic devices with minimal hand and arm movement, providing intuitive interaction and convenient access to tool functions without needing complex menus, enhancing user experience and reducing unnecessary movements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024007349_05122024_PF_FP_ABST
    Figure KR2024007349_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure provides a method in which an augmented reality device operates. The method may include the step of identifying a first object from an image of the first object, acquired through a camera. The method may include the step of identifying a grip location at which a user grips the first object. The method may include the step of performing an action on the basis of the grip location.
Need to check novelty before this filing date? Find Prior Art

Description

Augmented reality device and method of operation thereof

[0001] The present disclosure relates to an augmented reality device and a method of operating the same, and relates to a method of identifying a command from an image acquired by the augmented reality device through a camera and operating accordingly, and to the augmented reality device.

[0002] Augmented Reality (AR) technology synthesizes virtual objects or information into a real-world environment, making them appear as if they were real objects in the physical world. Modern computing and display technologies have enabled the development of systems for AR experiences, in which digitally recreated images or portions of them are presented to the user in a way that makes them appear, or can be perceived, as, real.

[0003] Augmented reality (AR) technology allows virtual images to be overlaid onto real-world physical environments or objects. As interest in AR technology grows, development of various technologies that implement AR is actively underway. AR devices (e.g., smart glasses) utilizing AR technology are being used in everyday life for purposes such as information retrieval, route guidance, and camera photography.

[0004] Augmented reality devices are everyday life devices that can acquire data from the user's perspective. In particular, smart glasses and other devices can display virtual images by overlaying them on images of the physical environment of the real world through a transparent display.

[0005] In an augmented reality environment where real-world objects and virtual objects coexist, various information about the real-world environment can be provided to users through augmented reality images.

[0006] One aspect of the present disclosure provides a method for an augmented reality device to operate based on an image of a first object acquired through a camera. The method for operating the augmented reality device may include a step of identifying the first object from the image acquired through the camera. The method for operating the augmented reality device may include a step of identifying a gripping position where a user grips the first object. The method for operating the augmented reality device may include a step of performing a corresponding action based on the identified gripping position.

[0007] One aspect of the present disclosure provides an augmented reality device that operates based on an image of a first object acquired through a camera. The augmented reality device may include a camera configured to acquire a real-world scene image. The augmented reality device may include a display unit. The augmented reality device may include a memory storing a program including at least one command. The augmented reality device may include at least one processor. The at least one processor may identify a first object from an image acquired through the camera, identify a gripping position where a user grips the first object, and perform a corresponding action based on the identified gripping position.

[0008] One aspect of the present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may store instructions readable by an augmented reality device for executing at least one of the disclosed methods on the augmented reality device.

[0009] FIG. 1 is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to identify a command from an image of a first object acquired through a camera and operate accordingly.

[0010] FIG. 2A is a flowchart of a method of operating an augmented reality device according to one embodiment of the present disclosure.

[0011] FIG. 2b is a flowchart of a method of operating an augmented reality device according to one embodiment of the present disclosure.

[0012] FIG. 3A is a diagram for explaining an operation in which a command is identified according to a grip position of a first object according to one embodiment of the present disclosure.

[0013] FIG. 3b is a drawing for explaining an operation in which a corresponding command is identified when there are multiple grip positions of a first object according to one embodiment of the present disclosure.

[0014] FIG. 4 is a flowchart of an operation for identifying rotation of a first object according to one embodiment of the present disclosure.

[0015] FIG. 5A is a drawing for explaining an operation of identifying the rotation of a first object based on a user's hand image according to one embodiment of the present disclosure.

[0016] FIG. 5b is a drawing for explaining an operation of identifying the rotation of a first object based on an angle of the first object according to one embodiment of the present disclosure.

[0017] FIG. 5c is a drawing for explaining an operation of identifying the rotation of a first object based on a feature of the first object according to one embodiment of the present disclosure.

[0018] FIG. 6 is a diagram illustrating an example of an operation performed by an augmented reality device according to one embodiment of the present disclosure.

[0019] FIG. 7 is a diagram illustrating an example of an augmented reality device according to one embodiment of the present disclosure providing a user with a guide to a grip position.

[0020] FIG. 8 is a block diagram of an augmented reality device according to one embodiment of the present disclosure.

[0021] FIG. 9 is a drawing illustrating a glass-type augmented reality device according to one embodiment of the present disclosure.

[0022] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, for the purpose of clearly explaining the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar parts are designated with similar reference numerals throughout the specification.

[0023] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0024] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0025] Throughout this disclosure, when a part is referred to as "including" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0026] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where they are "directly connected" but also the cases where they are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes the possibility of including other components, unless otherwise specifically stated.

[0027] The expression “configured to” as used herein can be used interchangeably with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean something that is “specifically designed to” in terms of hardware. Instead, in some contexts, the expression “a system configured to” can mean that the system, together with other devices or components, is “capable of.” For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a generic-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in memory.

[0028] The functions related to 'Artificial Intelligence (AI)' according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an AI-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or AI models stored in memory. If one or more processors are AI-only processors, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.

[0029] The predefined operation rules or artificial intelligence model are characterized by being created through learning. Here, being created through learning means that a basic artificial intelligence model (or deep learning model) is trained using a plurality of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0030] An "artificial intelligence model (or deep learning model)" may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the calculation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be modified so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), such as, but not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0031] In this disclosure, 'Augmented Reality' refers to displaying a virtual image together within a physical environment space of the real world or displaying a real object and a virtual image together.

[0032] In the present disclosure, an 'augmented reality device' is a device capable of expressing augmented reality (AR), and can display images including physical objects and virtual objects existing in reality. The augmented reality device may be, for example, an augmented reality glasses in the shape of glasses worn by a user on the face, a head-mounted display device (HMD, Apparatus) worn on the head, or an augmented reality helmet, but is not limited thereto.

[0033] Human-computer interaction requires a means of inputting data into the system. Systems such as virtual reality, augmented reality, or mixed reality often lack dedicated input terminals.

[0034] Advances in computer-based image processing and image recognition technologies are leading to the development of diverse applications utilizing image processing technology. Among these diverse applications, gesture recognition technology, a method of information transmission, is undergoing steady technological development. Gesture recognition technology, which enables computers to autonomously analyze and recognize human behavior, can include touch gesture technology and spatial gesture technology. Specifically, touch gesture technology, which allows users to select items using input devices such as touchscreens, is increasing in use with the proliferation of smartphones. Furthermore, spatial gesture technology uses input devices such as cameras to analyze how the motion of a tracking target changes over time and interprets these changes in abstract terms. A key element of gesture recognition technology may be its ability to quickly and accurately recognize user intent.

[0035] Today, touch screens are used as the primary input mechanism for a wide range of interactive devices, such as smartphones, tablet PCs, and entertainment systems. In HID (Human Interface Device) mobile devices, touch signals can be input using the user's finger on smartphones, tablet PCs, etc., or using a stylus pen, a type of digitizer pen that allows handwriting or drawing. Input using a stylus pen allows for more precise input than finger input, and can support functions such as detailed drawing and writing.

[0036] According to one embodiment of the present disclosure, when a user using a wearable augmented reality device controls another electronic device connected to the augmented reality device, the user can control the other electronic device connected through gesture recognition by manipulating an object held in the hand.

[0037] In one embodiment, when a user of a wearable augmented reality device is writing (or drawing) on ​​a tablet PC (or mobile phone) connected to the augmented reality device using a stylus pen held in his hand, the user can change the properties (color, thickness, type, etc.) of the pen while writing (controlling another electronic device) through gesture recognition by rotating the stylus pen held in his hand.

[0038] In one embodiment, if a user using a wearable augmented reality device is holding a pen in his hand and watching a TV connected to the augmented reality device, the user can change the channel of the TV being watched or adjust the volume (control other electronic devices) through gesture recognition by rotating the pen held in his hand.

[0039] In one embodiment, when a user using a wearable augmented reality device is holding a pen in his hand and using a computer connected to the augmented reality device, the user can control the scrolling up and down of an application screen running on the computer (controlling another electronic device) through gesture recognition by rotating the pen held in his hand.

[0040] In addition to the examples described above, according to one embodiment of the present disclosure, various electronic devices connected to the augmented reality device can be controlled by rotating various objects held in the hand of a user using the augmented reality device.

[0041] The present disclosure will be described in detail with reference to the attached drawings below.

[0042] FIG. 1 is a drawing for explaining a method in which an augmented reality device according to one embodiment of the present disclosure identifies a command from an image of a first object (10) acquired through a camera and operates accordingly.

[0043] An augmented reality device is a device capable of expressing augmented reality, and may be configured, for example, as augmented reality glasses in the shape of glasses that a user (U) wears on the face. The components of the augmented reality device will be described in more detail later in FIGS. 8 and 9.

[0044] In one embodiment, the augmented reality device can acquire an image of a first object (10) through an included camera. The first object (10) may be an object held in the hand of a user (U) of the augmented reality device, and is not limited to an electronic device. In one embodiment, the first object (10) may be an object having a thin and long shape. The first object (10) may be an electronic device such as a digitizer pen or a stylus pen, or an object such as a pencil, a pen, chopsticks, or a straw, but is not limited thereto. When the first object (10) is an electronic device, the first object (10) may be connected to the augmented reality device. When the first object (10) is an electronic device, the first object (10) may be connected to or paired with the augmented reality device through short-range wireless communication.

[0045] The augmented reality device can identify an image of a first object (10) from an image acquired through a camera. Image analysis technology can be used to identify the image of the first object (10).

[0046] An augmented reality device can acquire a real-world scene image through a camera. For example, the camera may be an RGB camera, and the real-world scene image acquired through the camera may be an RGB image. The augmented reality device can then segment the acquired real-world scene image. The segmentation operation may refer to the operation of extracting objects from the image at the pixel level. The segmentation operation can recognize or isolate objects in images or videos. Through the segmentation operation, each pixel can be labeled to identify the location and shape of the object in the image, and which pixels belong to which object.

[0047] In one embodiment, the augmented reality device can recognize a user's (U) hand and an object held by the user (U) in a real-world scene image through segmentation operations. Object recognition is a type of pattern recognition, and may, for example, identify objects contained in an image or video using a trained neural network model.

[0048] The augmented reality device can identify a gripping position (GP) where a user (U) grips the first object (10) after identifying a first object (10) from an image. In one embodiment, the act of identifying the gripping position (GP) may include an act of detecting at least one feature point from the acquired image, identifying a hand of the user (U) based on the detected at least one feature point, and determining the gripping position (GP) based on the identified position of the hand of the user (U) and the position of the first object (10).

[0049] When a user (U) holds a first object (10), there may be two or more contact points between the first object (10) and the user (U). When the user (U) holds a pen with one hand, the thumb and index finger may come into contact with the pen. When there are multiple areas where the user (U) comes into contact with the first object (10), the step of identifying the grip position (GP) may identify a contact area with the highest priority among the multiple contact areas, and determine the grip position (GP) based on the contact area with the highest priority.

[0050] In one embodiment, when there are multiple points of contact between the first object (10) and the user (U), the grip position (GP) may be determined according to a preset rule. For example, when the preset rule is "the point of contact where the user's right thumb contacts the first object (10) is set as the grip position (GP)," the grip position (GP) may be determined according to the corresponding rule.

[0051] The case where a user (U) holds a first object (10) with multiple contact areas will be described in more detail with reference to FIG. 3b, which will be described later.

[0052] In one embodiment, the gripping position (GP) may be determined based on the position at which the first object (10) is gripped. The position at which the first object (10) is gripped may be determined based on what percentage (%) of the entire length area of ​​the first object (10) is gripped from the front (in the case of a stylus pen, the pen tip). The entire length area of ​​the first object (10) may be divided into percentage (%) sections, and each may correspond to a different operation of the augmented reality device. For example, when the entire length of the first object (10) is considered 100%, the area from the pen tip (0%) to 30% may correspond to the A-operation, the area from 30% to 45% may correspond to the B-operation, the area from 45% to 60% may correspond to the C-operation, and the remaining area from 60% to 100% may correspond to the D-operation. In this case, if the user (U) rotates the pen while gripping the 40% portion from the pen tip, the B-motion can be performed.

[0053] Referring to FIG. 1, a user (U) using a wearable augmented reality device may be writing (or drawing) on ​​a tablet PC (or mobile phone) connected to the augmented reality device using a stylus pen (first object (10)) held in a hand. For example, a grip position (GP) where the user (U) holds the first object (10), which is a stylus pen, may be identified as a 40% portion from the pen tip, and an action (A) for transmitting a command to the connected tablet PC to change the saturation of the pen during writing may correspond to a region from 30% to 45%. The user (U) may change the saturation of the pen during writing to a desired value by rotating the stylus pen held in the user's hand by a certain angle.

[0054] In one embodiment, the grip position (GP) may be determined based on which section of the preset sections of the first object (10) has been gripped. The first object (10) may be divided into preset sections, each of which may correspond to a different operation of the augmented reality device. For example, among the entire length of the first object (10), a section from the front (pen tip, 0 cm) to 5 cm may correspond to the A-motion, a section from 5 cm to 7 cm may correspond to the B-motion, a section from 7 cm to 9 cm may correspond to the C-motion, and a section from 9 cm onward may correspond to the D-motion. In this case, when the user (U) grips a section from the pen tip to 8 cm and rotates the pen, the C-motion may be performed.

[0055] Referring to FIG. 1, a user (U) wearing a wearable augmented reality device may be writing (or drawing) on ​​a tablet PC (or mobile phone) connected to the augmented reality device using a stylus pen (first object (10)) held in a hand. For example, a gripping position (GP) where the user (U) holds the first object (10), which is a stylus pen, may be identified as an 8 cm portion from the pen tip, and an action (A) for transmitting a command to change the saturation of the pen during writing to the connected tablet PC may correspond to an area of ​​7 cm to 9 cm. The user (U) may change the saturation of the pen during writing to a desired value by rotating the stylus pen held in the user's hand by a certain angle.

[0056] In one embodiment, the augmented reality device can provide a user (U) with a guide to a corresponding action type for each grip position through a virtual image. By providing the user (U) with a guide to which part of the first object (10) corresponds to which action, the user (U) can easily control the augmented reality device. The guide that can be provided to the user (U) through the virtual image will be described in more detail in FIG. 7, which will be described later.

[0057] The augmented reality device can identify the rotation of the first object (10). In one embodiment, the augmented reality device identifying the rotation of the first object (10) may identify the rotation degree (e.g., rotation angle) by which the user (U) rotates the first object (10). In one embodiment, the augmented reality device can identify a first point in time when the rotation of the first object (10) begins, calculate the accumulated momentum from the first point in time to the current point in time, and determine the rotation (rotation degree) based on the accumulated momentum. In one embodiment, the first point in time may be determined as the earliest point in time among consecutive points in time at which the momentum per hour is greater than or equal to a preset threshold value. The determined first point in time may be regarded as the point in time at which the user (U) begins the action of rotating the first object (10). By determining the first point in time, only the rotation of the first object (10) that the user (U) intends can be corresponded to the command. The operation of the augmented reality device to identify rotation (degree of rotation) will be described in more detail with reference to FIGS. 4 to 5c below.

[0058] In one embodiment, the first object (10) may include a sensor such as a touch sensor or a rotation detection sensor as an electronic pen. In this case, the grip position (GP) or rotation may be sensed and received from the first object (10). In one embodiment, the sensor included in the first object (10) may assist in verifying at least one of the grip position (GP) or rotation identified by the augmented reality device. In one embodiment, the augmented reality device may verify the grip position (GP) and rotation derived from an image acquired through a camera using sensed data received from the first object (10).

[0059] The augmented reality device can perform a corresponding action based on the identified grip position (GP). In one embodiment, the augmented reality device may also perform a corresponding action further based on the identified rotation (degree of rotation). The augmented reality device can display a user interface (UI) (A) corresponding to the performed action as a virtual image and provide it to the user. The user interface displayed as a virtual image can be displayed around the first object (10) or can be displayed to overlap a portion of the first object (10).

[0060] Referring to FIG. 1, a user (U) using a wearable augmented reality device may be writing (or drawing) on ​​a tablet PC (or mobile phone) connected to the augmented reality device using a stylus pen (first object (10)) held in a hand. In one embodiment, when the augmented reality device performs an operation of transmitting a command to the connected tablet PC for changing the saturation of the pen while writing based on an identified grip position (GP) (or additionally identified rotation), the augmented reality device may display a user interface (A) corresponding to the operation as a virtual image and provide it to the user.

[0061] In one embodiment, the augmented reality device can display a spectral image for saturation and an indicator indicating the current saturation as a virtual image around the first object (10) based on the grip position (GP), and can display an indicator indicating the saturation to be changed as a virtual image based on the degree of rotation of the first object (10). In one embodiment, when the current saturation has a value of 50, the augmented reality device can lower the saturation value according to the degree of rotation (rotation angle) for an action of the user (U) rotating the first object (10) clockwise, and can increase the saturation value according to the degree of rotation (rotation angle) for an action of the user (U) rotating the first object (10) counterclockwise.

[0062] The operation of the augmented reality device performing a corresponding action based on the identified grip position (GP) (or, additionally, the identified rotation) will be described in more detail with reference to FIG. 6 below.

[0063] FIGS. 2A and 2B are flowcharts of a method of operating an augmented reality device according to one embodiment of the present disclosure.

[0064] FIG. 2a illustrates an embodiment in which the augmented reality device operates based on a gripping position at which the user grips a first object, and FIG. 2b illustrates an embodiment in which the augmented reality device operates based on a gripping position at which the user grips the first object and a rotational degree at which the first object is rotated.

[0065] In steps 210 of FIG. 2A and 215 of FIG. 2B , the augmented reality device recognizes a first object. The augmented reality device can identify the first object from an image acquired through a camera. The first object may be an object held in the hand of a user of the augmented reality device, and may be a thin and long object. The operation of the augmented reality device identifying an image of the first object from an image acquired through the camera may utilize image analysis technology.

[0066] In one embodiment, an augmented reality device can recognize a user's hand and an object held in the hand from a real-world scene image through segmentation. Object recognition can be performed through pattern recognition, for example, identifying objects contained in an image or video using a trained neural network model.

[0067] In step 220 of FIG. 2A, the augmented reality device can identify a gripping position where the user grips the first object. Referring to step 225 of FIG. 2B, the augmented reality device can identify a gripping position where the user grips the first object and additionally identify a rotation of the first object. The rotation of the first object may correspond to a rotation degree or rotation angle by which the user rotates the first object.

[0068] In one embodiment, the augmented reality device can detect at least one feature point from an image acquired through a camera, identify a user's hand based on the detected at least one feature point, and determine a grip position based on a position of the user's hand and a position of a first object. In one embodiment, the augmented reality device can separate an image corresponding to the user's hand and an image of an object held by the user from the image acquired through the camera. The augmented reality device can then determine a grip position where the user is gripping the object based on relative positions of the image corresponding to the user's hand and the image of the object held by the user on a real scene image.

[0069] In one embodiment, the entire length of the first object may be segmented into percentage (%) sections, each corresponding to a different action of the augmented reality device. In one embodiment, the grip position may be determined based on the percentage of the entire length of the first object from the front that has been gripped.

[0070] In one embodiment, the first object may be divided into preset sections, each corresponding to a different operation of the augmented reality device. The gripping position may be determined based on which of the preset sections of the first object has been gripped.

[0071] In one embodiment, if there are multiple areas where the user makes contact with the first object, the augmented reality device may identify the highest priority contact area among the multiple contact areas where the user and the first object make contact, and determine the gripping position based on the highest priority contact area. Alternatively, if there are multiple areas where the user makes contact with the first object, the gripping area may be determined according to a preset rule.

[0072] In one embodiment, the augmented reality device may provide the user with guidance on the type of action corresponding to each grip location through a virtual image.

[0073] In one embodiment, the augmented reality device can further identify the rotation of the first object. The augmented reality device can perform corresponding actions based on the identified grip position and the identified rotation.

[0074] In one embodiment, the augmented reality device can identify a first point in time when the rotation of a first object begins, calculate an accumulated momentum from the first point in time to a current point in time, and determine a degree of rotation or an angle of rotation based on the accumulated momentum. In this case, the first point in time can be determined as the earliest point in time among consecutive times when the momentum per hour is greater than or equal to a preset threshold. The first point in time can be regarded as the point in time when the user initiates an action to rotate the object, and by determining the first point in time, only the rotation of the first object that is intended by the user can be corresponded to the command.

[0075] In one embodiment, the first object may include a touch sensor as an electronic device, and the augmented reality device may obtain sensing information from the first object through the touch sensor. The augmented reality device may determine at least one of a grip position and a rotation degree (rotation angle) based on the sensing information obtained through the touch sensor of the first object.

[0076] In one embodiment, the first object may include a rotation detection sensor as an electronic device, and the augmented reality device may obtain sensing information from the rotation detection sensor of the first object. The augmented reality device may also determine the degree of rotation (rotation angle) based on the sensing information obtained from the rotation detection sensor of the first object.

[0077] In one embodiment, a sensor included in the first object may assist in verifying at least one of the grip position or rotation identified by the augmented reality device. In one embodiment, the augmented reality device may verify the grip position and rotation degree (rotation angle) derived from an image acquired through a camera using sensing data received from the first object.

[0078] In step 230 of FIG. 2A, the augmented reality device performs a corresponding action based on the identified grip position. Referring to step 235 of FIG. 2B, the augmented reality device may also perform a corresponding action based on the identified grip position and the rotation (degree of rotation or angle of rotation) of the identified first object. In one embodiment, the augmented reality device may display a user interface (UI) corresponding to the performed action as a virtual image. The user interface displayed as a virtual image may be displayed around the first object or may be displayed to overlap a portion of the first object.

[0079] FIG. 3a is a drawing for explaining an operation in which a command is identified according to a grip position of a first object (10) according to one embodiment of the present disclosure.

[0080] The augmented reality device can identify a first object (10) from an image and then identify a gripping position where the user grips the first object (10). The gripping position can indicate which part of the first object (10) the user is gripping.

[0081] In one embodiment, a specific number of candidate values ​​for a grip location may be preset. The number of distinct sections for grip locations and the corresponding command for each grip location may be determined by the user's settings.

[0082] Referring to FIG. 3A, the candidate values ​​for the grip position (i.e., the number of sections in which the grip position is distinguished) can be set to three. In one embodiment, the grip position can be distinguished as a first section (P1), a second section (P2), or a third section (P3). Different commands may correspond to an action of a user grabbing the first section (P1) of a first object (10) and rotating the first object (10), an action of a user grabbing the second section (P2) of the first object (10) and rotating the first object (10), and an action of a user grabbing the third section (P3) of the first object (10) and rotating the first object (10).

[0083] In one embodiment, when a user of a wearable augmented reality device is writing (or drawing) on ​​a tablet PC (or mobile phone) connected to the augmented reality device using a stylus pen (first object (10)) held in a hand, an action of holding a first section (P1) and rotating the first object (10) may correspond to an action of changing the type of pen used for writing (e.g., ballpoint pen, fountain pen, colored pencil, highlighter, brush pen, etc.), an action of holding a second section (P2) and rotating the first object (10) may correspond to an action of changing the color of the pen used for writing, and an action of holding a third section (P3) and rotating the first object (10) may correspond to an action of changing the thickness of the pen used for writing.

[0084] In one embodiment, the gripping position may be determined based on what percentage (%) of the entire length region of the first object (10) from the front (in the case of a stylus pen, the pen tip) is gripped. The entire length region of the first object (10) may be divided into percentage sections, each of which may correspond to a different operation of the augmented reality device. In one embodiment, when the length of the first object (10) is regarded as 100%, the area from the pen tip (0%) to 20% may be determined as a first area (P1), the area from 20% to 70% may be determined as a second area (P2), and the remaining area from 70% to 100% may be determined as a third area (P3). Different operations of the augmented reality device may be corresponding to an operation of the user gripping the first area (P1), the second area (P2), or the third area (P3) and rotating the pen, respectively.

[0085] In one embodiment, a first zone (P1), a second zone (P2), and a third zone (P3) may be distinguished according to preset length ranges. In one embodiment, a portion from the front (pen tip, 0 cm) to 3 cm of the length of the first object (10) may correspond to the first zone (P1), a portion from 3 cm to 9 cm may correspond to the second zone (P2), and a portion from 9 cm to the remaining portion may correspond to the third zone (P3). Different actions of the augmented reality device may correspond to actions of the user rotating the pen while holding the first zone (P1), the second zone (P2), or the third zone (P3), respectively.

[0086] In one embodiment, an augmented reality device may provide a user with a guide to a preset section as an indicator through a virtual image. The user may have difficulty identifying the boundary between two adjacent sections. For example, the user may not be able to identify which section of the first object (10) constitutes the first section (P1), and may incorrectly grasp a specific section within the second section (P2) even though he or she intends to grasp the first section (P1). In this case, the user may intend to perform a first action (e.g., change the type of pen being used for writing), but may grasp the wrong section, resulting in the user performing a second action (e.g., change the color of the pen being used for writing).

[0087] To prevent such grip position errors, the augmented reality device can provide the user with a guide for each distinct zone. In one embodiment, the augmented reality device can provide a virtual image with indicator lines indicating the boundaries of the first zone (P1) and the second zone (P2) and the boundaries of the second zone (P2) and the third zone (P3), or can display the first zone (P1), the second zone (P2), and the third zone (P3) with different colors or hatches. By providing the user with a guide distinguishing different zones on the first object (10) in the virtual image, the user can be guided to grasp the correct location.

[0088] In one embodiment, the number of candidate values ​​for the grip location (i.e., the number of segments in which the grip location is distinguished) may be set to n, where n may be an integer greater than 1 (n=1, 2, 3, etc.). The maximum number of segments in which the grip location is distinguished is not limited, but the number of segments may not be too many, such as about 6 or less, for the augmented reality device to accurately identify the grip location from an image acquired through a camera.

[0089] FIG. 3b is a drawing for explaining an operation in which a corresponding command is identified when the grip positions of the first object (10) according to one embodiment of the present disclosure are multiple (GP1, GP2).

[0090] When a user (U) holds a first object (10), there may be two or more points of contact between the first object (10) and the user (U). In one embodiment, when the user (U) holds a pen with one hand, the thumb and index finger of one hand may contact the pen. Alternatively, when the user (U) holds a pen with two hands, the thumbs and index fingers of both hands may contact the pen.

[0091] In one embodiment, when there are multiple areas where the user (U) contacts the first object (10), the step of identifying the grip location may include identifying a contact area with the highest priority among the multiple contact areas, and determining the grip location based on the contact area with the highest priority.

[0092] In one embodiment, the priority of a contact area may be determined based on the finger corresponding to the contact area. In one embodiment, if the user (U) is right-handed, the contact area of ​​the right thumb may have the highest priority. In this embodiment, the augmented reality device may determine the area (GP2) where the user (U)'s right thumb contacts the first object (10) as the gripping position.

[0093] In one embodiment, the priority of the contact area may be determined based on the distance from the pen tip. In one embodiment, the augmented reality device may determine the contact point closest to the pen tip among the multiple contact points where the user (U) and the first object (10) come into contact as the gripping position.

[0094] In one embodiment, a grip position may be determined by a pair of multiple contact points. In one embodiment, a contact point with the highest priority among the multiple contact points may be determined as the grip position, but a set of multiple contact points may also be determined as the 'grab position.'

[0095] In one embodiment, when a user (U) holds and rotates a first object (10) with both hands, the gripping positions may be determined as a pair of contact points where the thumbs of both hands of the user (U) contact the first object (10). A different command may be matched to the gripping and rotating the first object (10) with both hands than to the gripping and rotating the first object (10) with one hand at each hand location. In one embodiment, when locations on the first object (10) are distinguished into a first zone and a second zone, different commands may be matched when the gripping position is a single first zone, when the gripping position is a single second zone, and when the gripping positions are a pair of the first zone and the second zone, respectively.

[0096] In one embodiment, a user using a wearable augmented reality device is holding a ballpoint pen in his or her hand and watching a TV connected to the augmented reality device. The user can change the channel of the TV being watched by holding and rotating a first area of ​​the ballpoint pen, adjust the volume of the TV being watched by holding and rotating a second area of ​​the ballpoint pen, and change the input source of the TV by holding and rotating the first area and the second area with each hand.

[0097] In one embodiment, the cases in which a user (U) holds a first object (10) with both hands can be classified into three cases.

[0098] - When the user (U) supports the first object (10) with the left hand and rotates the first object (10) using only the right hand (first embodiment)

[0099] - When the user (U) supports the first object (10) with the right hand and rotates the first object (10) using only the left hand (second embodiment)

[0100] - When the user (U) rotates the first object (10) with both hands (3rd embodiment)

[0101] The three examples above can be identified based on the change in the position of the user's (U's) thumb in the image. In the first embodiment, if the user's (U's) left thumb and index finger do not change in position and the user's (U's) right thumb and index finger move clockwise with respect to the first object (10), the augmented reality device can identify that the user's (U's) is rotating the first object (10) using the right hand. Alternatively, if the user's (U's) right thumb and left thumb both rotate with respect to the first object (10), the augmented reality device can identify that the user's (U's) is rotating the first object (10) using both hands.

[0102] The gripping position can be determined depending on which hand rotates the first object (10). That is, if it is determined that the user (U) supports the first object (10) with the left hand and rotates the first object (10) using only the right hand, the area (GP2) where the right hand comes into contact with the first object (10) can be determined as the gripping position. If it is determined that the user (U) supports the first object (10) with the right hand and rotates the first object (10) using only the left hand, the area (GP1) where the left hand comes into contact with the first object (10) can be determined as the gripping position. If the user (U) rotates the first object (10) with both hands, the area (GP2) where the right hand comes into contact with the first object (10) and the area (GP1) where the left hand comes into contact with the first object (10) can both be gripping positions.

[0103] In this way, a grip position consisting of a single contact point can be determined from a plurality of contact points where the user (U) and the first object (10) come into contact, or a grip position consisting of a set of a plurality of contact points can be determined. The grip positions can be distinguished based on the contact position on the first object (10) and the number of contact points that come into contact with the first object (10). Different commands can be associated with different grip positions.

[0104] FIG. 4 is a flowchart of an operation for identifying rotation of a first object according to one embodiment of the present disclosure.

[0105] The augmented reality device can identify the rotation of the first object. The augmented reality device identifying the rotation of the first object may identify the degree or angle of rotation by which the user of the augmented reality device rotates the first object.

[0106] In step 410, the augmented reality device can calculate the amount of movement per hour. The amount of movement per hour can be calculated based on images acquired by the augmented reality device through its camera. The augmented reality device can capture images of a real-world scene at regular intervals.

[0107] In one embodiment, a "frame" may refer to a single still image captured by a camera. When capturing a series of scenes of a first object, the "next frame" may refer to image data captured one period (dt) after the current image, and the "previous frame" may refer to image data captured one period (dt) before the current image.

[0108] An augmented reality device can extract feature points from a first image acquired at a specific point in time t and a second image acquired at a point in time (t+dt), which is the next frame. In one embodiment, the feature points may include feature points according to the distal point of the user's hand, the boundary point of a fingernail, or the shape or form of a first object held by the user. The augmented reality device can match the feature points included in the first image with the feature points included in the second image. The matching of the feature points may correspond to the same feature points. The augmented reality device can calculate the amount of momentum per hour by comparing the feature points of the first image with the feature points corresponding to the feature points in the second image. The amount of momentum per hour can be calculated using the following mathematical equation 1.

[0109]

[0110] In mathematical equation 1, represents the amount of movement of the first object over a certain period of time, dx represents the change of the feature point in the horizontal direction of the first object from the first image to the second image, dy represents the change of the feature point in the vertical direction of the first object from the first image to the second image, and dt represents the time from the time when the first image is acquired to the time when the second image is acquired.

[0111] At step 420, the augmented reality device determines that the momentum per second of the first object at a specific point in time is a threshold ( ) can be judged as being ideal ( ). Threshold( ) can represent the minimum amount of momentum required to be determined as a rotational motion. When a user holds a first object, micro-rotations that are not intended by the user may occur. To determine that rotational motions that are not intended by the user are noise, a threshold value for determining that a rotational motion has occurred can be set.

[0112] In one embodiment, the augmented reality device may determine the earliest point in time among consecutive periods where the momentum per hour is greater than or equal to a preset threshold as the first point in time. The first point in time may correspond to the point in time when the rotation of the first object begins. In other words, the first point in time may be considered the point in time when the user initiates the action of rotating the first object. By determining the first point in time, only rotations of the first object that are intended by the user can be corresponded to the command.

[0113] At step 430, the augmented reality device calculates the accumulated momentum ( ) can be calculated. Cumulative momentum ( ) can be calculated using the mathematical formula 2 below.

[0114]

[0115] In step 440, the augmented reality device may determine a degree of rotation or an angle of rotation of the first object based on the accumulated momentum of the first object. The augmented reality device may then be controlled based on the value of the degree of rotation or the angle of rotation.

[0116] In one embodiment, when a user using a wearable augmented reality device is watching a TV connected to the augmented reality device while holding a ballpoint pen in his / her hand, the user can change the channel of the TV being watched through gesture recognition by rotating the ballpoint pen held in his / her hand. At this time, the TV channel can be increased or decreased by 1 every time the user rotates the ballpoint pen by a preset angle. In one embodiment, when the TV channel is set to increase or decrease by 1 every time the ballpoint pen is rotated by 10°, the channel can be increased by 1 when the user rotates the ballpoint pen by 10° clockwise, and the channel can be decreased by 1 when the user rotates the ballpoint pen by 10° counterclockwise.

[0117] FIG. 5A is a drawing for explaining an operation of identifying the rotation of a first object based on a user's hand image according to one embodiment of the present disclosure, and FIG. 5B is a drawing for explaining an operation of identifying the rotation of a first object based on an angle of the first object according to one embodiment of the present disclosure.

[0118] Referring to FIGS. 5A and 5B, an augmented reality device can acquire a first image at a specific point in time t1 (a) and can acquire a second image at a point in time (t1+dt) after the next frame dt (b). The augmented reality device can extract feature points from the first image and the second image, respectively.

[0119] Referring to Fig. 5a, the feature points may be related to the user's hand, such as the distal end of the user's hand or the boundary of a fingernail. Referring to Fig. 5b, the feature points may include feature points according to the shape, form, etc. of a first object (e.g., a ballpoint pen) held by the user.

[0120] The augmented reality device can match feature points included in a first image with feature points included in a second image. Matching of feature points may correspond to identical feature points.

[0121] Referring to FIG. 5A, the augmented reality device can calculate the amount of movement per hour by comparing the position of the user's thumb tip point in the first image with the position of the user's thumb tip point in the second image (see Equation 1).

[0122] Referring to FIG. 5b, the augmented reality device may also calculate the momentum per hour by comparing the angle of the first object in the first image and the angle of the first object in the second image (see Equation 1).

[0123] FIG. 5c is a drawing for explaining an operation of identifying the rotation of a first object based on a feature of the first object according to one embodiment of the present disclosure.

[0124] Referring to FIG. 5c, the augmented reality device can acquire a first image at a specific point in time t1 (a) and can acquire a second image at a point in time (t1+dt) after the next frame dt (b). The augmented reality device can extract feature points from each of the first image and the second image.

[0125] Referring to FIG. 5c, the feature points may include feature points according to the shape, form, etc. of the first object (e.g., a ballpoint pen) held by the user. In one embodiment, when there is text (51a, 51b) engraved or printed on the side of the first object, the corresponding portion may become a feature point. In one embodiment, when the first object has an angular shape, the corner portions (52a, 52b, 53a, 53b) created by the angular shape may become feature points.

[0126] The augmented reality device can match feature points included in a first image with feature points included in a second image. Matching of feature points may correspond to the same feature points. Referring to FIG. 5c, engraved or printed text (51a) of the first image may be matched to corresponding engraved or printed text (51b) of the second image, and corner portions (52a, 53a) of the first image may be matched to corresponding corner portions (52b, 53b) of the second image, respectively.

[0127] The augmented reality device can calculate the amount of movement per second by comparing the positions of feature points in the first image with the positions of corresponding feature points in the second image (see Equation 1).

[0128] In one embodiment of the present disclosure, the method for calculating the momentum per hour at a specific point in time is not limited to the embodiments illustrated in FIGS. 5A to 5C described above, and the momentum per hour at a specific point in time may also be calculated based on changes in various feature points or changes in various attributes that can be acquired through a camera.

[0129] The augmented reality device determines whether the momentum per hour at a given point in time is a threshold ( ) can be judged as being ideal ( ). Threshold( ) can represent the minimum amount of momentum required to be determined as a rotational motion. When a user holds a first object, micro-rotations of the first object that are not intended by the user may occur. In order to determine that rotational motions that are not intended by the user are noise, a threshold value for determining that a rotational motion has occurred can be set.

[0130] In one embodiment, the augmented reality device may determine the earliest point in time among consecutive periods where the momentum per hour is greater than or equal to a preset threshold as the first point in time. The first point in time may correspond to the point in time when the rotation of the first object begins. In other words, the determined first point in time may be viewed as the point in time when the user initiates the action of rotating the first object. By determining the first point in time, only rotations of the first object intended by the user can be corresponded to the command.

[0131] The augmented reality device then calculates the accumulated momentum ( ) can be calculated, and the degree of rotation or the angle of rotation of the first object can be determined based on the accumulated momentum of the first object. Thereafter, the augmented reality device can be controlled according to the value of the determined degree of rotation or the angle of rotation.

[0132] FIG. 6 is a diagram illustrating an example of an operation performed by an augmented reality device according to one embodiment of the present disclosure.

[0133] According to one embodiment of the present disclosure, a user using a wearable augmented reality device can control another electronic device (a second object) connected to the augmented reality device through gesture recognition by manipulating an object (a first object) held in a hand.

[0134] In one embodiment, when a user of a wearable augmented reality device is writing (or drawing) on ​​a tablet PC (or mobile phone) connected to the augmented reality device using a stylus pen held in his hand, the user can change the properties (color, thickness, type, etc.) of the pen while writing (controlling another electronic device) through gesture recognition by rotating the stylus pen held in his hand.

[0135] Referring to FIG. 6, a user can change the type of pen being used for writing or the thickness of the pen by rotating the stylus pen held in the user's hand. In one embodiment, the user can also perform screen switching of an application running on another electronic device (second object) connected to the augmented reality device by rotating the stylus pen held in the user's hand.

[0136] In one embodiment, if a user using a wearable augmented reality device is holding a pen in his hand and watching a TV connected to the augmented reality device, the user may change the channel of the TV being watched or adjust the volume (control other electronic devices) through gesture recognition by rotating the pen held in his hand.

[0137] In one embodiment, the augmented reality device may display an augmented reality image including a user interface corresponding to an action performed. The augmented reality device may perform an action of controlling another connected electronic device (a second object) in response to a user's action of rotating a first object.

[0138] The augmented reality device can determine an area in which to display an augmented reality image based on the location of the first object image within the real scene image. In one embodiment, the augmented reality device can display a virtual image including a user interface around the first object, taking into account the location of the first object within the acquired real scene image.

[0139] In one embodiment, an augmented reality image including a user interface may be displayed near a first object or may be displayed so as to overlap at least a portion of the first object image.

[0140] Meanwhile, the augmented reality image including the user interface may not be displayed near the image of the first object. In one embodiment, when attempting to manipulate a second object by rotating the first object, the second object may be manipulated immediately without displaying the augmented reality image, or the augmented reality image may be displayed near the second object or overlapped with the image of the second object. When the augmented reality image is displayed near the second object, the second object may be considered a manipulation target.

[0141] In one embodiment, when a user rotates a pen (first object) held in his / her hand to change the channel of a TV (second object) that he / she is watching, the channel of the TV (second object) may be changed immediately without displaying an augmented reality image, or a user interface corresponding to the channel changing action may be displayed as a virtual image near the TV (second object).

[0142] In one embodiment, a user using a wearable augmented reality device may be writing (or drawing) on ​​a tablet PC (or mobile phone) connected to the augmented reality device using a stylus pen held in a hand. When the augmented reality device performs an operation of transmitting a command to the connected tablet PC for changing the type of the pen being written with based on the identified grip position and rotation, the augmented reality device may provide the user with a user interface corresponding to the operation by displaying a virtual image. The augmented reality device may display a list image of the 'pen type' and an indicator indicating the current type as a virtual image around the stylus pen based on the grip position, and may display an indicator (> <) indicating the type to be changed based on the degree of rotation or the angle of rotation as a virtual image. For example, when the current type is 'Eyedropper,' the augmented reality device may change the type to 'Brush,' 'Eraser,' 'Gradient,' 'Fill,' or 'Stamp,' etc., based on the degree of rotation or the angle of rotation in response to the user's operation of rotating a first object.

[0143] In one embodiment, a user using a wearable augmented reality device may write (or draw) on a tablet PC (or mobile phone) connected to the augmented reality device using a stylus pen held in a hand. For example, when the augmented reality device performs an action of transmitting a command to the connected tablet PC to change the thickness of the pen while writing based on an identified grip position and rotation, the augmented reality device may display a user interface corresponding to the action as a virtual image and provide it to the user. For example, the augmented reality device may display a spectrum image for 'pen thickness' and an indicator indicating the current thickness as a virtual image around the stylus pen based on the grip position, and may display an indicator (> <) indicating the thickness to be changed based on the degree of rotation or the angle of rotation as a virtual image. For example, when the current pen thickness is '20', the augmented reality device may increase or decrease the thickness depending on the degree of rotation or the angle of rotation in response to an action of the user rotating a first object. In one embodiment, when the user rotates the first object clockwise, the pen thickness may increase, and when the user rotates the first object counterclockwise, the pen thickness may decrease. The rotation direction and the increase or decrease in pen thickness may be performed in the opposite direction to the above-described embodiments. In one embodiment, the thickness of the pen may increase or decrease by 1 each time the user rotates the stylus pen by a preset angle. For example, if the pen thickness is set to increase or decrease by 1 each time the pen is rotated by 10°, the pen thickness may increase by 1 when the user rotates the stylus pen by 10° clockwise, and the pen thickness may decrease by 1 when the user rotates the stylus pen by 10° counterclockwise.

[0144] In one embodiment, a user using a wearable augmented reality device may control a tablet PC (or mobile phone) connected to the augmented reality device using a stylus pen held in a hand. When the augmented reality device performs an operation of transmitting a command to the connected tablet PC for switching the execution screen of an application running on the tablet PC based on the identified grip position and rotation, the augmented reality device may display a user interface corresponding to the operation as a virtual image and provide it to the user. The augmented reality device may display a virtual image in the form of an icon indicating the type of application running on the tablet PC around the stylus pen based on the grip position, and may display an indicator (O) indicating a target application to be switched to a screen based on the degree of rotation or the angle of rotation as a virtual image. The augmented reality device may change the target application to be switched to a screen based on the degree of rotation or the angle of rotation in response to the user's operation of rotating a first object.

[0145] FIG. 7 is a diagram illustrating an example of an augmented reality device according to one embodiment of the present disclosure providing a user with a guide to a grip position.

[0146] In one embodiment, the augmented reality device can provide a user with a guide on the type of action corresponding to each grip position through a virtual image using indicators (I1, I2, I3, I4). The augmented reality device can display the indicators (I1, I2, I3, I4) on the UI to provide the user with a guide on which part of the first object (10) corresponds to which action. The user can control the augmented reality device based on the indicators (I1, I2, I3, I4).

[0147] In one embodiment, the augmented reality device may provide a user with a guide to a preset section using indicators (I1, I2, I3, I4) through a virtual image. The user may have difficulty identifying the boundary between two adjacent sections. The user may not be able to identify which section of the first object (10) is the first section (I1), and may intentionally grasp the first section (I1) but incorrectly grasp a specific section within the second section (I2). In this case, the user may intend to perform a first action (e.g., change the type of pen being used for writing), but may grasp the wrong section, resulting in a second action (e.g., change the color of the pen being used for writing).

[0148] To avoid such grip position errors, the augmented reality device can provide the user with guides (I1, I2, I3, I4) for each distinct zone. In one embodiment, the augmented reality device can provide a virtual image with indicator lines indicating the boundaries of two adjacent zones, or can display each zone with a frame, different colors, or hatching. By providing the user with guides (e.g., indicators I1, I2, I3, I4) distinguishing different areas on the first object (10) in a virtual image, the user can be guided to grasp the correct location.

[0149] The guidance provided to the user is not limited to the examples described above, and the augmented reality device can provide the user with information about the location of the grip through various methods.

[0150] Referring to FIG. 7, in one embodiment, the commands corresponding to the action of rotating the first object can be divided into four types according to the grip position. The augmented reality device can provide guidance to the user by displaying the grip positions corresponding to each command in different colors. In one embodiment, the virtual image may further include indicators indicating which action each area corresponds to.

[0151] FIG. 8 is a block diagram of an augmented reality device (100) according to one embodiment of the present disclosure.

[0152] Referring to FIG. 8, the augmented reality device (100) may include a communication interface (110), a camera (120), a sensor (130), a processor (140), a memory (150), and an output interface (160). The communication interface (110), the camera (120), the sensor (130), the processor (140), the memory (150), and the output interface (160) may be electrically and / or physically connected to each other, respectively. Meanwhile, the components of the augmented reality device (100) are not limited to those illustrated in FIG. 8. The augmented reality device (100) may be implemented with more components than those illustrated in FIG. 8, or may be implemented with fewer components than those illustrated in FIG. 8.

[0153] In one embodiment of the present disclosure, the augmented reality device (100) may be implemented as augmented reality glasses worn on a user's head, in which case the augmented reality device (100) may further include a power supply (e.g., a battery) that supplies driving power to a communication interface (110), a camera (120), a sensor (130), a processor (140), and an output interface (160). In one embodiment, the augmented reality device (100) may not include a speaker (164).

[0154] The communication interface (110) is configured to transmit and receive data with a server or an external device (e.g., a device of interest) via a wired or wireless communication network. The communication interface (110) may perform data communication with the server or the external device using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication. However, the present invention is not limited thereto, and when the augmented reality device (100) is implemented as a wearable device such as smart glasses, the communication interface (110) may transmit and receive data with a server or an external device through a network that follows a mobile communication standard such as a communication method using CDMA, WCDMA, 3G, 4G (LTE), 5G Sub 6, and / or millimeter wave (mmWave).

[0155] In one embodiment, the communication interface (110) may receive auxiliary sensing information from a first object or transmit a command to a second object to be manipulated under the control of the processor (140). The communication interface (110) may provide the received sensing information of the first object to the processor (140).

[0156] In one embodiment, the communication interface (110) is connected to at least one external object via a short-range communication method such as Bluetooth or Wi-Fi Direct, and may transmit and receive various information with the external object.

[0157] The camera (120) is configured to acquire two-dimensional (2D) image data by photographing a real space. The camera (120) may be implemented with a small form factor so that it can be mounted on the augmented reality device (100) and may be a lightweight RGB camera that consumes low power. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the camera (120) may be implemented with any type of camera known in the art, such as an RGB-depth camera including a depth estimation function, a Dynamic Vision Sensor camera, a stereo fish-eye camera, a grayscale camera, or an infrared camera. In one embodiment, at least some of the cameras (120) may be arranged in a direction facing a user and configured to photograph the user's face.

[0158] The camera (120) may include a lens module, an image sensor, and an image processing module. The camera (120) may acquire still images or video of a real-world scene by means of an image sensor (e.g., CMOS or CCD). The video may include multiple image frames acquired in real time by photographing a real-world area through the camera (120). The image processing module may encode still images composed of a single image frame acquired through the image sensor or video data composed of multiple image frames and transmit them to the processor (140).

[0159] According to one embodiment of the present disclosure, capturing a real scene image may include an operation of the augmented reality device (100) to control a camera (e.g., a camera including an image sensor and a lens) (120) provided in the augmented reality device (100) to convert an optical image formed through the lens into an electrical signal to acquire an image. In one embodiment, one or more processors (140) may control the camera (120) provided in the augmented reality device (100) to capture a surrounding area of ​​the augmented reality device (100) to acquire an image (e.g., a captured image) including one or more frames. Here, the image may include a live-view image.

[0160] The sensor (130) may include sensors configured to detect real space, location, situation, or user information. In one embodiment of the present disclosure, the sensor (130) may include, but is not limited to, an eye tracking sensor, an Inertial Measurement Unit (IMU) sensor, a Global Positioning System (GPS) sensor, a Bluetooth Low Energy (BLE) sensor, an Ultra Wide Broadband (UWB) sensor, or other sensors capable of sensing various signals.

[0161] The processor (140) can execute one or more instructions of a program stored in the memory (150). The processor (140) may be configured with hardware components that perform arithmetic, logic, and input / output operations and image processing. Although FIG. 8 illustrates a single processor (140), the present invention is not limited thereto, and the processor (140) may be configured with one or more multiple elements. The processor (140) may be a general-purpose processor such as a CPU (Central Processing Unit), an AP (Application Processor), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), a VPU (Vision Processing Unit), or an NPU (Neural Processing Unit), a graphics-only processor, or an NPU (Artificial Intelligence)-only processor. The processor (140) may be controlled to process input data according to a predefined operating rule or artificial intelligence model. Alternatively, when the processor (140) is an artificial intelligence-only processor, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0162] The memory (150) may be configured as at least one type of storage medium, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk.

[0163] The memory (150) may store instructions related to functions and / or operations for the operation of the augmented reality device (100). In one embodiment, the memory (150) may store at least one of instructions, algorithms, data structures, program codes, and application programs that can be read by the processor (140). The instructions, algorithms, data structures, and program codes stored in the memory (150) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0164] The processor (140) can execute instructions or program codes stored in the memory (150) and control the overall operation of the augmented reality device (100). The processor (140) can perform operations according to one embodiment of the present disclosure. In one embodiment, the processor (140) can control the communication interface (110), the camera (120), the sensor (130), and the output interface (160) overall by executing the programs stored in the memory (150).

[0165] The processor (140) may be configured with hardware components that perform arithmetic, logic, and input / output operations and signal processing. The processor (140) may be configured with at least one of, for example, a central processing unit (CPU), a microprocessor, a graphic processing unit (GRAP), an application specific integrated circuits (ASICs), a digital signal processor (DSPs), a digital signal processing device (DSPDs), a programmable logic device (PLDs), and a field programmable gate array (FPGAs), but is not limited thereto.

[0166] In one embodiment, the processor (140) may execute one or more commands stored in the memory (150) to acquire a real scene image through the camera (120), identify a first object from the acquired image, identify a gripping position where a user grips the first object, and perform a corresponding action based on the identified gripping position.

[0167] The output interface (160) is configured to output a user interface (UI) corresponding to an operation performed by the augmented reality device (100) under the control of the processor (140) as an augmented reality image (virtual image) or to output an audio signal. The output interface (160) may include a display unit (162) and a speaker (164).

[0168] The display unit (162) may be composed of at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display.

[0169] In one embodiment, when the augmented reality device (100) is configured as augmented reality glasses, the display unit (162) may be configured as a lens optical system and may include a waveguide and an optical engine. The optical engine may be configured as a projector that generates light of an augmented reality image composed of text, icons, virtual images, etc. and projects the light onto a waveguide. The optical engine may include an image panel, an illumination optical system, a projection optical system, etc. In one embodiment, the optical engine may be disposed on a frame or a temple of the augmented reality glasses.

[0170] The speaker (164) is configured to output an audio signal. In one embodiment, the speaker (164) can output a voice message or notification sound under the control of the processor (140).

[0171] FIG. 9 is a diagram illustrating a glass-type augmented reality device (900) according to one embodiment of the present disclosure.

[0172] Referring to FIG. 9, an augmented reality device (900) capable of displaying information about a device of interest through an augmented reality image is illustrated. The augmented reality device (900) is a device capable of providing augmented reality (AR) related services, and may generally include AR glasses in the shape of glasses worn on the user's face, a head mounted display (HMD) worn on the head, a virtual reality headset (VRH), or an AR helmet. In the case of a head mounted device, a display can be placed in front of the user's eyes to provide a very large screen to the user, and the screen moves according to the user's movements, so that a realistic virtual world can be provided.

[0173] In one embodiment, a user may wear an augmented reality device (900) capable of displaying visual augmented reality content. The augmented reality device (900) may include an audio module capable of providing audio augmented reality content to the user. In one embodiment, the augmented reality device (900) may include one or more cameras capable of capturing images and videos of an environment. The augmented reality device (900) may include an eye tracking system to determine the user's vergence distance. In one embodiment, the augmented reality device (900) may include a lightweight head-mounted display (HMD) (e.g., goggles, glasses, visor, etc.). In one embodiment, the augmented reality device (900) may include a device such as a lightweight portable display device or one or more laser projection glasses (e.g., glasses capable of projecting a low-powered laser onto a user's retina to project and display images or depth content to the user).

[0174] In one embodiment, the augmented reality device (900) may provide an AR service that outputs at least one virtual object to be superimposed on an area determined to be a user's field of view (FOV). For example, the area determined to be a user's field of view may be an area determined to be recognizable by a user wearing the augmented reality device (900) through the augmented reality device (900), and may be an area including all or at least a portion of the display of the augmented reality device (900). In one embodiment, the augmented reality device (900) may include a plurality of transparent members (920, 930) corresponding to each of the user's eyes.

[0175] In one embodiment, the augmented reality device (900) may include a display module (914), a camera, an audio output, and a support (921, 922).

[0176] The camera can capture images corresponding to the user's field of view or measure the distance to an object. The camera may correspond to the camera (120) of FIG. 8 described above. In one embodiment, the camera may be used for head tracking and spatial recognition. The camera may also recognize the user's movements.

[0177] In one embodiment, the camera may further include an 'ET (eye tracking) camera (912)' in addition to the camera (913) that acquires an image corresponding to the user's field of view, i.e., a real scene image, detects an object, or is used for spatial recognition. In one embodiment, the ET camera (912) may be used to detect and track the user's pupils. The ET camera (912) may be used to adjust the center of a virtual image projected on the augmented reality device (900) so that it is positioned according to the direction in which the pupils of the user wearing the augmented reality device (900) are looking. For example, the ET camera (912) may use a GS (Global shutter) camera to detect the pupil and track rapid eye movements without delay. The ET camera (912) may separately include a left-eye camera (912-1) and a right-eye camera (912-2).

[0178] In one embodiment, the display module (914) may include a first display (930) and a second display (920). The display module (914) may correspond to the display unit (162) of FIG. 8 described above. A virtual object output through the display module (914) may include information related to an application program running on the augmented reality device (900), information about a device of interest, or information related to an external object located in a real space corresponding to an area determined to be within the user's field of view. For example, the augmented reality device (900) may identify an external object included in at least a portion corresponding to an area determined to be within the user's field of view among image information related to a real space acquired through the camera (913). The augmented reality device (900) may output a virtual object related to the external object identified in at least a portion through an area determined to be within the user's field of view among the display areas of the augmented reality device (900). The external object may include an object existing in a real space.

[0179] In one embodiment, the display (920, 930) may include a light-collecting lens or a waveguide (waveguide or waveguide) in a transparent member. For example, the transparent member may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be completely transparent or translucent. In one embodiment, the transparent member may include a first transparent member (930) facing the right eye of a user wearing the augmented reality device (900) and a second transparent member (920) facing the left eye of the user. If the display is transparent, it may be positioned facing the user's eyes to display a screen.

[0180] A waveguide can transmit light generated from a light source of a display to a user's eyes. For example, the waveguide can be at least partially positioned on a portion of a transparent member (920, 930). According to one embodiment, light emitted from the display can be incident on one end of the waveguide, and the incident light can be transmitted to the user's eyes through total internal reflection within the waveguide. The waveguide can be made of a transparent material such as glass, plastic, or polymer, and can include a nano-pattern formed on one surface of an inner or outer surface, for example, a grating structure having a polygonal or curved shape. In one embodiment, the incident light can be propagated or reflected within the waveguide by the nano-pattern and provided to the user's eyes. In one embodiment, the waveguide can include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a mirror). In one embodiment, the waveguide can guide display light emitted from a light source to a user's eye using at least one diffractive element or reflective element.

[0181] In one embodiment, the display (920, 930) may include a display panel or a lens (e.g., glass). For example, the display panel may include a transparent material, such as glass or plastic. In one embodiment, the display may be formed of a transparent element, allowing a user to see through the display and perceive the actual space behind the display. The display may display a virtual object on at least a portion of the transparent element so that the user sees the virtual object superimposed on at least a portion of the actual space.

[0182] In one embodiment, the support (921, 922) may include a printed circuit board (PCB) (931-1, 931-2) for transmitting electrical signals to each component of the augmented reality device (900), a speaker (932-1, 932-2) for outputting audio signals, or a battery (933-1, 933-2) for supplying power. The speaker (932-1, 932-2) for outputting audio signals may correspond to the speaker (164) of FIG. 8 described above. For example, in a glasses-type augmented reality device (900), the support (921, 922) may be arranged on the temple portion of the glasses. The support (921, 922) may include a hinge portion (940-1, 940-2) for coupling to the main body portion of the augmented reality device (900). The speakers (932-1, 932-2) may include a first speaker (932-1) for transmitting audio signals to the user's left ear and a second speaker (932-2) for transmitting audio signals to the user's right ear.

[0183] Referring to FIG. 9, the augmented reality device (900) may include a microphone (941) for receiving a user's voice and ambient sounds. In addition, the augmented reality device (900) may include at least one illumination LED (942) to improve the accuracy of at least one camera (e.g., an ET camera (912), an outward-facing camera (913), or a recognition camera (911-1, 911-2)). For example, the illumination LED (942) may be used as an auxiliary means to improve the accuracy when capturing a user's pupil with the ET camera (912), and the illumination LED (942) may use an IR LED of an infrared wavelength rather than a visible light wavelength. For example, the illumination LED (942) may be used as an auxiliary means when capturing a user's gesture with the recognition camera (911-1, 911-2) when it is difficult to detect a subject in a dark environment.

[0184] According to one embodiment, the display module (914) may include a first light guide plate (930) corresponding to the right eye and a second light guide plate (920) corresponding to the left eye, and may provide visual information to the user through the first light guide plate (930) and the second light guide plate (920). According to one embodiment, the display module (914) may include a display panel and a lens (e.g., a glass lens, an LC lens). The display panel may include a transparent material such as glass or plastic.

[0185] In one embodiment, the display module (914) may be formed of a transparent element, such that a user can see through the display module (914) to perceive a real space behind the display module (914) and in front of the user. The display module (914) may display a virtual object on at least a portion of the transparent element so that the user appears to have the virtual object superimposed on at least a portion of the real space.

[0186] In one embodiment, the augmented reality device (900) can identify an external object included in at least a portion of the image information related to a real space acquired through the outward-facing camera (913), which corresponds to an area determined by the user's field of view (FoV). The augmented reality device (900) can output (or display) a virtual object related to the external object identified in at least a portion of the image information through an area determined by the user's field of view among the display areas of the augmented reality device (900). The external object may include an object existing in a real space. According to various embodiments, the display area where the augmented reality device (900) displays a virtual object may include a portion of the display module (914) (for example, at least a portion of the display panel). According to one embodiment, the display area may be an area corresponding to at least a portion of the first light guide plate (930) and the second light guide plate (920).

[0187] In one embodiment, the augmented reality device (900) may use an outward-facing camera (913) to measure a distance to a physical object positioned in the front direction of the augmented reality device (900). The outward-facing camera (913) may include a high-resolution camera, such as a high-resolution (HR) camera and a photo video (PV) camera.

[0188] The augmented reality device (900) according to one embodiment of the present disclosure is not limited to the configuration described above, and may include various components in various locations and in various numbers.

[0189] The present disclosure provides a method for an augmented reality device to operate based on an image of a first object acquired through a camera. The method may include a step of identifying the first object from the image. This allows interaction with the first object, such as a digitizer pen, in an augmented reality environment. The method may include a step of identifying a gripping position where a user grips the first object. The method may include a step of performing a corresponding action based on the identified gripping position. According to the present disclosure, a method for providing information to a user in an augmented reality environment is not limited to a two-dimensional display, and information about objects in real space can be utilized to display related information in a location close to the object, and the area that a user can utilize is not limited to the display of a specific electronic device.

[0190] In one embodiment, the step of identifying a grip location may include the steps of detecting at least one feature point from an image, identifying a hand of the user based on the detected at least one feature point, and determining the grip location based on a position of the hand of the user and a position of the first object.

[0191] In one embodiment, the full length region of the first object may be segmented into percentage intervals, each corresponding to a different action of the augmented reality device.

[0192] In one embodiment, the first object may be divided into preset sections, each corresponding to a different action of the augmented reality device.

[0193] In one embodiment, the method may further include the step of providing the user with guidance on the type of motion corresponding to each grip position through a virtual image.

[0194] In one embodiment, when there are multiple areas where the user contacts the first object, the step of identifying the grip location may include the step of identifying a contact area with the highest priority among the multiple contact areas, and the step of determining the grip location based on the contact area with the highest priority.

[0195] In one embodiment, the method may further comprise the step of identifying a rotation of the first object. The step of performing a corresponding action based on the identified grip position may be further based on the identified rotation.

[0196] In one embodiment, the step of identifying the rotation of the first object may include the steps of identifying a first point in time at which the rotation of the first object is initiated, calculating a cumulative momentum from the first point in time to a current point in time, and determining the rotation based on the cumulative momentum.

[0197] In one embodiment, the first time point may be determined as the earliest time point in a sequence of times at which the momentum per hour is greater than or equal to a preset threshold.

[0198] In one embodiment, the first object may include a touch sensor or a rotation detection sensor. The step of identifying the rotation of the first object may include the steps of obtaining sensing information from the first object and determining the rotation based on the obtained sensing information.

[0199] In one embodiment, the method may further comprise the step of displaying a user interface corresponding to the action being performed.

[0200] The present disclosure provides an augmented reality device that operates based on an image of a first object acquired through a camera. The augmented reality device may include a camera configured to acquire an image of a real scene, a display unit, a memory storing a program including at least one command, and at least one processor. The at least one processor may identify a first object from the image, identify a gripping position where a user grips the first object, and perform a corresponding action based on the identified gripping position.

[0201] In one embodiment, at least one processor can detect at least one feature point from an image, identify a hand of the user based on the detected at least one feature point, and determine a grip position based on a position of the hand of the user and a position of the first object.

[0202] In one embodiment, the full length region of the first object may be segmented into percentage intervals, each corresponding to a different action of the augmented reality device.

[0203] In one embodiment, the first object may be divided into preset sections, each corresponding to a different action of the augmented reality device.

[0204] In one embodiment, at least one processor may provide guidance to the user about the type of motion corresponding to each grip location through a virtual image.

[0205] In one embodiment, at least one processor can identify a contact area with the highest priority among the multiple contact areas when the user contacts the first object, and determine a gripping position based on the contact area with the highest priority.

[0206] In one embodiment, at least one processor can identify a rotation of a first object and perform a corresponding action based on the identified grip position and rotation.

[0207] In one embodiment, at least one processor can identify a first point in time at which rotation of a first object is initiated, calculate an accumulated momentum from the first point in time to a current point in time, and determine a rotation based on the accumulated momentum.

[0208] In one embodiment, at least one processor can determine as a first time point the earliest time point in a series of times at which the momentum per hour is greater than or equal to a preset threshold.

[0209] In one embodiment, the first object may include a touch sensor or a rotation detection sensor. At least one processor may obtain sensing information from the first object and determine rotation based on the obtained sensing information.

[0210] The present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may store instructions readable by an augmented reality device for executing at least one of the disclosed methods on the augmented reality device.

[0211] In this way, according to one embodiment of the present disclosure, a user can control an augmented reality device or an electronic device connected to an augmented reality device more simply and conveniently.

[0212] According to one embodiment of the present disclosure, when using a digitizer pen in an augmented reality environment, the user can avoid unnecessary movements and provide intuitive interaction with minimal hand and arm movement, thereby enhancing user convenience. Furthermore, the user can access toolbar functions directly from the current hand position without having to touch the toolbar displayed on the two-dimensional screen of the electronic device, thereby enhancing user convenience.

[0213] According to one embodiment of the present disclosure, when a user wearing an augmented reality device edits or manipulates content such as documents or images, the user can intuitively execute functions by manipulating various types of objects (such as pens or sticks) held in the user's hand without having to enter a complex menu-related user interface. The execution of the function may be performed based on at least one of the position at which the user holds the object, the movement of the object, or the movement of the user's hand.

[0214] Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and embodied in a computer-readable medium. In the present disclosure, “application” and “program” may refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, related data, or portions thereof suitable for implementation in computer-readable program code. “Computer-readable program code” may include various types of computer code, including source code, object code, and executable code. “Computer-readable medium” may include various types of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a compact disc (CD), a digital video disc (DVD), or various types of memory.

[0215] Additionally, the device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a 'non-transitory storage medium' is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is permanently stored in the storage medium and cases where it is temporarily stored. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. A computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and non-volatile media, removable and non-removable media. A computer-readable medium includes a medium on which data can be permanently stored and a medium on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable memory device.

[0216] The disclosed embodiments can be implemented as a software program including commands stored on a computer-readable storage medium. A computer, as a device capable of recalling commands stored on the storage medium and performing operations according to the disclosed embodiments in accordance with the recalled commands, can include an electronic device according to the disclosed embodiments.

[0217] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may include a software program and a computer-readable storage medium having the software program stored thereon. For example, the computer program product may include a product in the form of a software program (e.g., a downloadable app) distributed electronically by a device manufacturer or through an electronic market. For electronic distribution, at least a portion of the software program may be stored in the storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic market server, or a relay server that temporarily stores the software program.

[0218] In a system comprising a server and a device, the computer program product may include a storage medium of the server or a storage medium of the device. Alternatively, if a third device (e.g., a smartphone) exists that is communicatively connected to the server or device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the server to the device or the third device, or from the third device to the device.

[0219] In this case, one of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0220] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on the server to control a device in communication with the server to perform a method according to the disclosed embodiments.

[0221] As another example, a third device may execute a computer program product to control a device in communication with the third device to perform a method according to the disclosed embodiments. When the third device executes the computer program product, the third device may download the computer program product from a server and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a preloaded state to perform the method according to the disclosed embodiments.

[0222] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0223] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. In the method of operating the augmented reality device (100), A step of identifying a first object (10) from an image of the first object (10) acquired through a camera; A step for identifying a grip position where a user grips the first object (10); A method comprising the step of performing an operation based on the position of the phage.

2. In the first paragraph, the step of identifying the location of the phage is: A step of identifying at least one feature point from the image; a step of identifying the hand of the user based on at least one feature point; and A method comprising the step of identifying the grip position based on the position of the user's hand and the position of the first object (10).

3. In paragraph 1 or 2, A method in which the length area of ​​the first object (10) is divided into sections, each section corresponding to a different operation of the augmented reality device (100), and the sections represent a percentage (%) of the length area of ​​the first object (10).

4. In paragraph 1 or 2, The above first object (10) is divided into sections and each section corresponds to a different operation of the augmented reality device (100).

5. In any one of paragraphs 1 to 4, A method further comprising the step of providing the user with a guide on the types of actions corresponding to different grip positions through a virtual image.

6. In any one of paragraphs 1 to 5, the step of identifying the phage position comprises: When there are multiple areas where the user contacts the first object (10), a step of identifying a contact area with the highest priority among the multiple contact areas; and A method comprising the step of identifying the grip location based on the contact area having the highest priority.

7. In any one of paragraphs 1 to 6, Further comprising a step of identifying the rotation of the first object (10), A method, wherein the step of performing the corresponding action is based on the grip position and the rotation of the first object.

8. In paragraph 7, The step of identifying the rotation of the first object (10) is: A step of identifying a first point in time at which rotation of the first object (10) begins; A step of determining the accumulated momentum from the first point in time to the present point in time; and A method comprising the step of determining the rotation based on the accumulated momentum.

9. In paragraph 8, A method wherein the first time point is the fastest time point among consecutive times at which the momentum per hour is greater than or equal to a preset threshold value.

10. In any one of paragraphs 7 to 9, The first object (10) includes at least one of a touch sensor or a rotation detection sensor, The step of identifying the rotation of the first object (10) is: A step of obtaining sensing information of at least one of the touch sensor or the rotation detection sensor from the first object (10); and A method comprising a step of determining the rotation based on the acquired sensing information.

11. In any one of paragraphs 1 to 10, A method further comprising the step of displaying a user interface (UI) corresponding to the above-described performed operation.

12. In the augmented reality device (100), A camera (120) configured to acquire an image; Display section (162); A memory (150) storing at least one program including at least one instruction; and comprising at least one processor (140), The at least one processor (140) executes the at least one instruction, Identifying the first object (10) from the above image, The user identifies the grip location where the first object (10) is gripped, An augmented reality device (100) that performs an action based on the above-mentioned phage position.

13. In paragraph 12, The at least one processor (140) executes the at least one instruction, Identify at least one feature point from the image, Identifying the user's hand based on at least one feature point detected above, An augmented reality device (100) that identifies the grip location based on the position of the user's hand and the position of the first object (10).

14. In paragraph 12 or 13, The at least one processor (140) executes the at least one instruction, An augmented reality device (100) that identifies a contact area with the highest priority among the plurality of contact areas when the user has multiple contact areas on the first object (10), and identifies the grip location based on the contact area with the highest priority.

15. In paragraph 12 or 13, The at least one processor (140) executes the at least one instruction, Identify the rotation of the first object (10), An augmented reality device (100) that performs a corresponding action based on the position of the grip and the rotation of the first object.

Citation Information

Patent Citations

  • Information terminal device

    JP2017091336A

  • Mobile terminal and method for controlling mobile terminal

    KR1020140099599A

  • Stylus pen

    KR1020160016260A

  • Display device and mobile apparatus using the same

    KR102663412B1

  • KR20210020135A