Augmented reality device and operation method thereof

Abstract

The present disclosure provides a method by which an augmented reality device outputs an augmented reality image including information regarding a step located on a path of a user. The method may comprise a step of acquiring surrounding information of the user of the augmented reality device. The method may comprise a step of identifying the step on the path of the user. The method may comprise a step of outputting the augmented reality image including the information regarding the step.

Description

Augmented reality device and method of operation thereof

[0001] The present disclosure relates to an augmented reality device and a method of operating the same, and more particularly, to a method of using the augmented reality device to identify a step on a user's path and output a virtual image including information about the step, and to the augmented reality device.

[0002] Augmented Reality (AR) or Virtual Reality (VR) technology synthesizes virtual objects or information into a real-world environment, making them appear as if they were real objects in the physical world. Modern computing and display technologies have enabled the development of systems for AR experiences, in which digitally recreated images or portions of them are presented to the user in a way that makes them appear, or can be perceived, as, real.

[0003] Augmented reality (AR) technology allows virtual images to be overlaid onto the physical environment or objects of the real world. As interest in AR technology grows, development of various technologies for implementing AR is actively underway. AR devices (e.g., head-mounted displays (HMDs), smart glasses, etc.) utilizing AR technology are being utilized in everyday life for purposes such as providing notifications, providing route guidance, and capturing images.

[0004] Augmented reality devices or virtual reality devices are everyday life devices that can acquire data from the user's perspective, and in particular, smart glasses can display virtual images by overlaying them on images of the physical environment of the real world.

[0005] Users with low vision or other cognitive impairments may find it difficult to recognize obstacles in real-world spaces. The development of technology that can effectively provide route guidance to users with cognitive impairments through augmented reality environments where real-world and virtual objects coexist is essential.

[0006] One aspect of the present disclosure provides a method for an augmented reality device to output an augmented reality image including information about a step located on a user's path. The method for operating the augmented reality device may include a step of acquiring information about the user's surroundings of the augmented reality device. The method for operating the augmented reality device may include a step of identifying a step on the user's path. The method for operating the augmented reality device may include a step of outputting an augmented reality image including information about the step.

[0007] One aspect of the present disclosure provides an augmented reality device that outputs an augmented reality image. The augmented reality device may include a camera configured to acquire a real-world scene image. The augmented reality device may include a display unit. The augmented reality device may include a memory that stores a program including at least one command. The augmented reality device may include at least one processor. The at least one processor may acquire information about the user's surroundings, identify a step along the user's path, and output an augmented reality image including information about the step through the display unit.

[0008] One aspect of the present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may store instructions readable by an augmented reality device for executing at least one of the disclosed methods on the augmented reality device.

[0009] FIG. 1 is a drawing for explaining a method for an augmented reality device according to one embodiment of the present disclosure to output an augmented reality image including information about a step located on a user's path.

[0010] FIG. 2 is a flowchart of a method for operating an augmented reality device according to one embodiment of the present disclosure.

[0011] FIG. 3 is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to identify a step and output an augmented reality image including information about the step.

[0012] FIG. 4 is a drawing for explaining an operation of an augmented reality device identifying a step according to one embodiment of the present disclosure.

[0013] FIG. 5A is a drawing for explaining an operation of an augmented reality device according to one embodiment of the present disclosure to identify a direction of a step based on pose information of the augmented reality device.

[0014] FIG. 5b is a drawing for explaining an operation of an augmented reality device according to one embodiment of the present disclosure to identify a direction of a step based on pose information of the augmented reality device.

[0015] FIG. 6 is a drawing for explaining an operation of an augmented reality device according to one embodiment of the present disclosure to identify at least one step surface.

[0016] FIG. 7 is a drawing for explaining an operation of an augmented reality device according to one embodiment of the present disclosure to identify a movement of a step.

[0017] FIG. 8 is a block diagram of an augmented reality device according to one embodiment of the present disclosure.

[0018] FIG. 9 is a drawing illustrating a glass-type augmented reality device according to one embodiment of the present disclosure.

[0019] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, for the purpose of clearly explaining the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar parts are designated with similar reference numerals throughout the specification.

[0020] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0021] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0022] Throughout this disclosure, when a part is referred to as "including" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0023] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where they are "directly connected" but also the cases where they are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes the possibility of including other components, unless otherwise specifically stated.

[0024] The expression “configured to” as used herein can be used interchangeably with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean something that is “specifically designed to” in terms of hardware. Instead, in some contexts, the expression “a system configured to” can mean that the system, together with other devices or components, is “capable of.” For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a generic-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in memory.

[0025] The functions related to 'Artificial Intelligence (AI)' according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an AI-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or AI models stored in memory. If one or more processors are AI-only processors, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.

[0026] The predefined operation rules or artificial intelligence model are characterized by being created through learning. Here, being created through learning means that a basic artificial intelligence model (or deep learning model) is trained using a plurality of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0027] An "artificial intelligence model (or deep learning model)" may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the calculation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be modified so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), such as, but not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0028] In this disclosure, 'Augmented Reality' refers to displaying a virtual image together within a physical environment space of the real world or displaying a real object and a virtual image together.

[0029] In the present disclosure, an 'augmented reality device' is a device capable of expressing augmented reality (AR) or virtual reality (VR), and can display images including physical objects and virtual objects existing in reality. The augmented reality device may be, for example, not only augmented reality glasses in the shape of glasses worn by a user on the face, but also a head-mounted display device (HMD, Apparatus) worn on the head, or an augmented reality helmet.

[0030] The present disclosure will be described in detail with reference to the attached drawings below.

[0031] FIG. 1 is a drawing for explaining a method for an augmented reality device according to one embodiment of the present disclosure to output an augmented reality image (VI) including information about a step (S) located on a path of a user (U).

[0032] An augmented reality device (or virtual reality device) is a device capable of expressing augmented reality or virtual reality, and may be configured, for example, as augmented reality glasses in the shape of glasses worn by a user on the face. The components of the augmented reality device will be described in more detail in FIGS. 8 and 9 below.

[0033] In one embodiment, an augmented reality device may output an augmented reality image including information about a step (S) located on a path of a user (U). The augmented reality device may acquire information about the user's (U) surroundings, identify the step (S) on the user's (U) path, and output an augmented reality image (VI) including information about the step (S). In one embodiment, the augmented reality image (VI) may also be referred to as a virtual image.

[0034] In the present disclosure, the 'path' of the user (U) may represent the path along which the user is moving. In one embodiment, the augmented reality device may guide the user (U) along the path. In the present disclosure, the 'step (S)' may represent a ground with a step-by-step height difference, such as a staircase or a protruding obstacle. The step (S) may represent an area in the path of the user (U) where the height of the ground on which the user steps changes by a preset threshold value or more in an area of ​​a certain width or more. For example, the minimum width determined as the step (S) may be approximately 1 / 3 of the area of ​​the user's (U's) foot, and the threshold value of the height for determining the step (S) may be approximately 1 cm. The step (S) may include a plurality of step units. For example, if the step (S) is a staircase, a step unit may represent one step (one step or one step) of the staircase.

[0035] For users (U) with low vision or other cognitive impairments, it can be difficult to recognize obstacles in real space, and walking in daily life can be very difficult. In particular, walking is even more difficult in places with steps (S) or moving steps (S), such as stairs or escalators. In an augmented reality environment, virtual objects or information can be synthesized into a real environment, so that a virtual image (or augmented reality image) (VI) can be overlaid on the real environment and displayed together. By utilizing this, in one embodiment of the present disclosure, information about steps (S) or a walking guide is overlaid on a virtual image (VI) in a stairway or escalator area in an augmented reality environment, thereby effectively assisting walking (e.g., preventing collisions with obstacles and assisting in climbing stairs or boarding escalators) for users (U) with low cognitive impairments.

[0036] In one embodiment, a user (U) with low vision or other cognitive impairment may have difficulty recognizing a step (S) even if the edge of the step (S) is emphasized. In one embodiment of the present disclosure, when the user (U) passes over the step (S), an augmented reality image (VI) that emphasizes the step (S) that the user (U) must step on can be visually output by considering the direction of the step (S) and the size and shape of the step surface.

[0037] In one embodiment, the augmented reality image (VI) including information about a step (S) may include virtual images (VI1, VI2) that highlight the step (S) that the user (U) should step on. For example, the augmented reality image (VI) including information about the step (S) may include at least one of an indicator indicating a location where the user (U) should step, a notification guiding the timing of the user (U) stepping, or a directional indicator indicating the direction in which the step (S) moves.

[0038] In one embodiment, the indicators indicating where the user (U) will step may be displayed differently depending on the user's (U's) feet. For example, referring to FIG. 1, one or more first indicators (VI1) may indicate a step surface that the user (U) will step on with the right foot, and one or more second indicators (VI2) may indicate a step surface that the user (U) will step on with the left foot. The first indicators (VI1) and the second indicators (VI2) may be distinguished, for example, by having different colors or different patterns.

[0039] In one embodiment, the AR device may acquire additional user information about the user (U). The user information may include, for example, at least one of stride information, disability information, vision information, whether the user uses a walking aid (e.g., a cane), or whether the user uses a mobility aid (e.g., a wheelchair). The user information may be used by the AR device to identify whether the user (U) has low vision or other cognitive impairment.

[0040] In one embodiment, the user information may be used to generate an augmented reality image (VI) that includes information about a step (S). For example, if the user information identifies that the user (U) has a wide stride, the augmented reality device may determine the positions at which at least one first indicator (VI1) and a second indicator (VI2) are to be displayed according to the stride of the user (U). For example, if the user information identifies that the user (U) uses a cane as a walking aid, the augmented reality device may distinguish an area where the cane is to be held by a third indicator and determine the positions at which at least one first indicator (VI), a second indicator (VI2), and a third indicator are to be displayed. The third indicator may be distinguished from the first indicator (VI1) and the second indicator (VI2) by color, pattern, etc.

[0041] In this way, according to one embodiment of the present disclosure, the augmented reality device can identify a step (S) located on the path of the user (U) and provide information about the step (S) to the user as an augmented reality image (VI), thereby assisting the user (U) in recognizing the step (S) and effectively providing path guidance to the user (U).

[0042] FIG. 2 is a flowchart of a method for operating an augmented reality device according to one embodiment of the present disclosure.

[0043] In step 210, the augmented reality device acquires information about the user's surroundings. The operation of the augmented reality device acquiring information about the user's surroundings may include at least one of: acquiring an image of the user's surroundings through a camera; acquiring a depth map of the user's surroundings through a depth sensor; or acquiring information about the surroundings previously stored in memory.

[0044] In one embodiment, the augmented reality device can acquire an image of the user's surroundings via a camera. The image of the surroundings can be captured via the camera. The camera may be a camera built into the augmented reality device. The camera may be implemented in various types, such as an RGB camera, an RGB-depth camera with a depth estimation function, a dynamic vision sensor camera, a stereo fisheye camera, a grayscale camera, or an infrared camera.

[0045] In one embodiment, an augmented reality device can acquire a depth map of the surrounding environment via a depth sensor. The depth sensor can be implemented in various types, such as a stereo camera, a depth sensor, a depth camera, a proximity sensor, or a time-of-flight (ToF) type sensor.

[0046] In one embodiment, the AR device can obtain environmental information previously stored in memory. For example, the AR device can access information, such as a map of the user's surroundings, from its internal memory. The environmental information previously stored in the internal memory may be information stored by the AR device through previous operations or downloaded from an external server.

[0047] In step 220, the augmented reality device identifies a step along the user's path. In one embodiment, the step of the augmented reality device identifying the step may include the step of identifying the direction of the step and the step of identifying at least one step surface of the step. In one embodiment, the step of the augmented reality device identifying the step may further include the step of identifying the movement of the step.

[0048] The direction of the step may be distinguished as upward or downward. In one embodiment, the step of identifying the direction of the step by the augmented reality device may be based on a depth map acquired through a depth sensor built into the augmented reality device and pose information of the augmented reality device at the time of acquiring the depth map.

[0049] Pose information of an augmented reality device may include three-dimensional spatial angle information of the augmented reality device at the time of acquisition of the depth map. The three-dimensional spatial angle information may represent the yaw value, pitch value, and roll value of the augmented reality device, and may be expressed as, for example, Euler angles or quaternions.

[0050] In one embodiment, a depth map acquired through a depth sensor embedded in an augmented reality device may correspond to a depth map in the direction of gaze of a user wearing the augmented reality device. For example, the sensing area of ​​the depth sensor may correspond to a field of view (FOV) area of ​​a user wearing the augmented reality device, and in this case, the acquired depth map may include a depth value in the direction of the view of the user wearing the augmented reality device.

[0051] In one embodiment, the augmented reality device may include an inertial measurement unit (IMU). An augmented reality device including an inertial sensor may acquire pose information of the augmented reality device at the time of acquiring a depth map through the inertial sensor.

[0052] In one embodiment, the augmented reality device may directly calculate pose information based on at least one of an image of the surrounding environment acquired through a camera or a depth map acquired through a depth sensor. For example, the augmented reality device may perform image analysis (e.g., calculating a depth map of the image) on an acquired image of the surrounding environment to derive pose information of a camera that captured the image, and may derive pose information of the augmented reality device based on information regarding where the camera is embedded in the augmented reality device.

[0053] A step surface represents a surface on which a user can step. The step surface may be parallel to the ground and may represent the upper surface of a step unit included in the step. For example, if the step is a staircase, the step surface may represent a surface that a user steps on when going up or down the stairs. In one embodiment, the step of identifying at least one step surface of the step by the augmented reality device may be identifying at least one step surface by edge analysis of an image of the surrounding environment acquired through a camera built into the augmented reality device.

[0054] The movement of steps can be categorized as either upward or downward. An upward movement of steps can indicate an increase in the height of the step, as in, for example, an escalator going up or up. A downward movement of steps can indicate a decrease in the height of the step, as in, for example, an escalator going down or down.

[0055] In one embodiment, if the direction of the step is an upward step, the augmented reality device can identify the change in height of the first step, and determine the movement of the step as an upward movement if the height of the first step increases, and determine the movement of the step as a downward movement if the height of the first step decreases. The first step may represent the step unit closest to the user.

[0056] In one embodiment, when the direction of the step is a downward step, the augmented reality device can identify a change in at least one step surface and determine the movement of the step as an upward movement if the width of the first step surface decreases or the distance between the user and the edge of the at least one step surface decreases, and determine the movement of the step as a downward movement if the width of the first step surface increases or the distance between the user and the edge of the at least one step surface increases.

[0057] In one embodiment, when the direction of the step is an upward step, the augmented reality device can identify a change in at least one step surface and determine the movement of the step as a downward movement if the width of the first step surface decreases or the distance between the user and the edge of the at least one step surface decreases, and determine the movement of the step as an upward movement if the width of the first step surface increases or the distance between the user and the edge of the at least one step surface increases.

[0058] The operation of the augmented reality device to identify steps on the user's path will be described in more detail with reference to FIGS. 4 to 7 below.

[0059] In step 230, the augmented reality device displays an augmented reality image including information about the step. For example, the augmented reality image including information about the step may include at least one of an indicator indicating where the user should step, a notification guiding the timing of the user's step, or a directional indicator indicating the direction in which the step moves.

[0060] In one embodiment, when a user passes over a step along a moving path, the augmented reality device may visually output an augmented reality image that highlights the step that the user should step on, taking into account the direction of the step and the size and shape of the step surface. For example, the augmented reality image including information about the step may include a virtual image that highlights the step that the user should step on. In one embodiment, indicators indicating where the user should step may be displayed differently depending on the user's two feet. For example, the step surface that the user will step on with the right foot may be indicated through at least one first indicator having a first color or a first pattern, and the step surface that the user will step on with the left foot may be indicated through at least one second indicator having a second color or a second pattern. If the user is using a walking assistance tool such as a cane, the area where the walking assistance tool should be stepped on may be indicated through at least one third indicator having a third color or a third pattern.

[0061] In one embodiment, an augmented reality image including information about steps may include a notification that guides the user when to step. For example, if the steps are a moving escalator, the AR device may provide a notification to the user when to board the escalator. For example, the notification may be provided by blinking the entire display of the AR device, or may be provided as a countdown or various audiovisual signals. The user can step on the moving steps in time with the notification provided by the AR device (e.g., board the escalator in time).

[0062] In one embodiment, an augmented reality image containing information about a step may include a directional indicator indicating the direction in which the step moves. For example, if the step is a moving escalator, the AR device may provide information via a directional indicator indicating whether the escalator is moving up or down. For example, the directional indicator may be provided in the form of a virtual image in the shape of an arrow or various audiovisual signals, such as an audio signal.

[0063] In one embodiment, the augmented reality device may acquire additional user information about the user. The user information may include, for example, at least one of stride information, disability information, vision information, whether the user uses a walking aid, or whether the user uses a mobility aid.

[0064] In one embodiment, the AR device may generate an AR image that includes information about a step based on user information. For example, if the user information identifies that the user has a wide stride, the AR device may determine the location at which at least one indicator to be included in the AR image will be displayed based on the user's stride. In one embodiment, the AR device may determine, based on the user information, that a specific step is one that the user cannot ascend or descend. In this case, the AR device may exclude the step from the path and display the step as an obstacle in the AR image.

[0065] According to the augmented reality device of the present disclosure, by identifying a step located on the user's path and providing the user with information about the step as an augmented reality image, it is possible to assist the user in recognizing the step well and to effectively provide the user with path guidance.

[0066] FIG. 3 is a drawing for explaining a method for an augmented reality device according to one embodiment of the present disclosure to identify a step and output an augmented reality image (VI) including information about the step.

[0067] In step 310, the AR device may acquire an image of the user's surroundings. In one embodiment, the image of the surroundings may be captured using a camera. The camera may be a camera built into the AR device. The camera may be implemented in various types, such as an RGB camera, a grayscale camera, or an infrared camera.

[0068] In step 315, the augmented reality device may acquire a depth map of the user's surroundings. In one embodiment, the augmented reality device may acquire the depth map of the surroundings using a depth sensor. The depth sensor may be implemented in various types, such as a stereo camera, a depth sensor, a depth camera, a proximity sensor, or a time-of-flight (ToF) type sensor.

[0069] In step 320, the augmented reality device identifies a step along the user's path. In one embodiment, the step of the augmented reality device identifying the step may include the steps of identifying the direction of the step and identifying at least one step surface of the step. The step surface represents a surface on which the user can step. The step surface may be parallel to the ground and may represent an upper surface of a step unit included in the step. For example, if the step is a staircase, the step surface may represent a surface on which the user steps when ascending or descending the stairs.

[0070] In one embodiment, the augmented reality device can identify a step based on at least one of the image of the surrounding environment acquired in step 310 or the depth map of the surrounding environment acquired in step 315. For example, the augmented reality device can identify at least one step surface by edge analysis of the image of the surrounding environment acquired in step 310, or can identify a step as existing in an area where a depth value changes by more than a preset threshold value through the depth map of the surrounding environment acquired in step 315.

[0071] In step 330, the AR device displays an AR image containing information about the step. For example, the AR device may visually output an AR image (VI) highlighting the step the user is to step on. In one embodiment, the AR image (VI) containing information about the step may include virtual images (VI1, VI2) highlighting the step the user is to step on. For example, the AR image containing information about the step may include an indicator indicating the location where the user is to step.

[0072] In one embodiment, the augmented reality device can visually output an augmented reality image (VI) that emphasizes a step that the user should step on by considering the direction of the step and the size and shape of the step surface. In one embodiment, indicators indicating where the user should step can be displayed differently depending on the user's two feet. For example, the step surface that the user will step on with the right foot can be indicated through at least one first indicator (VI1) having a first color or a first pattern, and the step surface that the user will step on with the left foot can be indicated through at least one second indicator (VI2) having a second color or a second pattern. When the user is using a walking assistance tool such as a cane, the area where the walking assistance tool will be stepped on can be indicated through at least one third indicator having a third color or a third pattern.

[0073] In one embodiment, the AR device may provide a separate notification to the user regarding steps that deviate from the regular repetition of multiple steps. For example, on a staircase with a regular height difference, if a specific step has a different height than the other steps, a user who has been accustomed to walking on a regular height difference may not recognize the change in height with a specific step and may trip and fall. The AR device according to one embodiment of the present disclosure may provide a notification to the user regarding irregular steps through audiovisual materials such as blinking, different colors, or indicators in a section where the interval (height difference) between steps suddenly changes.

[0074] FIG. 4 is a drawing for explaining an operation of an augmented reality device identifying a step according to one embodiment of the present disclosure.

[0075] In step 410, the AR device identifies the direction of the step. The direction of the step can be distinguished as upward or downward. The direction of the step can be determined not by the direction in which the dynamic step (e.g., an escalator) moves, but by whether the step surface is located higher or lower relative to the user's current location. For example, a staircase leading to a higher floor relative to the user's current location can be identified as an upward step, and a staircase leading to a lower floor relative to the user's current location can be identified as a downward step.

[0076] In one embodiment, the step of identifying the direction of the step by the augmented reality device may be based on a depth map acquired through a depth sensor built into the augmented reality device and pose information of the augmented reality device at the time of acquiring the depth map.

[0077] Pose information of an augmented reality device may include three-dimensional spatial angle information of the augmented reality device at the time of acquisition of the depth map. The three-dimensional spatial angle information may represent the yaw value, pitch value, and roll value of the augmented reality device, and may be expressed as, for example, Euler angles or quaternions.

[0078] In one embodiment, a depth map acquired through a depth sensor embedded in an augmented reality device may correspond to a depth map in the direction of gaze of a user wearing the augmented reality device. For example, the sensing area of ​​the depth sensor may correspond to a field of view (FOV) area of ​​a user wearing the augmented reality device, and in this case, the acquired depth map may include a depth value in the direction of the view of the user wearing the augmented reality device.

[0079] In one embodiment, the augmented reality device may include an inertial sensor (inertial measurement unit, IMU). An augmented reality device including an inertial sensor may acquire pose information of the augmented reality device at the time of acquiring a depth map through the inertial sensor.

[0080] In one embodiment, the augmented reality device may directly calculate pose information based on at least one of an image of the surrounding environment acquired through a camera or a depth map acquired through a depth sensor. For example, the augmented reality device may perform image analysis (e.g., calculating a depth map of the image) on an acquired image of the surrounding environment to derive pose information of a camera that captured the image, and may derive pose information of the augmented reality device based on information regarding where the camera is embedded in the augmented reality device.

[0081] The operation of the augmented reality device to identify the direction of the step will be described in more detail with reference to FIGS. 5a, 5b, and 6, which will be described later.

[0082] In step 420, the augmented reality device identifies at least one step surface of a step. The step surface represents a surface on which a user can step. The step surface may be parallel to the ground and may represent the upper surface of a step unit included in the step. For example, if the step is a staircase, the step surface may represent a surface that a user steps on when ascending or descending the stairs.

[0083] In one embodiment, the step of identifying at least one step surface of a step by an augmented reality device may be identifying at least one step surface by edge analysis of an image of a surrounding environment acquired through a camera built into the augmented reality device. In one embodiment, the step of identifying at least one step surface of a step by an augmented reality device may be identifying that a step exists in an area where a depth value changes by more than a preset threshold value through a depth map of the surrounding environment acquired through a depth sensor built into the augmented reality device. For example, the preset threshold value may be about 1 cm.

[0084] The operation of the augmented reality device identifying at least one step surface will be described in more detail with reference to FIG. 6 below.

[0085] At step 430, the augmented reality device can identify the movement of steps. Step 430 may be optional. The movement of steps may be categorized as either upward or downward. An upward movement of steps may indicate an increase in the height of the steps, for example, an escalator moving up or down. A downward movement of steps may indicate a decrease in the height of the steps, for example, an escalator moving down or down.

[0086] In one embodiment, if the direction of the step is an upward step, the augmented reality device can identify the change in height of the first step, and determine the movement of the step as an upward movement if the height of the first step increases, and determine the movement of the step as a downward movement if the height of the first step decreases. The first step may represent the step unit closest to the user.

[0087] In one embodiment, when the direction of the step is a downward step, the augmented reality device can identify a change in at least one step surface and determine the movement of the step as an upward movement if the width of the first step surface decreases or the distance between the user and the edge of the at least one step surface decreases, and determine the movement of the step as a downward movement if the width of the first step surface increases or the distance between the user and the edge of the at least one step surface increases.

[0088] In one embodiment, when the direction of the step is an upward step, the augmented reality device can identify a change in at least one step surface and determine the movement of the step as a downward movement if the width of the first step surface decreases or the distance between the user and the edge of the at least one step surface decreases, and determine the movement of the step as an upward movement if the width of the first step surface increases or the distance between the user and the edge of the at least one step surface increases.

[0089] The operation of the augmented reality device to identify the movement of steps will be described in more detail with reference to Fig. 7 below.

[0090] The above-described steps 410, 420, and 430 may be performed without being limited to the illustrated order. For example, steps 410 and 420 may be performed simultaneously, or step 420 may be performed before step 410.

[0091] FIG. 5A and FIG. 5B are drawings for explaining an operation of an augmented reality device according to one embodiment of the present disclosure to identify a direction of a step based on pose information of the augmented reality device.

[0092] Figure 5a illustrates a case where the step type is a staircase and the step direction is upward.

[0093] Referring to (a) of Fig. 5a, when the pose of the augmented reality device is facing forward, the depth values ​​from the augmented reality device to each step (one step) may have a distribution of discrete values. In other words, since the distance from the depth sensor to each step has a clear difference, a depth map acquired from a depth sensor using a Time of Flight (ToF) method may include distinguishable boundaries of the steps. In other words, a depth map acquired when the augmented reality device is facing forward includes relatively clearly the boundaries of each step, so that the steps can be easily identified from the depth map.

[0094] Referring to (b) of Fig. 5a, when the pose of the augmented reality device is looking downward, the depth values ​​from the augmented reality device to each step (one stair) may not have a large difference. In other words, since the distance from the depth sensor to a portion of each step is the same, it may be difficult to identify the step from the depth map acquired from the depth sensor using the ToF method. In other words, the depth map acquired in a state where the augmented reality device is looking downward does not clearly include the boundary of each step, and it may be somewhat difficult to identify the step from the depth map.

[0095] Figure 5b illustrates a case where the step type is a staircase and the step direction is downward.

[0096] Referring to (a) of Fig. 5b, when the AR device is facing forward, each step may be outside the sensing area of ​​the depth sensor, and may appear as a cliff in the acquired depth map. In other words, it may be difficult to identify downward steps based on a depth map acquired when the AR device is facing forward.

[0097] Referring to (b) of Fig. 5B, when the pose of the augmented reality device is looking downward, the depth values ​​from the augmented reality device to each step (one step) may have a distribution of discrete values. In other words, since the distance from the depth sensor to each step has a clear difference, the depth map acquired from the depth sensor using the ToF method may include distinguishable boundaries of the steps. In other words, the depth map acquired in a state where the augmented reality device is looking downward includes relatively clearly the boundaries of each step, so that the steps can be easily identified from the depth map.

[0098] In this way, the cases illustrated in (a) of Fig. 5a and (b) of Fig. 5b may correspond to similar depth maps. Therefore, it may be difficult to identify the direction of the step based solely on the depth map.

[0099] An augmented reality device of one embodiment of the present disclosure identifies the direction of a step not only based on a depth map acquired through a depth sensor built into the augmented reality device, but also based on pose information of the augmented reality device at the time of acquiring the depth map, in order to accurately distinguish the direction of a step. For example, if the pose of the augmented reality device at the time of acquiring a depth map that includes relatively clear step boundaries was facing forward, the augmented reality device can identify the step corresponding to the depth map as an upward step. If the pose of the augmented reality device at the time of acquiring a depth map that includes relatively clear step boundaries was facing downward, the augmented reality device can identify the step corresponding to the depth map as a downward step.

[0100] FIG. 6 is a drawing for explaining an operation of an augmented reality device according to one embodiment of the present disclosure to identify at least one step surface.

[0101] Referring to FIG. 6, if a step is a staircase, the step surface may represent the surface that a user steps on when ascending or descending the stairs. The step surface may be parallel to the ground and may represent the upper surface of a step unit (one step) included in the step. In one embodiment, the step unit may include a longitudinal pattern. The longitudinal pattern may represent a specific pattern included in the corner portion of the staircase.

[0102] In one embodiment, the augmented reality device can identify a terminal pattern from an image of the surrounding environment acquired through a built-in camera, and identify at least one step surface through edge analysis of the image. For example, the augmented reality device can identify an area surrounded by the terminal pattern and at least one edge as a step surface.

[0103] In one embodiment, the AR device may identify the direction of a step through the end pattern. For example, if the end pattern is determined to be closer to the user on the corresponding step plane in the image acquired through the camera, the AR device may identify the step as an upward step. Additionally, if the end pattern is determined to be farther from the user on the corresponding step plane in the image acquired through the camera, the AR device may identify the step as a downward step.

[0104] FIG. 7 is a drawing for explaining an operation of an augmented reality device according to one embodiment of the present disclosure to identify a movement of a step.

[0105] In step 710, the augmented reality device can identify the direction of the step. Step 710 of FIG. 7 may correspond to step 410 of FIG. 4 described above.

[0106] The direction of a step can be categorized as upward or downward. The direction of a step is determined not by the direction in which a dynamic step (e.g., an escalator) moves, but by whether the step surface is higher or lower relative to the user's current position. For example, a staircase that ascends to a higher floor than the user's current position while stationary can be identified as an upward step, while a staircase that descends to a lower floor relative to the user's current position can be identified as a downward step.

[0107] If the step is identified as an upward step in step 710, the method continues to step 720, and if the step is identified as a downward step in step 710, the method continues to step 725.

[0108] An augmented reality device according to one embodiment of the present disclosure can identify the movement of steps. The movement of steps can be categorized as either upward or downward movement. An upward movement of a step can indicate an increase in the height of the step, for example, an escalator moving up or down. A downward movement of a step can indicate a decrease in the height of the step, for example, an escalator moving down or down.

[0109] At step 720, the direction of the step can be identified as an upward step.

[0110] In step 730, the augmented reality device can identify a change in the height (h) of the first step. The first step may represent the step unit closest to the user. The height (h) of the first step may represent the vertical distance from the user's current position level to the next step surface level.

[0111] In step 740, if the change in the height (h) of the first step is identified as increasing in step 730, the augmented reality device may determine the movement of the step as an upward movement. If the direction of the step is upward and the movement of the step is an upward movement, the corresponding escalator may correspond to an escalator going up. If the change in the height (h) of the first step is identified as decreasing in step 730, the augmented reality device may determine the movement of the step as a downward movement. If the direction of the step is upward and the movement of the step is a downward movement, the corresponding escalator may correspond to an escalator going down.

[0112] At step 725, the direction of the step can be identified as a downward step.

[0113] In step 735, the augmented reality device can identify a change in the first step surface (P). The first step surface can represent the step surface of the step unit closest to the user. In step 745, if the area of ​​the first step surface is identified as decreasing in step 735, the augmented reality device can determine the movement of the step as an upward movement. If the direction of the step is downward and the movement of the step is an upward movement, the corresponding escalator can correspond to an escalator going up. If the area of ​​the first step surface is identified as increasing in step 735, the augmented reality device can determine the movement of the step as a downward movement. If the direction of the step is downward and the movement of the step is a downward movement, the corresponding escalator can correspond to an escalator going down.

[0114] In step 735, the augmented reality device can identify a change in the distance between the user and an edge (E) of at least one step surface. The edge (E) of at least one step surface may include a terminal pattern of a step unit. In step 745, if the distance between the user and the edge (E) of at least one step surface is identified as decreasing in step 735, the augmented reality device can determine the movement of the step as an upward movement. If the direction of the step is downward and the movement of the step is an upward movement, the corresponding escalator may correspond to an escalator going up. If the distance between the user and the edge (E) of at least one step surface is identified as increasing in step 735, the augmented reality device can determine the movement of the step as a downward movement. If the direction of the step is downward and the movement of the step is a downward movement, the corresponding escalator may correspond to an escalator going down.

[0115] The operation of identifying the movement of the step is not limited to the above-described embodiment and various methods can be applied.

[0116] FIG. 8 is a block diagram of an augmented reality device (100) according to one embodiment of the present disclosure.

[0117] Referring to FIG. 8, the augmented reality device (100) may include a communication interface (110), a camera (120), a sensor (130), a processor (140), a memory (150), and an output interface (160). The communication interface (110), the camera (120), the sensor (130), the processor (140), the memory (150), and the output interface (160) may be electrically and / or physically connected to each other, respectively. Meanwhile, the components of the augmented reality device (100) are not limited to those illustrated in FIG. 8. The augmented reality device (100) may be implemented with more components than those illustrated in FIG. 8, or may be implemented with fewer components than those illustrated in FIG. 8.

[0118] In one embodiment of the present disclosure, the augmented reality device (100) may be implemented as augmented reality glasses worn on a user's head, in which case the augmented reality device (100) may further include a power supply (e.g., a battery) that supplies driving power to a communication interface (110), a camera (120), a sensor (130), a processor (140), and an output interface (160). In one embodiment, the augmented reality device (100) may not include a speaker (164).

[0119] The communication interface (110) is configured to transmit and receive data with a server or an external device (e.g., a device of interest) via a wired or wireless communication network. The communication interface (110) may perform data communication with the server or the external device using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication. However, the present invention is not limited thereto, and when the augmented reality device (100) is implemented as a wearable device such as smart glasses, the communication interface (110) may transmit and receive data with a server or an external device through a network that follows a mobile communication standard such as a communication method using CDMA, WCDMA, 3G, 4G (LTE), 5G Sub 6, and / or millimeter wave (mmWave).

[0120] In one embodiment, the communication interface (110) is connected to at least one external object via a short-range communication method such as Bluetooth or Wi-Fi Direct, and may transmit and receive various information with the external object.

[0121] The camera (120) is configured to acquire two-dimensional (2D) or three-dimensional (3D) image data by photographing a real space. The camera (120) may be implemented in a small form factor so that it can be mounted on the augmented reality device (100) and may be a lightweight RGB camera that consumes low power. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the camera (120) may be implemented as any type of camera known in the art, such as an RGB-depth camera including a depth estimation function, a Dynamic Vision Sensor camera, a stereo fisheye camera, a grayscale camera, or an infrared camera. In one embodiment, at least some of the cameras (120) may be arranged in a direction facing a user and configured to photograph the user's face.

[0122] The camera (120) may include a lens module, an image sensor, and an image processing module. The camera (120) may acquire still images or video of a real-world scene by means of an image sensor (e.g., CMOS or CCD). The video may include multiple image frames acquired in real time by photographing a real-world area through the camera (120). The image processing module may encode still images composed of a single image frame acquired through the image sensor or video data composed of multiple image frames and transmit them to the processor (140).

[0123] According to one embodiment of the present disclosure, capturing a real scene image may include an operation of the augmented reality device (100) to control a camera (e.g., a camera including an image sensor and a lens) (120) provided in the augmented reality device (100) to convert an optical image formed through the lens into an electrical signal to acquire an image. For example, one or more processors (140) may control the camera (120) provided in the augmented reality device (100) to capture a surrounding area of ​​the augmented reality device (100) to acquire an image (e.g., a captured image) including one or more frames. Here, the image may include a live-view image.

[0124] The sensor (130) may include sensors configured to detect real space, location, situation, or user information. In one embodiment of the present disclosure, the sensor (130) may include, but is not limited to, a depth sensor, an eye tracking sensor, an Inertial Measurement Unit (IMU) sensor, a Global Positioning System (GPS) sensor, a Bluetooth Low Energy (BLE) sensor, an Ultra Wide Broadband (UWB) sensor, or other sensors capable of sensing various signals.

[0125] The processor (140) can execute one or more instructions of a program stored in the memory (150). The processor (140) may be configured with hardware components that perform arithmetic, logic, and input / output operations and image processing. Although FIG. 8 illustrates a single processor (140), the present invention is not limited thereto, and the processor (140) may be configured with one or more multiple elements. The processor (140) may be a general-purpose processor such as a CPU (Central Processing Unit), an AP (Application Processor), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), a VPU (Vision Processing Unit), or an NPU (Neural Processing Unit), a graphics-only processor, or an NPU (Artificial Intelligence)-only processor. The processor (140) may be controlled to process input data according to a predefined operating rule or artificial intelligence model. Alternatively, when the processor (140) is an artificial intelligence-only processor, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0126] The memory (150) may be configured as at least one type of storage medium, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk.

[0127] The memory (150) may store instructions related to functions and / or operations for the operation of the augmented reality device (100). In one embodiment, the memory (150) may store at least one of instructions, algorithms, data structures, program codes, and application programs that can be read by the processor (140). The instructions, algorithms, data structures, and program codes stored in the memory (150) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0128] The processor (140) can execute instructions or program codes stored in the memory (150) and control the overall operation of the augmented reality device (100). The processor (140) can perform operations according to one embodiment of the present disclosure. For example, the processor (140) can control the communication interface (110), the camera (120), the sensor (130), and the output interface (160) overall by executing programs stored in the memory (150).

[0129] The processor (140) may be configured with hardware components that perform arithmetic, logic, and input / output operations and signal processing. The processor (140) may be configured with at least one of, for example, a central processing unit (CPU), a microprocessor, a graphic processing unit (GRAP), an application specific integrated circuits (ASICs), a digital signal processor (DSPs), a digital signal processing device (DSPDs), a programmable logic device (PLDs), and a field programmable gate array (FPGAs), but is not limited thereto.

[0130] In one embodiment, the processor (140) may acquire user's surrounding information, identify steps on the user's path, and output an augmented reality image including information about the steps through the display unit (162) by executing one or more commands stored in the memory (150).

[0131] The output interface (160) may be configured to output an augmented reality image (virtual image) containing information about a step under the control of the processor (140), or to output an audio signal. The output interface (160) may include a display unit (162) and a speaker (164).

[0132] The display unit (162) may be composed of at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display.

[0133] In one embodiment, when the augmented reality device (100) is configured as augmented reality glasses, the display unit (162) may be configured as a lens optical system and may include a waveguide and an optical engine. The optical engine may be configured as a projector that generates light of an augmented reality image composed of text, icons, virtual images, etc. and projects the light onto a waveguide. The optical engine may include, for example, an image panel, an illumination optical system, a projection optical system, etc. In one embodiment, the optical engine may be disposed on a frame or a temple of the augmented reality glasses.

[0134] The speaker (164) is configured to output an audio signal. In one embodiment, the speaker (164) can output a voice message or notification sound under the control of the processor (140).

[0135] FIG. 9 is a diagram illustrating a glass-type augmented reality device (900) according to one embodiment of the present disclosure.

[0136] Referring to FIG. 9, an augmented reality device (900) that outputs an augmented reality image including information about steps located on a user's path is illustrated. The augmented reality device (900) is a device that can provide augmented reality (AR) related services, and may generally include AR glasses in the shape of glasses worn on the user's face, a head mounted display (HMD) worn on the head, a virtual reality headset (VRH), or an AR helmet. In the case of a head mounted device, a display can be placed in front of the user's eyes to provide a very large screen to the user, and since the screen moves according to the user's movements, a realistic virtual world can be provided.

[0137] In one embodiment, a user may wear an augmented reality device (900) capable of displaying visual augmented reality content. The augmented reality device (900) may include an audio module capable of providing audio augmented reality content to the user. In one embodiment, the augmented reality device (900) may include one or more cameras capable of capturing images and videos of an environment. The augmented reality device (900) may include an eye tracking system to determine the user's vergence distance. In one embodiment, the augmented reality device (900) may include a lightweight head-mounted display (HMD) (e.g., goggles, glasses, visor, etc.). In one embodiment, the augmented reality device (900) may include a device such as a lightweight portable display device or one or more laser projection glasses (e.g., glasses capable of projecting a low-powered laser onto a user's retina to project and display images or depth content to the user).

[0138] In one embodiment, the augmented reality device (900) may provide an AR service that outputs at least one virtual object to be overlapped in an area determined to be a user's field of view (FOV). For example, the area determined to be a user's field of view may be an area determined to be recognizable by a user wearing the augmented reality device (900) through the augmented reality device (900), and may be an area including all or at least a portion of the display of the augmented reality device (900). In one embodiment, the augmented reality device (900) may include a plurality of transparent members (920, 930) corresponding to each of the user's eyes.

[0139] In one embodiment, the augmented reality device (900) may include a display module (914), a camera, an audio output, and a support (921, 922).

[0140] The camera can capture images corresponding to the user's field of view or measure the distance to an object. The camera may correspond to the camera (120) of FIG. 8 described above. In one embodiment, the camera may be used for head tracking and spatial recognition. The camera may also recognize the user's movements.

[0141] In one embodiment, the camera may further include an 'ET (eye tracking) camera (912)' in addition to the camera (913) that acquires an image corresponding to the user's field of view, i.e., a real scene image, detects an object, or is used for spatial recognition. In one embodiment, the ET camera (912) may be used to detect and track the user's pupils. The ET camera (912) may be used to adjust the center of a virtual image projected on the augmented reality device (900) so that it is positioned according to the direction in which the pupils of the user wearing the augmented reality device (900) are looking. For example, the ET camera (912) may use a GS (Global shutter) camera to detect the pupil and track rapid eye movements without delay. The ET camera (912) may separately include a left-eye camera (912-1) and a right-eye camera (912-2).

[0142] In one embodiment, the display module (914) may include a first display (930) and a second display (920). The display module (914) may correspond to the display unit (162) of FIG. 8 described above. The virtual object output through the display module (914) may include a viewing window in which image data captured through another camera is displayed.

[0143] In one embodiment, the display (920, 930) may include a light-collecting lens or a waveguide (waveguide or waveguide) in a transparent member. For example, the transparent member may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be completely transparent or translucent. In one embodiment, the transparent member may include a first transparent member (930) facing the right eye of a user wearing the augmented reality device (900) and a second transparent member (920) facing the left eye of the user. If the display is transparent, it may be positioned facing the user's eyes to display a screen.

[0144] A waveguide can transmit light generated from a light source of a display to a user's eyes. For example, the waveguide can be at least partially positioned on a portion of a transparent member (920, 930). According to one embodiment, light emitted from the display can be incident on one end of the waveguide, and the incident light can be transmitted to the user's eyes through total internal reflection within the waveguide. The waveguide can be made of a transparent material such as glass, plastic, or polymer, and can include a nano-pattern formed on one surface of an inner or outer surface, for example, a grating structure having a polygonal or curved shape. In one embodiment, the incident light can be propagated or reflected within the waveguide by the nano-pattern and provided to the user's eyes. In one embodiment, the waveguide can include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a mirror). In one embodiment, the waveguide can guide display light emitted from a light source to a user's eye using at least one diffractive element or reflective element.

[0145] In one embodiment, the display (920, 930) may include a display panel or a lens (e.g., glass). For example, the display panel may include a transparent material, such as glass or plastic. In one embodiment, the display may be formed of a transparent element, allowing a user to see through the display and perceive the actual space behind the display. The display may display a virtual object, such as a viewing window, on at least a portion of the transparent element, such that the user appears to have a virtual object superimposed on at least a portion of the actual space.

[0146] In one embodiment, the support (921, 922) may include a printed circuit board (PCB) (931-1, 931-2) for transmitting electrical signals to each component of the augmented reality device (900), a speaker (932-1, 932-2) for outputting audio signals, or a battery (933-1, 933-2) for supplying power. The speaker (932-1, 932-2) for outputting audio signals may correspond to the speaker (164) of FIG. 8 described above. For example, in a glasses-type augmented reality device (900), the support (921, 922) may be arranged on the temple portion of the glasses. The support (921, 922) may include a hinge portion (940-1, 940-2) for coupling to the main body portion of the augmented reality device (900). The speakers (932-1, 932-2) may include a first speaker (932-1) for transmitting audio signals to the user's left ear and a second speaker (932-2) for transmitting audio signals to the user's right ear.

[0147] Referring to FIG. 9, the augmented reality device (900) may include a microphone (941) for receiving a user's voice and ambient sounds. In addition, the augmented reality device (900) may include at least one illumination LED (942) to increase the accuracy of at least one camera (e.g., an ET camera (912), an outward-facing camera (913), or a recognition camera (911-1, 911-2)). For example, the illumination LED (942) may be used as an auxiliary means to increase the accuracy when capturing a user's pupil with the ET camera (912), and the illumination LED (942) may use an IR LED of an infrared wavelength rather than a visible light wavelength. For example, the illumination LED (942) may be used as an auxiliary means when capturing a user's gesture, etc. with the recognition camera (911-1, 911-2) when it is difficult to detect a subject in a dark environment.

[0148] According to one embodiment, the display module (914) may include a first light guide plate (930) corresponding to the right eye and a second light guide plate (920) corresponding to the left eye, and may provide visual information to the user through the first light guide plate (930) and the second light guide plate (920). According to one embodiment, the display module (914) may include a display panel and a lens (e.g., a glass lens, an LC lens). The display panel may include a transparent material such as glass or plastic.

[0149] In one embodiment, the display module (914) may be formed of a transparent element, such that a user can see through the display module (914) to perceive a real space behind the display module (914) and in front of the user. The display module (914) may display a virtual object on at least a portion of the transparent element so that the user appears to have the virtual object superimposed on at least a portion of the real space.

[0150] In one embodiment, the augmented reality device (900) may acquire information about the user's surroundings through image information related to a real space acquired through an outward-facing camera (913). According to various embodiments, a display area where the augmented reality device (900) displays an augmented reality image including a virtual object may include a portion of the display module (914) (e.g., at least a portion of the display panel). According to one embodiment, the display area may be an area corresponding to at least a portion of the first light guide plate (930) and the second light guide plate (920).

[0151] In one embodiment, the augmented reality device (900) may use an outward-facing camera (913) to measure a distance to a physical object positioned in the front direction of the augmented reality device (900). The outward-facing camera (913) may include a high-resolution camera, such as a high-resolution (HR) camera and a photo video (PV) camera.

[0152] The augmented reality device (900) according to one embodiment of the present disclosure is not limited to the configuration described above, and may include various components in various locations and in various numbers.

[0153] The present disclosure provides a method for an augmented reality device to output an augmented reality image including information about a step located along a user's path. The method may include a step of acquiring information about the user's surroundings. The information about the user's surroundings may be composed of an RGB image or a depth map. The method may include a step of identifying a step along the user's path. The step may represent a ground surface with a gradual height difference, such as a staircase or a protruding obstacle. The method may include a step of outputting an augmented reality image including information about the step. By having the augmented reality device identify a step located along the user's path and provide information about the step to the user through an augmented reality image, the user can be assisted in recognizing the step and effectively provide a route guidance service to the user.

[0154] In one embodiment, the method may further include the step of the augmented reality device obtaining user information about the user. The augmented reality image including information about the step may be generated based on the user information.

[0155] In one embodiment, the user information may include at least one of stride information, disability information, vision information, whether the user uses a walking assistance device, or whether the user uses a mobility assistance device.

[0156] In one embodiment, the step of acquiring user's surrounding information may include at least one of: acquiring an image of the user's surrounding environment via a camera, acquiring a depth map of the user's surrounding environment via a depth sensor, or acquiring surrounding environment information previously stored in memory. If the electronic device acquires surrounding environment information previously stored in memory, it can notify the user of obstacles in the vicinity even when the user is not walking while looking ahead.

[0157] In one embodiment, the step of identifying the step may include the steps of identifying a direction of the step and identifying at least one step surface of the step.

[0158] In one embodiment, the step of identifying the step may further include the step of identifying movement of the step.

[0159] In one embodiment, the step of identifying the direction of the step may be identifying the direction of the step based on a depth map acquired through a depth sensor included in the augmented reality device and pose information of the augmented reality device at the time of acquiring the depth map.

[0160] In one embodiment, the step of identifying at least one step surface may be identifying at least one step surface by edge analyzing an image of the surrounding environment acquired through a camera included in the augmented reality device.

[0161] In one embodiment, the step of identifying the movement of the step may include, when the direction of the step is an upward step, identifying a change in the height of the first step and determining an upward movement if the height of the first step increases, and determining a downward movement if the height of the first step decreases, and, when the direction of the step is a downward step, identifying a change in at least one step surface and determining an upward movement if the area of ​​the first step surface decreases or the distance between the user and the edge of the at least one step surface decreases, and determining a downward movement if the area of ​​the first step surface increases or the distance between the user and the edge of the at least one step surface increases.

[0162] In one embodiment, an augmented reality image including information about a step may include at least one of an indicator indicating where the user should step, a notification guiding the timing of the user's step, or a directional indicator indicating the direction in which the step moves.

[0163] The present disclosure provides an augmented reality device that outputs an augmented reality image including information about steps located on a user's path. The augmented reality device may include a camera configured to acquire a real-world scene image, a display unit, a memory storing a program including at least one command, and at least one processor. The at least one processor may acquire information about the user's surroundings, identify steps on the user's path, and output an augmented reality image including information about the steps through the display unit.

[0164] In one embodiment, at least one processor may obtain user information about a user and generate an augmented reality image including information about a step based on the user information.

[0165] In one embodiment, the user information may include at least one of stride information, disability information, vision information, whether the user uses a walking assistance device, or whether the user uses a mobility assistance device.

[0166] In one embodiment, the augmented reality device may further include a sensor configured to acquire a depth map. At least one processor may acquire an image of the user's surroundings via a camera, acquire a depth map of the user's surroundings via a sensor, or acquire surroundings information previously stored in memory.

[0167] In one embodiment, at least one processor can identify a direction of a step and identify at least one step surface of the step.

[0168] In one embodiment, at least one processor can identify movement of a step.

[0169] In one embodiment, the augmented reality device may further include a sensor configured to acquire a depth map. At least one processor may identify the direction of the step based on the depth map acquired through the depth sensor and pose information of the augmented reality device at the time of acquiring the depth map.

[0170] In one embodiment, at least one processor can identify at least one step surface by edge analyzing an image of the surrounding environment acquired through a camera.

[0171] In one embodiment, at least one processor may determine, when the direction of the steps is upward, a change in the height of a first step and determine an upward movement if the height of the first step increases, and determine a downward movement if the height of the first step decreases, and when the direction of the steps is downward, determine a change in at least one step surface and determine, when the width of the first step surface decreases or the distance between the user and the edge of at least one step surface decreases, an upward movement if the width of the first step surface increases or the distance between the user and the edge of at least one step surface increases, a downward movement if the width of the first step surface increases.

[0172] The present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may store instructions readable by an augmented reality device for executing at least one of the disclosed methods on the augmented reality device.

[0173] In this way, according to one embodiment of the present disclosure, the augmented reality device can assist the user in recognizing the step by identifying a step located on the user's path and providing the user with information about the step as an augmented reality image, thereby effectively providing the user with path guidance.

[0174] Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and embodied in a computer-readable medium. In the present disclosure, “application” and “program” may refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, related data, or portions thereof suitable for implementation in computer-readable program code. “Computer-readable program code” may include various types of computer code, including source code, object code, and executable code. “Computer-readable medium” may include various types of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a compact disc (CD), a digital video disc (DVD), or various types of memory.

[0175] Additionally, the device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a 'non-transitory storage medium' is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is permanently stored in the storage medium and cases where it is temporarily stored. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. A computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and non-volatile media, removable and non-removable media. A computer-readable medium includes a medium on which data can be permanently stored and a medium on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable memory device.

[0176] The disclosed embodiments can be implemented as a software program including commands stored on a computer-readable storage medium. A computer, as a device capable of recalling commands stored on the storage medium and performing operations according to the disclosed embodiments in accordance with the recalled commands, can include an electronic device according to the disclosed embodiments.

[0177] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may include a software program and a computer-readable storage medium having the software program stored thereon. For example, the computer program product may include a product in the form of a software program (e.g., a downloadable app) distributed electronically by a device manufacturer or through an electronic market. For electronic distribution, at least a portion of the software program may be stored in the storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic market server, or a relay server that temporarily stores the software program.

[0178] In a system comprising a server and a device, the computer program product may include a storage medium of the server or a storage medium of the device. Alternatively, if a third device (e.g., a smartphone) exists that is communicatively connected to the server or device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the server to the device or the third device, or from the third device to the device.

[0179] In this case, one of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0180] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on the server to control a device in communication with the server to perform a method according to the disclosed embodiments.

[0181] As another example, a third device may execute a computer program product to control a device in communication with the third device to perform a method according to the disclosed embodiments. When the third device executes the computer program product, the third device may download the computer program product from a server and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a preloaded state to perform the method according to the disclosed embodiments.

[0182] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0183] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. In a method for outputting an augmented reality image including information about a step located on a user's path by an augmented reality device (100), A step of acquiring surrounding information of the user; A step of identifying a step on the user's path; A method comprising the step of outputting an augmented reality image including information about the above step.

2. In paragraph 1, Further comprising a step of obtaining user information about the user; A method wherein the augmented reality image including information about the step is generated based on the user information.

3. In any one of paragraphs 1 and 2, The step of obtaining the user's surrounding information is as follows: A step of acquiring an image of the user's surrounding environment through a camera (120); A step of acquiring a depth map of the user's surrounding environment through a depth sensor (130); or A method comprising at least one step of acquiring surrounding environment information stored in a memory (150).

4. In any one of paragraphs 1 to 3, The step of identifying the above step is: a step of identifying the direction of the step; and A method comprising the step of identifying at least one step surface of the step.

5. In paragraph 4, A method wherein the step of identifying the step further comprises the step of identifying the movement of the step.

6. In paragraph 4 or 5, A method in which the step of identifying the direction of the step is performed based on a depth map acquired through a depth sensor (130) included in the augmented reality device (100) and pose information of the augmented reality device (100) at the time of acquiring the depth map.

7. In any one of paragraphs 4 to 6, A method in which the step of identifying at least one step surface is performed by edge-analyzing an image of the surrounding environment acquired through a camera (120) included in the augmented reality device (100) to identify the at least one step surface.

8. In an augmented reality device (100) that outputs an augmented reality image, A camera (120) configured to acquire a real-world scene image; Display section (162); A memory (150) storing a program including at least one instruction; and comprising at least one processor (140), At least one processor (140) above, Obtain information about the user's surroundings, Identify the steps on the user's path, An augmented reality device (100) that outputs an augmented reality image including information about the step through the display unit (162).

9. In paragraph 8, At least one processor (140) above, Obtain user information about the above user, An augmented reality device (100) that generates an augmented reality image including information about the step based on the user information.

10. In paragraph 8 or 9, Further comprising a sensor (130) configured to obtain a depth map, At least one processor (140) above, Obtaining an image of the user's surroundings through the camera (120), or Obtain a depth map of the user's surrounding environment through the above sensor (130), or An augmented reality device (100) that acquires surrounding environment information previously stored in the above memory (150).

11. In any one of paragraphs 8 to 10, At least one processor (140) above, Identify the direction of the above steps, An augmented reality device (100) that identifies at least one step surface of the above step.

12. In paragraph 11, An augmented reality device (100) wherein at least one processor (140) identifies the movement of the step.

13. In paragraph 11 or 12, Further comprising a sensor (130) configured to obtain a depth map, An augmented reality device (100) in which at least one processor (140) identifies the direction of the step based on the depth map acquired through the depth sensor (130) and the pose information of the augmented reality device (100) at the time of acquiring the depth map.

14. In any one of paragraphs 11 to 13, An augmented reality device (100) in which at least one processor (140) identifies at least one step surface by edge-analyzing an image of the surrounding environment acquired through the camera (120).

15. A computer-readable recording medium having recorded thereon a program for performing the method of any one of clauses 1 to 7 on a computer.

Images