Augmented reality device for providing augmented reality service matched to context of real-world space and operating method therefor

Abstract

Disclosed are an augmented reality device for providing an augmented reality service matched to a real-world space, and an operating method therefor. The augmented reality device according to an embodiment of the present disclosure may recognize a plurality of objects in a real-world space from an image acquired through a camera, recognize a relative positional relationship including positions and directions between the recognized plurality of objects, and display a virtual object indicating information relating to the recognized plurality of objects at a preconfigured position, on the basis of pre-stored information relating to the relative positional relationship.

Description

Augmented reality device and its operation method that provide augmented reality service matching the context of real space

[0001] The present disclosure relates to an augmented reality device that provides an augmented reality service that matches the context of a real space, and a method for operating the same. Specifically, the present disclosure discloses an augmented reality device that recognizes the context of a space in which a user wearing the augmented reality device is located, and provides an augmented reality service that displays information corresponding to the recognized context, and a method for operating the same.

[0002] Augmented Reality (AR) is a technology that overlays virtual images onto the physical environment or real world objects of the real world. AR devices (e.g., smart glasses) that utilize AR technology are being used in everyday life for purposes such as information retrieval, route guidance, and camera shooting. In particular, smart glasses are worn as fashion items and are mainly used for outdoor activities.

[0003] Augmented reality devices allow users to record and observe information about real-world spaces in all areas of daily life. Users wearing AR devices while working, playing games, enjoying leisure activities, or simply relaxing must manually determine and launch applications for specific spaces as they move around. For users who frequently move around, pre-determining and launching applications for specific spaces can be cumbersome and degrade the user experience.

[0004] One aspect of the present disclosure provides a method for an augmented reality device to provide an augmented reality service that matches a real space. The method for operating the augmented reality device may include a step of recognizing a plurality of objects in a real space from an image acquired through a camera. The method for operating the augmented reality device may include a step of recognizing a relative positional relationship, including a position and direction, between the plurality of recognized objects. The method for operating the augmented reality device may include a step of displaying a virtual object representing information related to the plurality of recognized objects at a preset position based on previously stored information regarding the relative positional relationship.

[0005] Another aspect of the present disclosure provides an augmented reality device that provides an augmented reality service that matches a real space. The augmented reality device of the present disclosure may include a camera that captures an image of a real space, a memory that stores at least one instruction, at least one processor that executes the at least one instruction, and a display unit. The at least one processor may recognize a plurality of objects in a real space from an image acquired through the camera. The at least one processor may recognize a relative positional relationship including a position and direction between the recognized plurality of objects, and control the display unit to display a virtual object representing information related to the recognized plurality of objects at a preset position based on information about the relative positional relationship that is previously stored in the memory.

[0006] Another aspect of the present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an augmented reality device, such that the augmented reality device performs the following operations: recognizing a plurality of objects in a real space from an image acquired through a camera; recognizing a relative positional relationship including positions and directions between the recognized plurality of objects; and displaying a virtual object representing information related to the recognized plurality of objects at a preset position based on previously stored information about the relative positional relationship.

[0007] The present disclosure can be readily understood by the combination of the following detailed description and the accompanying drawings, wherein reference numerals refer to structural elements.

[0008] FIG. 1 is a conceptual diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to provide an augmented reality service that matches the context of a real space.

[0009] FIG. 2 is a flowchart illustrating a method of operating an augmented reality device according to one embodiment of the present disclosure.

[0010] FIG. 3 is a block diagram illustrating components of an augmented reality device according to one embodiment of the present disclosure.

[0011] FIG. 4 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to identify a matching spatial context preset based on a scene graph of a real space.

[0012] FIG. 5A is a diagram illustrating a scene graph obtained by an augmented reality device based on distances between objects according to one embodiment of the present disclosure.

[0013] FIG. 5b is a diagram illustrating a scene graph obtained by an augmented reality device according to one embodiment of the present disclosure based on direction vectors of objects.

[0014] FIG. 5c is a diagram illustrating a scene graph obtained by an augmented reality device according to one embodiment of the present disclosure based on sign vectors of objects.

[0015] FIG. 6 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to measure the similarity of a scene graph and identify a spatial context preset that matches a spatial context of a real space.

[0016] FIG. 7 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to measure the similarity of a sub-graph of a scene graph and identify a spatial context preset that matches a spatial context of a real space.

[0017] FIG. 8 is a flowchart illustrating operations performed by an augmented reality device according to one embodiment of the present disclosure depending on whether a spatial context preset matching a spatial context of a real space is identified.

[0018] FIG. 9 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to merge a new spatial context with an existing spatial context preset.

[0019] FIG. 10 is a flowchart illustrating a method for an augmented reality device to display a virtual object matching a spatial context according to one embodiment of the present disclosure.

[0020] FIG. 11 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to display a virtual object matching a spatial context.

[0021] FIG. 12 is a flowchart illustrating a method for an augmented reality device to display a new virtual object according to one embodiment of the present disclosure.

[0022] FIG. 13 is a diagram illustrating an operation of an augmented reality device displaying a new virtual object according to one embodiment of the present disclosure.

[0023] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0024] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0025] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0026] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0027] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0028] In this disclosure, 'Augmented Reality' means displaying a virtual image together within a physical environment space of the real world or displaying a real object and a virtual image together.

[0029] In the present disclosure, an 'augmented reality device' is a device capable of expressing augmented reality, and may be, for example, not only augmented reality glasses in the shape of glasses worn on the face by a user, but also a head-mounted display apparatus (HMD: Head Mounted Display Apparatus) worn on the head, or an augmented reality helmet.

[0030] In the present disclosure, "spatial context" refers to information indicating the characteristics of a physical environment space in the real world, such as information on the characteristics of a space for work, study, leisure, games, rest, etc. In one embodiment of the present disclosure, the spatial context may indicate the relative positional relationship between a plurality of objects within a space in the real world. In one embodiment of the present disclosure, the spatial context may include information regarding the positions and directions between a plurality of real objects.

[0031] In the present disclosure, functions related to 'artificial intelligence' are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0032] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0033] In the present disclosure, the 'artificial intelligence model' may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs neural network operations through operations between the operation results of the previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the plurality of weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network model may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0034] In the present disclosure, "vision recognition" refers to image signal processing that inputs an image into an artificial intelligence model and, through inference using the artificial intelligence model, recognizes (detection) an object from the input image, classifies the object into a specific category, or segments the object. In one embodiment of the present disclosure, vision recognition may refer to image processing that uses an artificial intelligence model to recognize an object from an image captured by a camera and obtains classification information or type information about the object.

[0035] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0036] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0037] FIG. 1 is a conceptual diagram for explaining an operation of an augmented reality device (100) according to one embodiment of the present disclosure to provide an augmented reality service matching the context of a real space (10).

[0038] The augmented reality device (100) is a device capable of expressing augmented reality, and may be configured as, for example, augmented reality glasses in the shape of glasses that a user wears on his or her face. In FIG. 1, the augmented reality device (100) is illustrated as augmented reality glasses, but is not limited thereto. For example, the augmented reality device (100) may be implemented as a head-mounted display apparatus (HMD) worn on the head, an augmented reality helmet, or the like.

[0039] Referring to FIG. 1, the augmented reality device (100) includes a camera (110), and the camera (110) may be configured with a left-eye camera (110L) and a right-eye camera (110R). The left-eye camera (110L) is positioned adjacent to the user's left eye when the user wears the augmented reality device (100) and may capture a real space (10) to obtain a left-eye image. The right-eye camera (110R) is positioned adjacent to the user's right eye when the user wears the augmented reality device (100) and may capture a real space (10) to obtain a right-eye image. In one embodiment of the present disclosure, the left-eye camera (110L) and the right-eye camera (110R) may configure a stereo camera that acquires a three-dimensional position coordinate value of an object through triangulation based on a two-dimensional image acquired in an area where their fields of view overlap and a positional relationship between the cameras. In FIG. 1, the plurality of cameras (110L, 110R) are illustrated and described as including two cameras, but are not limited thereto. The augmented reality device (100) may also include three or more cameras.

[0040] In FIG. 1, only the minimum components for explaining the function and / or operation of the augmented reality device (100) are illustrated, and the components included in the augmented reality device (100) are not limited to those illustrated in FIG. 1. The components of the augmented reality device (100) will be described in detail in FIG. 3.

[0041] In the embodiment illustrated in FIG. 1, the augmented reality device (100) recognizes a plurality of objects (11, 12, 13, 14, 15) in a real space (10) from images acquired through a left-eye camera (110L) and a right-eye camera (110R) (operation ①), recognizes a spatial context of the real space (10) based on a relative positional relationship between the recognized plurality of objects (11, 12, 13, 14, 15) (operation ②), compares the recognized spatial context with a spatial context preset previously stored in a memory (140, see FIG. 3) to identify a matching spatial context preset (operation ③), and displays a virtual object acquired from the identified spatial context preset (operation ④). Hereinafter, the function and / or operation of the augmented reality device (100) will be described in detail with reference to FIGS. 1 and 2 together.

[0042] FIG. 2 is a flowchart illustrating an operation method of an augmented reality device (100) according to one embodiment of the present disclosure.

[0043] In step S210, the augmented reality device (100) recognizes a plurality of objects in a real space from images acquired through a camera. Referring to FIG. 1, the augmented reality device (100) includes a left-eye camera (110L) and a right-eye camera (110R), and can acquire a left-eye image by photographing a real space (10) using the left-eye camera (110L), and can acquire a right-eye image by photographing a real space (10) using the right-eye camera (110R). The augmented reality device (100) can perform vision recognition using an artificial intelligence model to recognize a plurality of objects (11, 12, 13, 14, 15) in the real space (10) from the left-eye image and the right-eye image. In one embodiment of the present disclosure, the artificial intelligence model may be a deep neural network model trained to recognize an object from an image and output classification information of the object through a supervised learning method that applies an image as input and applies a label value representing classification information of the object as the ground truth. The deep neural network model may be, for example, a convolutional neural network (CNN) model, but is not limited thereto. In the embodiment illustrated in FIG. 1, the augmented reality device (100) inputs an image into a deep neural network model, recognizes a plurality of objects (11, 12, 13, 14, 15) from the image through inference using the deep neural network model, and obtains 'monitor' as classification information of a first object (11), 'laptop' as classification information of a second object (12), 'keyboard' as classification information of a third object (13), 'mouse' as classification information of a fourth object (14), and 'chair' as classification information of a fifth object (15).

[0044] In step S220 of FIG. 2, the augmented reality device (100) recognizes a relative positional relationship including positions and directions between a plurality of recognized objects. In one embodiment of the present disclosure, the augmented reality device (100) can recognize a spatial context of a real space based on the relative positional relationship between a plurality of objects. In the present disclosure, the 'spatial context' refers to information indicating characteristics of a physical environment space in the real world, and means, for example, characteristic information of a space for work, study, leisure, games, rest, etc. In one embodiment of the present disclosure, the spatial context may include a scene graph indicating the relative positional relationship between a plurality of objects.

[0045] Referring to FIG. 1 together, the augmented reality device (100) can obtain three-dimensional position coordinate information of each of a plurality of objects (11, 12, 13, 14, 15) recognized from an image, and can obtain a relative positional relationship of the plurality of objects (11, 12, 13, 14, 15) based on the obtained three-dimensional position coordinate information. In one embodiment of the present disclosure, the augmented reality device (100) can obtain three-dimensional position coordinate values ​​of objects including depth values ​​(z-axis coordinate information) of the plurality of objects (11, 12, 13, 14, 15) through a stereo vision method using a left-eye image obtained from a left-eye camera (110L) and a right-eye image obtained from a right-eye camera (110R). The augmented reality device (100) can obtain a relative positional relationship regarding a three-dimensional distance and direction between a plurality of objects (11, 12, 13, 14, 15) based on the obtained three-dimensional positional coordinate values. In one embodiment of the present disclosure, the relative positional relationship can include at least one of information on a distance, a direction vector, and a sign vector of the plurality of objects (11, 12, 13, 14, 15) in a three-dimensional space. In one embodiment of the present disclosure, the augmented reality device (100) can obtain a scene graph (20) based on classification information or type information of the plurality of objects (11, 12, 13, 14, 15) and the relative positional relationship between the plurality of objects (11, 12, 13, 14, 15). The scene graph (20) may include a plurality of nodes representing classification information or types of each of a plurality of objects (11, 12, 13, 14, 15) and a plurality of edges representing relative positional relationships between the plurality of nodes.

[0046] In step S230 of FIG. 2, the augmented reality device (100) displays a virtual object representing information related to a plurality of recognized objects at a preset location based on information regarding a previously stored relative positional relationship. In one embodiment of the present disclosure, the augmented reality device (100) may identify a spatial context preset that matches the recognized spatial context based on the relative positional relationship between the plurality of objects. In one embodiment of the present disclosure, the augmented reality device (100) may pre-store a spatial context preset that represents the characteristics of a space, such as work, shared space, game, leisure, and rest, based on the relative positional relationship between objects. The augmented reality device (100) may measure the similarity by comparing a spatial context recognized from an image with a pre-stored spatial context preset, and identify a spatial context preset that matches the recognized spatial context based on the measured similarity.

[0047] Referring also to FIG. 1, the augmented reality device (100) may include a spatial context database (146) storing spatial context presets (31, 32, 33, ...). In one embodiment of the present disclosure, the spatial context presets (31, 32, 33, ...) stored in the spatial context database (146) may include a scene graph representing the relative positional relationship of objects according to the characteristics of the space. For example, a first spatial context preset (31) may include a scene graph regarding the relative positional relationship of objects representing a work space, a second spatial context preset (32) may include a scene graph regarding the relative positional relationship of objects representing a game space, and a third spatial context preset (33) may include a scene graph regarding the relative positional relationship of objects representing a rest space. The augmented reality device (100) can measure similarity by comparing a scene graph (20) representing the relative positional relationship of a plurality of objects (11, 12, 13, 14, 15) recognized from an image with scene graphs of a plurality of spatial context presets (31, 32, 33, ...) pre-stored in a spatial context database (146). The augmented reality device (100) can identify a scene graph whose measured similarity exceeds a preset threshold, thereby identifying a spatial context preset that matches a spatial context among the plurality of spatial context presets (31, 32, 33, ...).

[0048] In one embodiment of the present disclosure, the augmented reality device (100) obtains a virtual object from a positional relationship preset and displays the obtained virtual object at a preset position. In one embodiment of the present disclosure, the augmented reality device (100) may obtain a virtual object from a spatial context preset that matches a recognized spatial context based on a relative positional relationship between a plurality of objects, and obtain display position information of the virtual object. In the present disclosure, a 'virtual object' means a virtual graphic object composed of text, an image, or a combination thereof that is related to an object in a real space or represents information provided by an application executed by the augmented reality device (100). In one embodiment of the present disclosure, the virtual object may be displayed in the form of a graphical user interface (GUI) or a widget. In one embodiment of the present disclosure, the augmented reality device (100) may display the virtual object at a preset position based on information regarding the display position of the virtual object.

[0049] Referring to FIG. 1 together, the augmented reality device (100) can load and acquire virtual objects (41, 42, 43) from a spatial context preset identified from a spatial context database (146), and display the acquired virtual objects (41, 42, 43) at preset display positions. The augmented reality device (100) can acquire information about the display positions of the virtual objects (41, 42, 43) from the spatial context preset. In one embodiment of the present disclosure, the display positions of the virtual objects (41, 42, 43) can be preset as relative positions with respect to positions of a plurality of objects (11, 12, 13, 14, 15) within a real space (10) or as specific positions with respect to the position of the augmented reality device (100). In the embodiment illustrated in FIG. 1, the first virtual object (41) is a To do list widget for tasks, and is preset to be displayed adjacent to the right side of the monitor, the second virtual object (42) is a graphic UI indicating the progress of a task, and may be preset to be displayed adjacent to the upper side of the monitor. The third virtual object (43) is a calendar widget, and may be preset to be displayed adjacent to the upper left side of the lens portion of the augmented reality device (100).

[0050] The augmented reality device (100) can display virtual objects (41, 42, 43) based on preset display position information. In the embodiment illustrated in FIG. 1, the augmented reality device (100) can display the first virtual object (41) at a position spaced apart from the right side of the monitor by a preset position, and can display the second virtual object (42) at a position spaced apart from the upper side of the monitor by a preset position. The augmented reality device (100) can display the third virtual object (43) so that the third virtual object (43) is displayed at the upper left of the lens based on the position of the augmented reality device (100).

[0051] An augmented reality device (100) can record and observe information about real space (10) in all areas of daily life. Users wearing the augmented reality device (100) while working, playing games, engaging in leisure activities, or resting must determine and execute applications to be used in specific spaces as they move through the space. If the user frequently moves around, the act of determining and executing applications to be used in specific spaces in advance can be cumbersome and degrade the user experience.

[0052] The present disclosure aims to provide an augmented reality device (100) and an operating method thereof that recognizes the relative positional relationship between a plurality of objects in a real space (10) where a user is located, and provides and displays information corresponding to the recognized relative positional relationship to provide a continuous augmented reality experience to the user.

[0053] The augmented reality device (100) of the present disclosure can recognize a plurality of objects (11, 12, 13, 14, 15) included in a real space (10), recognize a relative positional relationship between the recognized plurality of objects (11, 12, 13, 14, 15), and display virtual objects (41, 42, 43) related to the plurality of objects (11, 12, 13, 14, 15) at preset display locations based on previously stored positional relationship information. Through this, the augmented reality device (100) of the present disclosure provides a technical effect of improving the efficiency of daily life or work by enhancing the user's cognitive ability. The augmented reality device (100) according to one embodiment of the present disclosure can individually configure and manage environments for multiple spaces having the same spatial context, such as Work From Anywhere, thereby providing continuity of work, games, leisure, or rest, etc., in various environments. In addition, the augmented reality device (100) according to one embodiment of the present disclosure can provide a consistent user experience for each spatial context, and can recognize the environment according to the configuration of the current real space (10) and continuously provide information (e.g., virtual objects (41, 42, 43)).

[0054] FIG. 3 is a block diagram illustrating components of an augmented reality device (100) according to one embodiment of the present disclosure.

[0055] Referring to FIG. 3, the augmented reality device (100) may include a camera (110), a sensor (120), a processor (130), a memory (140), and a display unit (150). The camera (110), the sensor (120), the processor (130), the memory (140), and the display unit (150) may be electrically and / or physically connected to each other, respectively. Only components for explaining the operation of the augmented reality device (100) are illustrated in FIG. 3, and the components included in the augmented reality device (100) are not limited to those illustrated in FIG. 3. In one embodiment of the present disclosure, the augmented reality device (100) may further include a communication interface for performing data communication with an external device or server. In one embodiment of the present disclosure, the augmented reality device (100) is implemented as a portable device, and in this case, the augmented reality device (100) may further include a battery that supplies driving power to a camera (110), a sensor (120), a processor (130), and a display unit (150).

[0056] The camera (110) is configured to acquire an image of an object by photographing the object in a real space. The camera (110) may include a lens module, an image sensor, and an image processing module. The camera (110) may acquire a still image or video of the object by the image sensor (e.g., CMOS or CCD). The video may include a plurality of image frames continuously acquired by photographing the object through the camera (110). The image processing module may encode a still image composed of a single image frame acquired through the image sensor or video data composed of a plurality of image frames and transmit the encoded data to the processor (130).

[0057] In one embodiment of the present disclosure, the camera (110) may include a left-eye camera positioned adjacent to the user's left eye when the user wears the augmented reality device (100) and capturing a left-eye image by photographing a real space, and a right-eye camera positioned adjacent to the user's right eye and capturing a right-eye image by photographing a real space. The left-eye camera and the right-eye camera may constitute a stereo camera. However, the present invention is not limited thereto, and the camera (110) may be implemented as any type of camera known in the art, such as an RGB-depth camera, a Time-of-Flight (ToF) camera, a grayscale camera, or an infrared camera.

[0058] The sensor (120) may include a position sensor (122) and an IMU sensor (124).

[0059] The position sensor (122) is a sensor configured to obtain position information of the augmented reality device (100) in a three-dimensional space. The position sensor (122) may include, for example, a GPS (Global Positioning System) sensor that obtains three-dimensional position coordinate value information of the augmented reality device (100).

[0060] The IMU sensor (Inertial Measurement Unit) (124) is a sensor configured to measure the movement speed, direction, angle, and gravitational acceleration of the augmented reality device (100). The IMU sensor (124) may further include an acceleration sensor, a gyro sensor, and a magnetometer. The accelerometer is a sensor configured to measure acceleration according to a change in movement when a dynamic force such as acceleration, vibration, or impact is generated in the augmented reality device (100). In one embodiment of the present disclosure, the acceleration sensor may be configured as a three-axis accelerometer that measures acceleration in the horizontal direction, the horizontal direction, and the height direction. The gyro sensor is a sensor configured to measure angular velocity, which is a rotational change amount of the augmented reality device (100). In one embodiment of the present disclosure, the gyro sensor may include a three-axis gyro sensor that measures roll, pitch, and yaw angular velocity.

[0061] The processor (130) can execute one or more instructions of a program stored in the memory (140). The processor (130) may be configured with hardware components that perform arithmetic, logic, and input / output operations and image processing. Although the processor (130) is illustrated as a single element in FIG. 3 , it is not limited thereto. In one embodiment of the present disclosure, the processor (130) may be configured with one or more multiple elements. One or more processors included in the processor (140) may be circuitry such as a system on chip (SoC), an integrated circuit (IC), etc. The processor (130) may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), a graphics-only processor such as a graphics processing unit (GPU), a vision processing unit (VPU), or an artificial intelligence-only processor such as a neural processing unit (NPU). The processor (130) can be controlled to process input data according to a predefined operating rule or artificial intelligence model by executing at least one instruction or program code stored in the memory (150). Alternatively, if the processor (130) is an artificial intelligence-only processor, the artificial intelligence-only processor can be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0062] The memory (140) may be configured as at least one type of storage medium, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk.

[0063] The memory (140) may store instructions related to functions and / or operations that allow the augmented reality device (100) to recognize relative positional relationships between a plurality of objects in a real space, identify positional relationship information matching the recognized relative positional relationships, and display virtual objects acquired from the identified positional relationship information at preset display locations to provide augmented reality services matching the real space. In one embodiment of the present disclosure, the memory (140) may store at least one of instructions, an algorithm, a data structure, a program code, and an application program that can be read by the processor (130). The instructions, algorithms, data structures, and program codes stored in the memory (140) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0064] The memory (140) may store instructions, algorithms, data structures, or program codes related to a spatial context recognition module (142), a spatial context matching module (144), and a spatial context database (146). A 'module' included in the memory (140) refers to a unit that processes a function or operation performed by the processor (130), and this may be implemented as software such as instructions, algorithms, data structures, or program codes.

[0065] In the following embodiments, the processor (130) may be implemented by executing instructions or program codes stored in the memory (140).

[0066] The spatial context recognition module (142) is configured with commands or program codes related to a function and / or operation of recognizing a plurality of objects in real space from an image and recognizing a spatial context based on a relative positional relationship between the plurality of recognized objects. The processor (130) can recognize a plurality of objects in real space from an image acquired by the camera (110) by executing the commands or program codes of the spatial context recognition module (142). In one embodiment of the present disclosure, the spatial context recognition module (142) may include an artificial intelligence model trained to recognize objects from images. The artificial intelligence model may be a deep neural network model trained to recognize objects through a supervised learning method that applies bounding box images that can be recognized as objects from tens of thousands or hundreds of millions of input images as input data and applies label values ​​regarding ground truth values ​​of objects within the bounding boxes as output. Deep neural network models can be implemented as object recognition models, such as, but not limited to, a Convolutional Neural Network model, a Region-based Convolutional Neural Network (R-CNN), YOLO v4, CenterNet, or MobileNet.

[0067] However, it is not limited to this, and visual cues such as markers, for example, QR codes, are attached to objects in real space, and the processor (130) can recognize objects based on visual cues captured through the camera (110).

[0068] The processor (130) can recognize a spatial context based on a relative positional relationship including the positions and directions of a plurality of recognized objects. In one embodiment of the present disclosure, the 'relative positional relationship' may include at least one of a distance, a direction vector, and a sign vector in a three-dimensional space between a plurality of objects. In the present disclosure, the 'spatial context' refers to information indicating the characteristics of a physical environment space in the real world, and means, for example, characteristic information of a space for work, study, leisure, games, rest, etc.

[0069] In one embodiment of the present disclosure, the processor (130) may obtain three-dimensional position coordinate information of each of a plurality of objects from images acquired from the camera (110), and may obtain a relative positional relationship between the plurality of objects based on the acquired three-dimensional position coordinate information of the plurality of objects. The camera (110) may include a left-eye camera and a right-eye camera, and the processor (130) may obtain three-dimensional position coordinate information including a depth value of the object through a stereo vision method using a left-eye image acquired from the left-eye camera and a right-eye image acquired from the right-eye camera. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the camera (110) may be configured as a ToF (Time-of-Flight) camera, and the processor (130) may obtain three-dimensional position coordinate information of the object using the ToF camera. The processor (130) may obtain a relative positional relationship regarding distance and direction in a three-dimensional space between the plurality of objects based on the acquired three-dimensional position coordinate values ​​of the objects. In one embodiment of the present disclosure, information regarding direction can be obtained in the form of a direction vector or a sine vector. The processor (130) can recognize a spatial context based on the relative positional relationship between recognized objects.

[0070] In one embodiment of the present disclosure, the processor (130) may obtain a scene graph representing the relative positional relationship of a plurality of objects. The 'scene graph' may include nodes representing the types or classified categories of a plurality of objects recognized from an image, and edges representing the relative positional relationship between the nodes of the plurality of objects. In one embodiment of the present disclosure, the edges may represent a relative positional relationship including at least one of a distance, a direction vector, or a sine vector in a three-dimensional space between the plurality of nodes. The scene graph will be described in detail with reference to FIGS. 5A to 5C .

[0071] The spatial context matching module (144) is configured with commands or program codes related to functions and / or operations for comparing the recognized spatial context with spatial context presets pre-stored in the spatial context database (146) and identifying a spatial context preset matching the recognized spatial context. The spatial context database (146) may store at least one spatial context preset preset that is preset for at least one spatial characteristic. The processor (130) may execute the commands or program codes of the spatial context matching module (144) to compare the spatial context recognized from the image with at least one spatial context preset pre-stored in the spatial context database (146) and identify a spatial context preset matching the spatial context based on the comparison result. In one embodiment of the present disclosure, the processor (130) may measure similarity by comparing a scene graph of a spatial context with a scene graph of at least one spatial context preset pre-stored in the spatial context database (146). The processor (130) can identify a spatial context preset that matches the spatial context based on the result of comparing the measured similarity with a preset threshold. A specific embodiment in which the processor (130) measures the similarity between the scene graph of the spatial context and the scene graph of the previously stored spatial context preset, and identifies a spatial context preset that matches the spatial context based on the measured similarity will be described in detail with reference to FIGS. 4 to 6.

[0072] In one embodiment of the present disclosure, a scene graph may include at least one sub-graph having a hierarchical structure. In this case, the processor (130) may measure similarity by comparing each of the at least one sub-graph with a scene graph of a spatial context preset previously stored in the spatial context database (146), and may identify a spatial context preset having a scene graph whose measured similarity exceeds a preset threshold. A specific embodiment in which the processor (130) measures the similarity of a scene graph formed in a hierarchical structure and identifies a spatial context preset based on the measured similarity will be described in detail with reference to FIG. 7.

[0073] In one embodiment of the present disclosure, if a spatial context preset matching the spatial context recognized from an image is not identified, the processor (130) may generate a new spatial context preset based on information regarding the relative positional relationship between multiple objects constituting the spatial context. The functions and / or operations of the processor (130) depending on whether the spatial context matches will be described in detail in FIG. 8.

[0074] The spatial context database (146) is a storage device that stores at least one spatial context preset that is preset for at least one spatial characteristic, and may be configured with a non-volatile memory. Non-volatile memory refers to a storage medium that stores and maintains information even when power is not supplied, and can use the stored information again when power is supplied. The non-volatile memory may be configured with at least one of, for example, a flash memory, a hard disk, an SSD (Solid State Drive), a multimedia card micro type, an external memory of a card type (e.g., SD or XD memory, etc.), a ROM (Read Only Memory; ROM), a magnetic disk, or an optical disk. In FIG. 3, the spatial context database (146) is illustrated as a component included in the memory (140), but the present disclosure is not limited thereto. In one embodiment of the present disclosure, the spatial context database (146) may be configured as a database within the augmented reality device (100), which is a separate component from the memory (140). However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the spatial context database (146) may be configured as a web storage or cloud server that is accessible via a network and performs a storage function. In this case, the augmented reality device (100) may communicate with the web storage or cloud server via a communication interface and perform data transmission and reception to access the spatial context preset.

[0075] The processor (130) can obtain a virtual object representing information related to a plurality of objects from a spatial context preset identified from the spatial context database (146). In the present disclosure, a 'virtual object' means a virtual graphic object composed of text, an image, or a combination thereof representing information related to an object in a real space or provided by an application executed by the augmented reality device (100). In one embodiment of the present disclosure, the virtual object can be displayed in the form of a graphical user interface (GUI) or a widget. The processor (130) can control the display unit (150) so that the virtual object is displayed through the display unit (150).

[0076] The processor (130) may obtain display position information of a virtual object from a spatial context preset and display the virtual object based on the display position information. In one embodiment of the present disclosure, the 'display position information' may be set to a relative position based on the positions of objects in real space or may be set to a relative position based on the position of the augmented reality device (100). In the case of a virtual object for which the display position information is set to a relative position based on the positions of objects in real space, the processor (130) may identify an object that matches a plurality of objects included in the spatial context preset among a plurality of objects in real space recognized from an image, and display the virtual object at a display position for the identified object. In one embodiment of the present disclosure, the processor (130) may display the virtual object at a display position adjacent to an object in real space by projecting at least one of a virtual image, an icon, a letter, a number, a special symbol, and a combination thereof included in a graphical user interface or a widget onto a waveguide of the display unit (150).

[0077] In the case of a virtual object whose display location information is set as a relative location with respect to the location of the augmented reality device (100), the processor (130) can display the virtual object at a location spaced apart according to a preset distance and direction based on the current location of the augmented reality device (100). The location of the augmented reality device (100) can be acquired through a technology such as SLAM (Simultaneous Localization and Mapping) using location information acquired by the location sensor (122) and measurement values ​​acquired by the IMU sensor (124).

[0078] The display unit (150) is configured to display a virtual object under the control of the processor (130). When the augmented reality device (100) is configured as augmented reality glasses, the display unit (150) may be configured as a lens optical system and may include a waveguide and an optical engine. The optical engine may be configured as a projector that generates light of a virtual object composed of a virtual image, an icon, or text, and projects the light onto a waveguide. The optical engine may include, for example, an image panel, an illumination optical system, a projection optical system, etc. In one embodiment of the present disclosure, the optical engine may be disposed on the frame or glasses temples of the augmented reality glasses. In one embodiment of the present disclosure, the optical engine may display a virtual object by projecting the virtual object onto a waveguide under the control of the processor (130).

[0079] However, it is not limited thereto, and the display unit (150) may be configured as at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display.

[0080] FIG. 4 is a flowchart illustrating a method for identifying matching positional relationship information based on a scene graph of a real space by an augmented reality device (100) according to one embodiment of the present disclosure.

[0081] Step S410 of FIG. 4 is a step that embodies the operation of step S220 illustrated in FIG. 2. Step S410 may be performed after the operation of step S210 illustrated in FIG. 2 is performed. Steps S420 and S430 of FIG. 4 are steps that embody the operation of step S230 illustrated in FIG. 2. After step S430 of FIG. 4 is performed, step S240 illustrated in FIG. 2 may be performed.

[0082] In step S410, the augmented reality device (100) obtains a scene graph indicating the types of a plurality of objects and the relative positional relationships between the plurality of objects. In one embodiment of the present disclosure, the augmented reality device (100) inputs an image acquired through a camera into an object recognition model implemented as a deep neural network model, and performs inference using the object recognition model to obtain classification information of the object from the image. In one embodiment of the present disclosure, the augmented reality device (100) can obtain information about the types of the plurality of objects as a result of inference through the object recognition model. The augmented reality device (100) can obtain three-dimensional positional coordinate value information of the objects in a stereo manner, and obtain a relative positional relationship regarding distance and direction in a three-dimensional space between the plurality of objects based on the obtained three-dimensional positional coordinate values ​​of the objects. The augmented reality device (100) can obtain a scene graph including nodes indicating the types of the plurality of objects and edges indicating the relative positional relationships between the plurality of objects.

[0083] In one embodiment of the present disclosure, a scene graph may be implemented in a format such as Extensible Markup Language (XML). Nodes of the scene graph may include semantic labels indicating the type or category of a recognized object, and edges may indicate a relative positional relationship between a reference object and another object, including at least one of a distance, a direction vector, or a sine vector in three-dimensional space. In one embodiment of the present disclosure, edges may include information regarding not only the relative positional relationship between objects, but also a dependency relationship between objects (e.g., a monitor and a keyboard).

[0084] The scene graph will be described with reference to Figures 5a to 5c.

[0085] FIG. 5a is a diagram illustrating a scene graph (500a) obtained by an augmented reality device (100) according to one embodiment of the present disclosure based on the distance between objects.

[0086] Referring to FIG. 5A, a scene graph (500A) may include a plurality of nodes (n1 to n5) and a plurality of edges. The plurality of nodes (n1 to n5) may include object attribute information regarding a category representing a type or classification result of a plurality of objects recognized from an image. In one embodiment of the present disclosure, the plurality of nodes (n1 to n5) may further include information regarding the name, type, or attribute of the plurality of recognized objects. In the embodiment illustrated in FIG. 5A, a first node (n1) may include information regarding a type or category of a first object in a real space, namely, 'monitor'. Similarly, a second node (n2) may include information regarding a type or category of a second object, namely, 'laptop', a third node (n3) may include information regarding a type or category of a third object, namely, 'keyboard', a type or category of a third object, namely, 'mouse', a type or category of a fourth object, and namely, a fifth node (n5) may include information regarding a type or category of a fifth object, namely, 'chair'.

[0087] An edge may include information about a distance in three-dimensional space between a plurality of nodes (n1 to n5). In the embodiment illustrated in FIG. 5A, a first edge may include information about a first distance (dx1, dy1, dz1) in three-dimensional space between a first node (n1) and a second node (n2). The first distance (dx1, dy1, dz1) may include information about a distance in three-dimensional space calculated based on three-dimensional position coordinate values ​​of a first object representing the first node (n1) and a second object representing the second node (n2). Similarly, the second edge may include information about a second distance (dx2, dy2, dz2) in a three-dimensional space between the first node (n1) and the third node (n3), the third edge may include information about a third distance (dx3, dy3, dz3) in a three-dimensional space between the first node (n1) and the fourth node (n4), the fourth edge may include information about a fourth distance (dx4, dy4, dz4) in a three-dimensional space between the first node (n1) and the fifth node (n5), and the fifth edge may include information about a fifth distance (dx5, dy5, dz5) in a three-dimensional space between the second node (n2) and the fifth node (n5).

[0088] FIG. 5b is a diagram illustrating a scene graph (500b) obtained by an augmented reality device (100) according to one embodiment of the present disclosure based on direction vectors of objects.

[0089] Referring to FIG. 5b, a scene graph (500b) may include a plurality of nodes (n1 to n5) and a plurality of edges. The plurality of nodes (n1 to n5) illustrated in FIG. 5b are the same as those illustrated in FIG. 5a, and thus, a duplicate description thereof will be omitted. The plurality of edges included in the scene graph (500b) may include information regarding a direction vector indicating a direction between nodes. In one embodiment of the present disclosure, the direction vector may have both a scalar value and a direction value, but may also have only information regarding the direction. In this case, the direction information included in the plurality of edges may be defined by the following formula.

[0090]

[0091] In the embodiment illustrated in FIG. 5B, the first edge may include information about a first direction vector (v1) indicating a direction between a first object representing a first node (n1) and a second object representing a second node (n2). Similarly, the second edge may include information about a second direction vector (v2) indicating a direction between the first node (n1) and a third node (n3), the third edge may include information about a third direction vector (v3) indicating a direction between the first node (n1) and a fourth node (n4), the fourth edge may include information about a fourth direction vector (v4) indicating a direction between the first node (n1) and a fifth node (n5), and the fifth edge may include information about a fifth direction vector (v5) indicating a direction between the second node (n2) and the fifth node (n5).

[0092] FIG. 5c is a diagram illustrating a scene graph (500c) obtained by an augmented reality device (100) according to one embodiment of the present disclosure based on sign vectors of objects.

[0093] Referring to Fig. 5c, the scene graph (500c) may include a plurality of nodes (n1 to n5) and a plurality of edges. The plurality of nodes (n1 to n5) illustrated in Fig. 5c are the same as those illustrated in Fig. 5a, and thus, a duplicate description will be omitted. The plurality of edges included in the scene graph (500c) may include information regarding a sine vector indicating a direction between nodes. The sine vector may be defined in a positive (+) or negative (-) direction with respect to a reference node, as in the following equation.

[0094]

[0095] In the embodiment illustrated in Fig. 5c, the first edge may include a first sine vector indicating a plus (+) direction, which is the direction of the second node (n2) with respect to the first object representing the first node (n1). The plus (+) direction and the minus (-) direction may be defined as directions with respect to a reference node. If the counterpart node is positioned to the right of the reference node, it may be defined as plus (+), and if the counterpart node is positioned to the left of the reference node, it may be defined as minus (-). Similarly, the second edge may include a second sine vector indicating a plus (+) direction of the third node (n3) with respect to the first node (n1), the third edge may include a third sine vector indicating a plus (+) direction of the fourth node (n4) with respect to the first node (n1), the fourth edge may include a fourth sine vector indicating a minus (-) direction of the fifth node (n5) with respect to the first node (n1), and the fifth edge may include a fifth sine vector indicating a plus (+) direction of the fifth node (n5) with respect to the second node (n2). In the scene graph (500c) illustrated in FIG. 5c, the sine vector is determined according to the reference node, and if the reference node changes, the direction (+ direction / - direction) of the sine vector may change in the opposite direction.

[0096] Referring back to FIG. 4, in step S420, the augmented reality device (100) measures the similarity by comparing the acquired scene graph with a scene graph of a previously stored relative positional relationship. In step S430, the augmented reality device (100) identifies positional relationship information matching the relative positional relationship between a plurality of objects based on the result of comparing the measured similarity with a preset threshold. Steps S420 and S430 will be described with reference to FIG. 6.

[0097] FIG. 6 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to measure the similarity of a scene graph (600) and identify positional relationship information matching the relative positional relationship between a plurality of objects in a real space.

[0098] Referring to FIG. 6, the processor (130, see FIG. 3) of the augmented reality device (100) can measure similarity by comparing a scene graph (600) obtained based on the relative positional relationship between a plurality of objects in a real space with a plurality of spatial context presets (610-1, 610-2, 610-3, ...) previously stored in a spatial context database (146). In one embodiment of the present disclosure, the similarity can be calculated using the following mathematical equation.

[0099]

[0100] Referring to the above mathematical expression 3, the similarity is the similarity between multiple objects (obj) included in the scene graph (600). i , i=1, 2, 3, ...) and objects (obj) included in the scene graph of each of multiple spatial context presets (601-1, 610-2, 610-3, ...) j , j=1, 2, 3, ...) and multiple edges (Ei) of the scene graph (600) compared m, m=1, 2, 3, ...) and edges (Ej) included in the scene graph of each of multiple spatial context presets (601-1, 610-2, 610-3, ...). n , n=1, 2, 3, ...) can be calculated based on the distance value between the objects. The similarity between objects can be calculated as the similarity of node properties including at least one of the type, category, or number of nodes of the scene graph representing the objects. The similarity between edges can be calculated as the distance difference between edges, but is not limited thereto. In one embodiment of the present disclosure, the similarity between edges is measured as the directional similarity between edges, and the directional similarity can be calculated as the cosine similarity between edges.

[0101] The processor (130) can compare the calculated similarity with a preset threshold, and identify a spatial context preset among a plurality of spatial context presets (610-1, 610-2, 610-3, ...) whose calculated similarity exceeds the threshold as a spatial context preset that matches the spatial context recognized from the image.

[0102] FIG. 7 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to measure the similarity of sub-graphs (721, 722, 723) of a scene graph (700) to identify a spatial context preset that matches a spatial context of a real space.

[0103] Referring to FIG. 7, a scene graph (700) may include multiple scene graphs having a hierarchical structure. The scene graph (700) may include a main graph (710) and at least one subgraph (721, 722, 723) that is a lower layer of the main graph (710). In the embodiment illustrated in FIG. 7, at least one subgraph (721, 722, 723) may be composed of three subgraphs. However, the present invention is not limited thereto.

[0104] The processor (130, see FIG. 3) of the augmented reality device (100) can measure similarity by comparing the main graph (710) or at least one subgraph (721, 722, 723) with the scene graph (730) of the spatial context preset stored in the spatial context database (146). A specific embodiment of how the processor (130) measures the similarity of the scene graph is the same as that described in FIG. 6, and therefore, a redundant description is omitted.

[0105] The processor (130) can identify a spatial context preset having a scene graph whose measured similarity exceeds a preset threshold. In the embodiment illustrated in FIG. 7, the processor (130) can identify a second subgraph (722) among at least one subgraph (721, 722, 723) whose calculated similarity between the scene graph (730) of the spatial context preset exceeds the threshold, and can identify a spatial context preset that matches the entire scene graph (700) based on the identification result.

[0106] In the embodiment illustrated in FIG. 7, the augmented reality device (100) may determine a spatial context preset as matching the entire scene graph (700) when the similarity with the second sub-graph (722), which is a lower region of the entire scene graph (700), exceeds a preset threshold in the similarity judgment of the scene graph. Through this, the augmented reality device (100) according to one embodiment of the present disclosure can identify a matching spatial context preset only with the similarity of a sub-graph representing the relative positional relationship of some objects, rather than a spatial context according to the relative positional relationship of all objects in a real space, thereby increasing the recognizability of the spatial context and providing the user with a user experience regarding work continuity in various work environments.

[0107] FIG. 8 is a flowchart illustrating an operation performed by an augmented reality device (100) according to one embodiment of the present disclosure depending on whether positional relationship information matching the relative positional relationship between a plurality of objects in a real space is identified.

[0108] Step S810 may be performed after the operation according to step S220 illustrated in FIG. 2 is performed. In step S810, the augmented reality device (100) measures the similarity by comparing the acquired scene graph with a scene graph regarding a previously stored relative position relationship. A specific embodiment of how the augmented reality device (100) measures the similarity of the scene graph has been described in detail in FIGS. 6 and 7, and thus, a redundant description will be omitted.

[0109] In step S820, the augmented reality device (100) determines whether positional relationship information matching the scene graph is identified based on the measured similarity. In one embodiment of the present disclosure, the augmented reality device (100) may calculate a similarity between a scene graph of a spatial context and a scene graph of a spatial context preset, and determine whether a spatial context preset matching the spatial context is identified based on the calculated similarity. In one embodiment of the present disclosure, the augmented reality device (100) may determine whether a matching spatial context preset is identified based on whether the calculated similarity exceeds a preset threshold.

[0110] As a result of the determination, if positional relationship information matching the scene graph is identified (step S830), the augmented reality device (100) displays a virtual object obtained from the identified positional relationship information at a preset location. In one embodiment of the present disclosure, if a spatial context preset matching the scene graph of the spatial context is identified, the augmented reality device (100) can display a virtual object obtained from the identified spatial context preset at a preset location. Step S830 is the same as step S240 illustrated in FIG. 2, and thus, redundant descriptions are omitted.

[0111] As a result of the determination, if positional relationship information matching the scene graph is not identified (step S840), the augmented reality device (100) generates new positional relationship information based on the relative positional relationship between the plurality of objects. In one embodiment of the present disclosure, if a spatial context preset matching the scene graph of the spatial context is not identified, the augmented reality device (100) may generate a new spatial context preset indicating the relative positional relationship between the plurality of objects. In this case, the augmented reality device (100) may generate a new scene graph including nodes indicating types or categories of the plurality of objects and edges indicating information on distances, direction vectors, or sine vectors between the plurality of objects.

[0112] In step S850, the augmented reality device (100) stores new location information. The augmented reality device (100) may store the new spatial context preset in the memory (140). In one embodiment of the present disclosure, the augmented reality device (100) may store the new scene graph as a new spatial context preset in the spatial context database (146, see FIG. 3).

[0113] FIG. 9 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to merge a new spatial context with an existing spatial context preset.

[0114] Referring to FIG. 9, the augmented reality device (100) recognizes a new spatial context indicating the relative positional relationship of a plurality of objects (91, 92, 93, 94) in a real space (90) (Operation ①). The augmented reality device (100) includes a left-eye camera (110L) and a right-eye camera (110R), and can capture images of a plurality of objects (91, 92, 93, 94) in the real space (90) using the left-eye camera (110L) and the right-eye camera (110R). The augmented reality device (100) can perform vision recognition using an artificial intelligence model to recognize a plurality of objects (91, 92, 93, 94) in the real space (90) from the images. In one embodiment of the present disclosure, the augmented reality device (100) can obtain three-dimensional position coordinate values ​​of objects including depth values ​​(z-axis coordinate information) of a plurality of objects (91, 92, 93, 94) through a stereo vision method using a left-eye image obtained from a left-eye camera (110L) and a right-eye image obtained from a right-eye camera (110R). The augmented reality device (100) can obtain a relative positional relationship regarding a three-dimensional distance and direction between the plurality of objects (91, 92, 93, 94) based on the obtained three-dimensional positional coordinate values. In one embodiment of the present disclosure, the relative positional relationship can include at least one of information on a distance, a direction vector, and a sign vector in a three-dimensional space of the plurality of objects (91, 92, 93, 94).

[0115] The augmented reality device (100) stores a new spatial context preset (operation ②). In one embodiment of the present disclosure, the processor (130, see FIG. 3) of the augmented reality device (100) may obtain a scene graph (900) based on classification information or type information of a plurality of objects (91, 92, 93, 94) and relative positional relationships between the plurality of objects (91, 92, 93, 94). The scene graph (900) may include a plurality of nodes representing classification information or types of each of the plurality of objects (91, 92, 93, 94) and a plurality of edges representing relative positional relationships between the plurality of nodes. The processor (130) may store the obtained scene graph (900) as a new spatial context preset in a spatial context database (146, see FIG. 3).

[0116] The augmented reality device (100) receives a user input for merging a new spatial context preset with a previously stored spatial context preset (Operation ③). In one embodiment of the present disclosure, the augmented reality device (100) may display a first graphical user interface (GUI) (910) for receiving a user input for determining whether to merge the new spatial context preset with an existing spatial context preset. In the embodiment illustrated in FIG. 9, the first graphical user interface (910) may be configured with text for receiving a user input for determining whether to merge, such as “Do you want to merge the new spatial context with the existing spatial context?” The augmented reality device (100) may receive a user’s hand pointing input regarding whether to merge the spatial context preset. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the augmented reality device (100) may also receive an input based on a user’s gaze direction with respect to the first graphical user interface (910). In this case, the augmented reality device (100) may track the gaze direction of the user's two eyes and determine whether to merge the spatial context based on the position information of the gaze point where the gaze direction converges between "yes" and "no" of the first graphical user interface (910).

[0117] The augmented reality device (100) receives a user input for selecting a context preset to be merged with a new spatial context preset from among a plurality of previously stored context presets (operation ④). In one embodiment of the present disclosure, the augmented reality device (100) may display a second graphical user interface (GUI) (920) for receiving a user input for selecting any one of a plurality of spatial context presets previously stored in a spatial context database (146, see FIG. 3). In the embodiment illustrated in FIG. 9, the second graphical user interface (920) may include text for receiving a user input for determining a spatial context preset to be merged with a new spatial context preset, such as “Select a spatial context to merge.” In one embodiment of the present disclosure, the second graphical user interface (920) may include a graphical object representing a plurality of scene graphs (921, 922, 923) relating to the plurality of spatial context presets.

[0118] The augmented reality device (100) may receive a user's hand pointing input for selecting a spatial context preset to be merged with a new spatial context preset through a second graphical user interface (920). In the embodiment illustrated in FIG. 9, the augmented reality device (100) may receive a user's hand pointing input for selecting a third scene graph (923) of a third spatial context preset to be merged with a new spatial context preset from among a plurality of scene graphs (921, 922, 923) for a plurality of spatial context presets included in the second graphical user interface (920). The augmented reality device (100) may select the third spatial context preset corresponding to the third scene graph (923) from among the plurality of spatial context presets based on the received hand pointing input. However, it is not limited thereto, and in one embodiment of the present disclosure, the augmented reality device (100) may track the gaze direction of the user's two eyes to obtain position information of a gaze point where the gaze direction converges among a plurality of scene graphs (921, 922, 923) for a plurality of spatial context presets, and may select a third spatial context preset as a spatial context preset to be merged with a new spatial context preset based on the position information of the gaze point.

[0119] The augmented reality device (100) merges a spatial context selected by a user input with a new spatial context (operation ⑤). In one embodiment of the present disclosure, the processor (130) of the augmented reality device (100) may merge a scene graph (900) of a new spatial context preset with a scene graph (923) of a third spatial context preset selected by a user input from among a plurality of spatial context presets previously stored in a spatial context database (146). In one embodiment of the present disclosure, the merging between the scene graphs may be, but is not limited to, a parallel merging between the scene graph (900) of the new spatial context preset and the third scene graph (923) of the previously stored third spatial context preset. If the relationship between the scene graph (900) of the new spatial context preset and the third scene graph (923) of the existing spatial context preset is a dependent relationship, the merging between the scene graphs may form a hierarchical structure according to the dependent relationship.

[0120] FIG. 10 is a flowchart illustrating a method for an augmented reality device (100) according to one embodiment of the present disclosure to display a virtual object matching a spatial context.

[0121] Steps S1010 to S1040 of FIG. 10 are steps that specify the operation of step S240 illustrated in FIG. 2. Step S1010 may be performed after the operation of step S230 illustrated in FIG. 2 is performed.

[0122] In step S1010, the augmented reality device (100) obtains information on a virtual object from the identified location relationship information. In the present disclosure, a 'virtual object' refers to a virtual graphic object composed of text, an image, or a combination thereof that is related to an object in a real space or represents information provided by an application executed by the augmented reality device (100). In one embodiment of the present disclosure, the virtual object may be displayed in the form of a graphical user interface (GUI) or a widget.

[0123] The augmented reality device (100) can obtain display position information of a virtual object from a spatial context preset. In one embodiment of the present disclosure, the 'display position information' may be set to a relative position based on the positions of objects in real space or may be set to a relative position based on the position of the augmented reality device (100). When the display position information of the virtual object is set to a relative position based on the positions of objects in real space, the display position information may include information about the type or category of the object on which the virtual object is to be displayed, and setting information about the direction and distance from the object. For example, the display position information may include information that the first virtual object is located 10 cm away from the right side of the monitor. When the display position information of the virtual object is set to a relative position based on the position of the augmented reality device (100), the display position information may include information about the distance and direction away from the position of the augmented reality device (100). For example, the display position information may include three-dimensional position coordinate value information (dx, dy, dz) of the upper left side of the lens part of the augmented reality device (100).

[0124] In step S1020, the augmented reality device (100) determines whether an object included in the location relationship information is identified among a plurality of objects in a real space recognized from an image. In one embodiment of the present disclosure, the augmented reality device (100) can recognize a plurality of objects in a real space from an image captured by a camera, and identify an object included in a spatial context preset among the plurality of recognized objects.

[0125] When an object included in the location relationship information from the image is identified (step S1030), i.e., when an object included in the spatial context preset is identified, the augmented reality device (100) displays a virtual object at a display location for the object in the identified real space.

[0126] If an object included in the location relationship information from the image is not identified (step S1040), i.e., if an object included in the spatial context preset is not identified, the augmented reality device (100) displays a virtual object at a preset location based on the location of the device. Steps S1030 and S1040 will be described in detail with reference to FIG. 11.

[0127] FIG. 11 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to display a virtual object (1110, 1120, 1130) matching a spatial context.

[0128] Referring to FIG. 11, the augmented reality device (100) can obtain information about virtual objects (1110, 1120, 1130) from a spatial context preset. Based on the obtained information, the augmented reality device (100) can recognize that the display positions of the first virtual object (1110), which is a To-do list widget indicating a task, and the second virtual object (1120), which is a graphical user interface indicating a task progress, are set to relative positions based on the objects. For example, the display position of the first virtual object (1110) can be set to a position spaced apart by a preset distance in the right direction of the monitor, and the display position of the second virtual object (1120) can be set to a position spaced apart by a preset distance in the upper direction of the monitor. Based on the obtained information, the augmented reality device (100) can recognize that the display position of the third virtual object (1130), which is a calendar widget, is set to a relative position based on the position of the augmented reality device (100). For example, the display position information of the third virtual object (1130) may be set based on the position of the augmented reality device (100) so that it can be displayed at the upper left position of the lens part of the augmented reality device (100). In this case, the position information of the augmented reality device (100) may be obtained through a technology such as SLAM (Simultaneous Localization and Mapping) using three-dimensional position coordinate information obtained by a position sensor (122, see FIG. 3) such as a GPS sensor and a measurement value obtained by an IMU sensor (124, see FIG. 3).

[0129] The augmented reality device (100) can recognize a plurality of objects in a real space (10) from an image captured by a camera, and identify an object that matches an object included in a spatial context preset among the recognized objects. In the embodiment illustrated in FIG. 11, the processor (130, see FIG. 3) of the augmented reality device (100) can recognize a first object (e.g., a 'monitor') that matches a 'monitor' included in a spatial context preset from an image captured by a camera. The processor (130) can display a first virtual object (1110) and a second virtual object (1120), which are virtual objects whose display position information is set based on the positions of the objects, at preset positions. For example, the processor (130) can display the first virtual object (1110) at a position spaced apart by a preset position in the right direction of the first object (11), which is a monitor in the real space (10), and can display the second virtual object (1120) at a position spaced apart by a preset position in the upper direction of the first object (11).

[0130] If the augmented reality device (100) does not match an object in the real space (10), the third virtual object (1130) can be displayed at a display position set based on the position of the augmented reality device (100). For example, the processor (130) of the augmented reality device (100) can display the third virtual object (1130) at the upper left of the lens unit based on the position of the augmented reality device (100).

[0131] In one embodiment of the present disclosure, the display unit includes a waveguide and an optical engine, and the processor (130) controls the optical engine to project light constituting the virtual object (1110, 1120, 1130) onto the waveguide, thereby displaying the virtual object (1110, 1120, 1130).

[0132] FIG. 12 is a flowchart illustrating a method for an augmented reality device (100) to display a new virtual object according to one embodiment of the present disclosure.

[0133] FIG. 13 is a drawing for explaining an operation of an augmented reality device (100) displaying a new virtual object (1300) according to one embodiment of the present disclosure.

[0134] Hereinafter, with reference to FIGS. 12 and 13, the operation of the augmented reality device (100) creating and displaying a new virtual object (1300) will be described.

[0135] In step S1210 of FIG. 12, the augmented reality device (100) generates new virtual object information. The new virtual object may be, but is not limited to, a graphical user interface (GUI) or a widget. In the embodiment illustrated in FIG. 13, the new virtual object (1300) may be a To-do list widget indicating a list of tasks.

[0136] In step S1220 of FIG. 12, the augmented reality device (100) receives a user input for setting a display position of a new virtual object. The display position of the new virtual object may be set based on the position of the object in real space or based on the position of the augmented reality device (100).

[0137] When a user input for setting a display position based on a real object is received (step S1230), the augmented reality device (100) selects a target object existing on a scene graph. In one embodiment of the present disclosure, the augmented reality device (100) may recognize a spatial context representing a relative positional relationship between objects in a real space, obtain a scene graph representing the recognized spatial context, and select a target object for displaying a virtual object among objects included in a node of the scene graph. Referring also to FIG. 13, when the display position of a virtual object is set based on an object position (②-1), the augmented reality device (100) may recognize a plurality of objects (11, 12, 13, 14, 15) in a real space (10), and receive a user input for selecting a target object on which a new virtual object (1300) is to be displayed among the plurality of recognized objects (11, 12, 13, 14, 15). In one embodiment of the present disclosure, the augmented reality device (100) can receive a user's hand pointing input for selecting a target object among a plurality of objects (11, 12, 13, 14, 15). In the embodiment illustrated in FIG. 13, the augmented reality device (100) can receive a user's hand pointing input for selecting a first object (11), which is a monitor, among the plurality of objects (11, 12, 13, 14, 15).

[0138] In step S1240 of FIG. 12, the augmented reality device (100) displays a new virtual object by placing it at a relative position with respect to the target object based on a user input. In one embodiment of the present disclosure, the augmented reality device (100) may receive a user input for setting a distance and direction in which the new virtual object is to be displayed based on the position of the target object. Referring also to FIG. 13, the augmented reality device (100) may set a display position of a new virtual object (1300) based on a user's hand pointing input (③-1). In the embodiment illustrated in FIG. 13, the augmented reality device (100) may receive a user's hand pointing input for selecting and dragging a new virtual object (1300), and display the new virtual object (1300) at a position moved by the dragged distance along the right direction of the first object (11) based on the received hand pointing input.

[0139] When receiving a user input for setting a display position based on the position of the augmented reality device (100) (step S1250), the augmented reality device (100) displays a new virtual object by placing it at a relative position with respect to the position of the device based on the user input. Referring to FIG. 13 together, the augmented reality device (100) can set the display position of the new virtual object (1300) at a relative position based on the position of the augmented reality device (100) (②-2). For example, the augmented reality device (100) can set the display position of the new virtual object (1300) so that the new virtual object (1300) is displayed at the lower left of the lens of the augmented reality device (100). The augmented reality device (100) can set the display position of the new virtual object (1300) based on a user's hand pointing input (③-2). In the embodiment illustrated in FIG. 13, the augmented reality device (100) may select a new virtual object (1300), receive a user's hand pointing input for dragging in the right direction, and display the new virtual object (1300) by moving the display position to the central area of ​​the lower part of the lens based on the received hand pointing input.

[0140] One aspect of the present disclosure provides a method for providing an augmented reality service in which an augmented reality device (100) matches a real space. The method for operating the augmented reality device (100) may include a step (S210) of recognizing a plurality of objects in a real space from an image acquired through a camera (110). The method for operating the augmented reality device (100) may include a step (S220) of recognizing a relative positional relationship including a position and a direction between the recognized plurality of objects. The method for operating the augmented reality device (100) may include a step (S230) of displaying a virtual object representing information related to the recognized plurality of objects at a preset position based on information regarding the previously stored relative positional relationship.

[0141] In one embodiment of the present disclosure, the step (S220) of recognizing the relative positional relationship of the plurality of objects may include the step of obtaining three-dimensional positional coordinate information of each of the plurality of objects from an image; and the step of obtaining the relative positional relationship between the plurality of objects based on the three-dimensional positional coordinate information of the plurality of objects.

[0142] In one embodiment of the present disclosure, the relative positional relationship may include at least one of information of a distance, a direction vector, and a sign vector in a three-dimensional space between a plurality of objects.

[0143] In one embodiment of the present disclosure, the step of recognizing the relative positional relationship of the plurality of objects (S220) may include a step of obtaining a scene graph indicating the types of the plurality of recognized objects and the relative positional relationship between the plurality of objects (S410). The step of recognizing the relative positional relationship of the plurality of objects (S220) may include a step of measuring a similarity by comparing the obtained scene graph with a previously stored scene graph (S420); and a step of identifying positional relationship information matching the relative positional relationship based on a result of comparing the measured similarity with a preset threshold (S430).

[0144] In one embodiment of the present disclosure, in the step (S420) of measuring the similarity of the scene graph, the augmented reality device (100) can calculate the similarity by comparing node attribute information including at least one of classification information, type, and number of a plurality of objects included in the scene graph with node attribute information included in pre-stored location relationship information.

[0145] In one embodiment of the present disclosure, the acquired scene graph may include at least one sub-graph having a hierarchical structure. The step (S430) of identifying the relative positional relationship information may include the step of measuring similarity by comparing each of the at least one sub-graph with the scene graph of the previously stored positional relationship information; and the step of identifying positional relationship information having a scene graph whose measured similarity exceeds a preset threshold.

[0146] In one embodiment of the present disclosure, the method of operating the augmented reality device (100) may further include a step (S840) of generating new positional relationship information based on information regarding the relative positional relationship between the plurality of objects when positional relationship information matching the relative positional relationship between the plurality of recognized objects is not identified. The method of operating the augmented reality device (100) may further include a step (S850) of storing the generated new positional relationship information.

[0147] In one embodiment of the present disclosure, the method of operating the augmented reality device (100) may further include a step of merging the generated new location relationship information with previously stored location relationship information.

[0148] In one embodiment of the present disclosure, the step of displaying the virtual object (S230) may include the step of obtaining display position information of the virtual object from position relationship information matching the relative position relationship between a plurality of objects (S1010); the step of identifying an object matching a plurality of objects included in the position relationship information among a plurality of objects in a real space recognized from an image (S1020); and the step of displaying the virtual object at a display position for the object based on the obtained display position information.

[0149] In one embodiment of the present disclosure, in the step of displaying the virtual object (S240), the augmented reality device (100) can display the virtual object according to a preset distance and direction based on the position of the augmented reality device (100).

[0150] Another aspect of the present disclosure provides an augmented reality device (100) that provides an augmented reality service matching a real space. The augmented reality device (100) of the present disclosure may include a camera (110) that captures an image of a real space, a memory (140) that stores at least one instruction, at least one processor (130) that executes the at least one instruction, and a display unit (150). The at least one processor (130) may recognize a plurality of objects in a real space from an image acquired through the camera (110). The at least one processor (130) may recognize a relative positional relationship including a position and a direction between the recognized plurality of objects, and may control the display unit (150) to display a virtual object representing information related to the recognized plurality of objects at a preset position based on information about the relative positional relationship previously stored in the memory (140).

[0151] In one embodiment of the present disclosure, the at least one processor (130) may, by executing the at least one command, obtain a relative positional relationship between a plurality of objects based on three-dimensional positional coordinate information of each of the plurality of objects obtained from an image. The relative positional relationship may include at least one of a distance, a direction vector, and a sign vector in a three-dimensional space between the plurality of objects.

[0152] In one embodiment of the present disclosure, the at least one processor (130) may obtain a scene graph indicating the types of a plurality of recognized objects and the relative positional relationships between the plurality of objects by executing the at least one command. The at least one processor (130) may measure a similarity by comparing the obtained scene graph with a scene graph of positional relationship information previously stored in a memory (140), and may identify the positional relationship information matching the relative positional relationship between the plurality of objects based on a result of comparing the measured similarity with a preset threshold.

[0153] In one embodiment of the present disclosure, the at least one processor (130) may calculate a similarity by comparing node attribute information including at least one of classification information, type, and number of a plurality of objects included in a scene graph with node attribute information included in pre-stored location relationship information by executing the at least one command.

[0154] In one embodiment of the present disclosure, the acquired scene graph may include at least one sub-graph having a hierarchical structure. The at least one processor (130) may, by executing the at least one command, measure similarity by comparing the at least one sub-graph with the scene graph of previously stored location relationship information. The at least one processor (130) may identify location relationship information having a scene graph whose measured similarity exceeds a preset threshold.

[0155] In one embodiment of the present disclosure, the at least one processor (130) may, by executing the at least one command, generate new positional relationship information based on information regarding the relative positional relationship between the plurality of objects when positional relationship information matching the relative positional relationship between the plurality of recognized objects is not identified. The at least one processor (130) may store the generated new positional relationship information in the memory (140).

[0156] In one embodiment of the present disclosure, the at least one processor (130) may merge the generated new location relationship information with the previously stored location relationship information by executing the at least one command.

[0157] In one embodiment of the present disclosure, the at least one processor (130) may, by executing the at least one command, obtain display position information of a virtual object from position relationship information matching a relative positional relationship between a plurality of objects, and identify an object that matches a plurality of objects included in the position relationship information among a plurality of objects in a real space recognized from an image. The at least one processor (130) may control the display unit (150) to display a virtual object at a display position for the identified object based on the obtained display position information.

[0158] In one embodiment of the present disclosure, the at least one processor (130) may control the display unit (150) to display a virtual object at a location spaced apart from the location of the augmented reality device (100) according to a preset distance and direction by executing the at least one command.

[0159] Another aspect of the present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an augmented reality device (100), such that the augmented reality device (100) performs the following operations: recognizing a plurality of objects in a real space from an image acquired through a camera (110); recognizing a relative positional relationship including positions and directions between the recognized plurality of objects; and displaying a virtual object representing information related to the recognized plurality of objects at a preset position based on information about the previously stored relative positional relationship.

[0160] The program executed by the augmented reality device (100) described in the present disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0161] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.

[0162] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The media may be readable by a computer, stored in a memory, and executed by a processor.

[0163] A computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium does not contain signals and is tangible, but does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0164] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0165] A computer program product may include a software program, a computer-readable storage medium having the software program stored thereon. For example, the computer program product may be provided by a manufacturer of an augmented reality device (100) or an electronic market (e.g., Samsung Galaxy Store). TM) may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically. For electronic distribution, at least a portion of the software program may be stored in a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server of the augmented reality device (100), a server of an electronic market, or a relay server that temporarily stores the software program.

[0166] The computer program product may include a storage medium of the server or the storage medium of the augmented reality device (100) in a system comprising an augmented reality device (100) and / or a server. Alternatively, if there is a third device (e.g., a mobile device) that is communicatively connected to the augmented reality device (100), the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the augmented reality device (100) to the third device, or from the third device to an electronic device.

[0167] In this case, either the augmented reality device (100) or the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, at least one of the augmented reality device (100) and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0168] For example, the augmented reality device (100) may execute a computer program product stored in a memory (140, see FIG. 3) to control another electronic device that is in communication with the augmented reality device (100) to perform a method according to the disclosed embodiments.

[0169] As another example, a third device may execute a computer program product to control an electronic device in communication with the third device to perform a method according to the disclosed embodiment.

[0170] When a third device executes a computer program product, the third device may download the computer program product from the augmented reality device (100) and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the disclosed embodiments.

[0171] Although the embodiments described above have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

1. In a method for providing an augmented reality service that matches an augmented reality device (100) to a real space, Step (S210) of recognizing multiple objects in real space from images acquired through a camera (110); A step (S220) of recognizing a relative positional relationship including positions and directions between the above-mentioned recognized plurality of objects; and A step (S230) of displaying a virtual object representing information related to the recognized plurality of objects at a preset location based on information about a previously stored relative positional relationship; A method comprising:

2. In paragraph 1, The step (S220) of recognizing the relative positional relationship of the above multiple objects is A step of obtaining three-dimensional position coordinate information of each of the plurality of objects from the image; and A step of obtaining a relative positional relationship between the plurality of objects based on three-dimensional positional coordinate information of the plurality of objects; A method comprising:

3. In either of paragraphs 1 or 2, The step (S220) of recognizing the relative positional relationship of the above multiple objects is A step (S410) of obtaining a scene graph representing the types of the plurality of recognized objects and the relative positional relationship between the plurality of objects; A step (S420) of measuring similarity by comparing the acquired scene graph with the previously stored scene graph; and A step (S430) of identifying positional relationship information matching the relative positional relationship based on the result of comparing the measured similarity with a preset threshold; A method comprising:

4. In paragraph 3, The step (S420) of measuring the similarity of the above scene graph is A method for calculating a similarity by comparing node attribute information including at least one of classification information, types, and numbers of the plurality of objects included in the scene graph with node attribute information included in the previously stored location relationship information.

5. In paragraph 3, The above acquired scene graph includes at least one sub-graph having a hierarchical structure, The step (S430) of identifying the above relative position relationship information is: A step of measuring similarity by comparing at least one of the subgraphs with the scene graph of the previously stored location relationship information; and A step of identifying positional relationship information having a scene graph whose measured similarity exceeds a preset threshold; A method comprising:

6. In any one of clauses 1 to 5, If position relationship information matching the relative position relationship between the above-described plurality of objects is not identified, a step (S840) of generating new position relationship information based on information about the relative position relationship between the plurality of objects; and Step of storing the newly generated location relationship information (S850); A method further comprising:

7. In any one of clauses 1 to 6, The step of displaying the above virtual object (S230) is A step (S1010) of obtaining display location information of the virtual object from location relationship information matching the relative location relationship between the plurality of objects; Step (S1020) of identifying an object that matches a plurality of objects included in the positional relationship information among a plurality of objects in the real space recognized from the image; and A step of displaying the virtual object at the display location for the identified object based on the acquired display location information; A method comprising:

8. In an augmented reality device (100) that provides an augmented reality service matching the real space, A camera (110) that captures images by photographing real space; A memory (140) storing at least one instruction; At least one processor (130) for executing at least one of the above instructions; and Display section (150); Including, At least one processor (130) above, Recognize multiple objects in real space from images acquired through the above camera (110), Recognize the relative positional relationship including the position and direction between the above recognized multiple objects, An augmented reality device (100) that controls the display unit (150) to display a virtual object representing information related to the recognized plurality of objects at a preset location based on information about the relative positional relationship previously stored in the memory (140).

9. In paragraph 8, The at least one processor (130) executes the at least one instruction: Obtaining a relative positional relationship between the plurality of objects based on the three-dimensional positional coordinate information of each of the plurality of objects obtained from the image, The above relative positional relationship is, An augmented reality device (100) including at least one piece of information among a distance, a direction vector, and a sign vector in a three-dimensional space between the plurality of objects.

10. In either of paragraphs 8 or 9, The at least one processor (130) executes the at least one instruction: Obtain a scene graph representing the types of the above-described plurality of objects and the relative positional relationship between the above-described plurality of objects, The obtained scene graph is compared with the scene graph of the location relationship information previously stored in the memory (140) to measure the similarity, An augmented reality device (100) that identifies positional relationship information that matches the relative positional relationship between the plurality of objects based on the result of comparing the measured similarity with a preset threshold.

11. In clause 10, The at least one processor (130) executes the at least one instruction: An augmented reality device (100) that calculates a similarity by comparing node attribute information including at least one of classification information, types, and numbers of the plurality of objects included in the scene graph with node attribute information included in the previously stored location relationship information.

12. In paragraph 10, The above acquired scene graph includes at least one sub-graph having a hierarchical structure, The at least one processor (130) executes the at least one instruction: Measure the similarity by comparing at least one of the above subgraphs with the scene graph of the above stored location relationship information, An augmented reality device (100) that identifies location relationship information having a scene graph whose measured similarity exceeds a preset threshold.

13. In any one of paragraphs 8 to 12, The at least one processor (130) executes the at least one instruction: If position relationship information matching the relative position relationship between the above recognized plurality of objects is not identified, new position relationship information is generated based on the information about the relative position relationship between the above plurality of objects, An augmented reality device (100) that stores the newly generated location relationship information in the memory (140).

14. In any one of paragraphs 8 to 13, The at least one processor (130) executes the at least one instruction: Obtaining display position information of the virtual object from position relationship information matching the relative position relationship between the plurality of objects, and identifying an object that matches the plurality of objects included in the position relationship information among the plurality of objects in the real space recognized from the image, An augmented reality device (100) that controls the display unit (150) to display the virtual object at the display location for the identified object based on the acquired display location information.

15. In a computer program product including a computer-readable storage medium, The above storage medium, An operation of recognizing multiple objects in real space from images acquired through a camera (110); An operation of recognizing a relative positional relationship including positions and directions between the recognized plurality of objects; and An action of displaying a virtual object representing information related to a plurality of recognized objects at a preset location based on information about previously stored relative positional relationships; A computer program product including instructions regarding a method for providing an augmented reality service matching a real space by an augmented reality device (100).

Images