Method and device for displaying information using augmented reality image

The augmented reality device addresses the challenge of providing device information by using a camera and processor to identify and display relevant data from connected devices, ensuring efficient and user-friendly information delivery.

WO2024248546A1PCT designated stage expired Publication Date: 2024-12-05SAMSUNG ELECTRONICS CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/007499
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-08-30
Filing Date
2024-05-31
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing augmented reality technologies lack efficient methods to provide information about connected devices to users in a seamless and user-friendly manner, particularly when devices do not have their own displays or when the user cannot easily access the information due to display orientation or distance.

Method used

An augmented reality device equipped with a camera, processor, and display unit identifies devices of interest and determines whether to display information through an augmented reality image, considering factors like device identification, display presence, orientation, and distance, to provide notifications or information to the user.

Benefits of technology

Enables effective and timely information delivery about connected devices, reducing redundant information and improving user experience by selectively displaying relevant data in a visually intuitive manner.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024007499_05122024_PF_FP_ABST
    Figure KR2024007499_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Provided is a method by which an augmented reality device displays information about another device. An operating method of the augmented reality device may comprise a step for identifying a device of interest for displaying information. The operating method of the augmented reality device may comprise a step for determining whether to display information about the device of interest. The operating method of the augmented reality device may comprise a step for displaying an augmented reality image corresponding to the information about the device of interest.
Need to check novelty before this filing date? Find Prior Art

Description

Method and device for displaying information using augmented reality images

[0001] The present disclosure relates to an augmented reality device and a method of operating the same, and to a method and device for providing a user with information related to another device by displaying information as an augmented reality image.

[0002] Augmented Reality (AR) technology synthesizes virtual objects or information into a real-world environment, making them appear as if they were real objects in the physical world. Modern computing and display technologies have enabled the development of systems for AR experiences, in which digitally recreated images or portions of them are presented to the user in a way that makes them appear, or can be perceived, as, real.

[0003] Augmented reality (AR) technology allows virtual images to be overlaid onto the physical environment or objects of the real world. As interest in AR technology grows, various technologies for implementing AR are being developed. AR devices (e.g., smart glasses) utilizing AR technology are being used for useful purposes, such as information retrieval, route guidance, and camera recording.

[0004] Augmented reality devices are devices that can acquire data from the user's perspective, and in particular, smart glasses can display virtual images by overlaying them on images of the physical environment of the real world through a transparent display.

[0005] In an augmented reality environment where real-world objects and virtual objects coexist, various information about the real-world environment can be provided to users through augmented reality images.

[0006] One aspect of the present disclosure provides a method for an augmented reality device to display information about another device. The method of operating the augmented reality device may include identifying a device of interest for displaying information. The method of operating the augmented reality device may include determining whether to display information about the device of interest. The method of operating the augmented reality device may include displaying an augmented reality image including information about the device of interest.

[0007] One aspect of the present disclosure provides an augmented reality device that displays information about a connected device. The augmented reality device may include a camera configured to acquire a real scene image. The augmented reality device may include a display unit. The augmented reality device may include a memory that stores a program including at least one command. The augmented reality device may include at least one processor. The at least one processor may identify a device of interest for displaying information, determine whether to display information about the device of interest, and display an augmented reality image including information about the device of interest through the display unit.

[0008] One aspect of the present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may store instructions readable by an augmented reality device for executing at least one of the disclosed methods on the augmented reality device.

[0009] FIG. 1 is a drawing for explaining an operation of an augmented reality device according to one embodiment of the present disclosure to display information of a device of interest.

[0010] FIG. 2 is a flowchart of a method for displaying information by an augmented reality device according to one embodiment of the present disclosure.

[0011] FIG. 3 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to display an augmented reality image including information about a device of interest.

[0012] FIG. 4 is a diagram illustrating an operation of identifying a device of interest based on a user's selection according to one embodiment of the present disclosure.

[0013] FIG. 5 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to identify a device of interest and display an augmented reality image including information about the device of interest.

[0014] FIG. 6 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to display an augmented reality image including information about a plurality of devices of interest.

[0015] FIG. 7A is a diagram illustrating an example of an augmented reality device according to one embodiment of the present disclosure displaying an augmented reality image including information about a device of interest.

[0016] FIG. 7b is a diagram illustrating an example of an augmented reality device according to one embodiment of the present disclosure displaying an augmented reality image including information about a device of interest.

[0017] FIG. 7c is a diagram illustrating an example of an augmented reality device according to one embodiment of the present disclosure displaying an augmented reality image including information about a device of interest.

[0018] FIG. 7d is a diagram illustrating an example of an augmented reality device according to one embodiment of the present disclosure displaying an augmented reality image including information about a device of interest.

[0019] FIG. 8 is a block diagram of an augmented reality device according to one embodiment of the present disclosure.

[0020] FIG. 9 is a drawing illustrating a glass-type augmented reality device according to one embodiment of the present disclosure.

[0021] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, for the purpose of clearly explaining the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar parts are designated with similar reference numerals throughout the specification.

[0022] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0023] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0024] Throughout this disclosure, when a part is referred to as "including" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," and the like used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0025] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where they are "directly connected" but also the cases where they are "electrically connected" with other elements intervening. Furthermore, when a part is said to "include" a component, this does not exclude other components, but rather includes the possibility of including other components, unless otherwise specifically stated.

[0026] The expression “configured to” as used herein can be used interchangeably with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean something that is “specifically designed to” in terms of hardware. Instead, in some contexts, the expression “a system configured to” can mean that the system, together with other devices or components, is “capable of.” For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a generic-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in memory.

[0027] The functions related to 'Artificial Intelligence (AI)' according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an AI-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or AI models stored in memory. If one or more processors are AI-only processors, the AI-only processor may be designed with a hardware structure specialized for processing a specific AI model.

[0028] The predefined operation rules or artificial intelligence model are characterized by being created through learning. Here, being created through learning means that a basic artificial intelligence model (or deep learning model) is trained using a plurality of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0029] An "artificial intelligence model (or deep learning model)" may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the calculation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be modified so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), such as, but not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0030] In this disclosure, 'Augmented Reality' refers to displaying a virtual image together within a physical environment space of the real world or displaying a real object and a virtual image together.

[0031] In the present disclosure, an 'augmented reality device' is a device capable of expressing augmented reality (AR), and can display images including physical objects and virtual objects existing in reality. The augmented reality device may be, for example, not only augmented reality glasses in the shape of glasses worn by a user on the face, but also a head-mounted display device (HMD, Apparatus) worn on the head, or an augmented reality helmet.

[0032] The present disclosure will be described in detail with reference to the attached drawings below.

[0033] FIG. 1 is a drawing for explaining an operation of an augmented reality device (100) according to one embodiment of the present disclosure to display information of a device of interest (200).

[0034] The augmented reality device (100) is a device capable of displaying (or expressing) augmented reality, and may be configured, for example, as augmented reality glasses in the shape of glasses that a user (U) wears on the face. The components of the augmented reality device (100) will be described in more detail with reference to FIGS. 8 and 9, which will be described later.

[0035] In one embodiment, the augmented reality device (100) can identify a device of interest (200). The device of interest (200) can be a target object for information provision through the augmented reality device (100). Information related to the device of interest (200) can be displayed as an augmented reality image (VI) through the augmented reality device (100).

[0036] Among the devices connected to the augmented reality device (100), a device requiring notification may be a device of interest (200). For example, the device of interest (200) may include at least one of a device connected to the augmented reality device (100) in which an event has occurred, a device selected by a user (U) among the devices connected to the augmented reality device (100), or a device newly connected to the augmented reality device (100).

[0037] At least one device (e.g., an electronic device) may be connected or paired with the augmented reality device (100) via short-range wireless communication. When an event requiring notification to a user (U) occurs in a specific device connected to the augmented reality device (100), the device may display the notification on its own through its display. In one embodiment, the at least one device may not include a display for displaying the notification on its own. In this case, the augmented reality device (100) may be utilized to provide the user (U) with a notification related to the device.

[0038] In one embodiment, when a device that was not connected to the augmented reality device (100) is newly connected to the augmented reality device (100), the newly connected device may be a device of interest (200) that requires notification through the augmented reality device (100). When a device that was not connected to the augmented reality device (100) is newly paired with the augmented reality device (100), it may be necessary to notify the user (U) of the augmented reality device (100) of this pairing fact. In this case, the augmented reality device (100) may display an augmented reality image (VI) to notify the user (U) of the pairing fact of the new device.

[0039] In one embodiment, the device of interest (200) may be a device selected by the user (U) from among devices connected to the augmented reality device (100). The user (U) may select a device of which he or she wants to know related information as the device of interest (200). The user (U) may select the device of interest (200) through a voice signal or may select the device of interest (200) by gazing at a specific device. Information about the device of interest (200) selected by the user (U) may be provided to the user (U) in the form of a notification through an augmented reality image (VI).

[0040] In one embodiment, the augmented reality device (100) can identify a device that a user (U) is looking at through an eye tracking sensor. For example, the augmented reality device (100) can obtain a gaze direction of the user (U) through the eye tracking sensor. In one embodiment, the gaze direction of the user (U) can be represented by the position coordinates of an area that the user (U) is looking at in a real environment, or can be represented by a vector from the augmented reality device (100) to a location that the user (U) is looking at. In one embodiment, the gaze direction of the user (U) can be obtained by eye fixation analysis. In eye fixation analysis, if the gaze of the user (U) stays in an area within a certain range for a preset period of time, the device located in the area can be determined as the device that the user (U) is looking at. For example, the preset period of time can be approximately 150 ms to 300 ms. In one embodiment, the range of gaze direction fluctuations or the duration of gaze upon a specific device may vary depending on the user (U) of the augmented reality device (100). Accordingly, the range or preset duration for eye gaze analysis may vary depending on the user (U) of the augmented reality device (100).

[0041] Also, referring to FIG. 1, the augmented reality device (100) can acquire a real-world scene image (RI) through a camera. For example, the camera may be a depth camera or an RGB camera. For example, the real-world scene image (RI) acquired through the camera may be a depth image or an RGB image, respectively. Thereafter, the augmented reality device (100) can segment (or divide) the acquired real-world scene image (RI). The segmentation operation may represent or correspond to an operation of extracting an object from an image on a pixel-by-pixel basis. The segmentation operation can recognize or separate an object from an image or video. For example, each pixel can be labeled for the purpose of determining the location and shape of an object in an image through the segmentation operation, and identifying which pixel belongs to which object.

[0042] In one embodiment, the augmented reality device (100) can segment a real scene image (RI) according to (or based on) the shape, location, type, etc. of an object. The augmented reality device (100) can segment the real scene image (RI) using, for example, a panoptic segmentation technique that acquires object-specific identification information. Each segmented image piece can represent one object. That is, object recognition can be performed from the real scene image (RI) through the segmentation operation. Object recognition or object identification can refer to an operation of identifying an object included in an image or video. For example, object recognition can include analyzing visual information within an image to classify what type of object it is. An object can include any object in the image that is distinct from the background. Depending on the embodiment, the process of determining whether an object exists in the image and the location of the object may also be included in object recognition. Object recognition is a type of pattern recognition that can be used, for example, to identify objects in images or videos using trained neural network models.

[0043] An image fragment corresponding to one object can be expressed as an 'object image'. The augmented reality device (100) can identify at least one device image from a segmented reality scene image (RI). For example, the augmented reality device (100) can store identification information for at least one connected device, for example, information such as the type, shape, or color of each connected device. The augmented reality device (100) can match at least one object image to a connected device based on the stored device identification information and the acquired reality scene image (RI). An object image matched to a device can be identified as a device image.

[0044] In one embodiment, a segmented real scene image (RI) may include an object image that the augmented reality device (100) cannot identify which device it corresponds to. For example, a real scene image (RI) (acquired by the augmented reality device (100)) may include two or more devices of the same type. In this case, it may be difficult for the augmented reality device (100) to identify which object image corresponds to which of the two or more devices. For example, if a real scene image (RI) acquired by the augmented reality device (100) includes two object images for 'wireless earphones', the augmented reality device (100) may have difficulty identifying which of the two object images corresponds to the wireless earphones connected to it.

[0045] In one embodiment, the augmented reality device (100) may use other sensors or signal exchange methods to match an object image with a device. The augmented reality device (100) may match an object image with a device using a Global Positioning System (GPS), a Bluetooth Low Energy (BLE), or an Ultra Wide Broadband (UWB) method. For example, the GPS method may be effective when the device and the augmented reality device (100) are far apart, and the BLE method or UWB method may be effective when the device and the augmented reality device (100) are close together, for example, when the distance between the device and the augmented reality device (100) is less than 100 m. In one embodiment, the augmented reality device (100) may also exchange signals with a connected device to match the device and an object image. For example, the augmented reality device (100) may request a connected device (e.g., wireless earphones) to output sound, and the device that receives the request may output the sound. Sound can have various frequencies and is not limited to the range that the augmented reality device (100) or the user (U) of the device can hear. The augmented reality device (100) can receive the sound output by the device to determine the characteristics of the device (e.g., direction, position, distance), and can identify an object image matching the device using information about the determined characteristics of the device (e.g., direction, position, distance). In one embodiment, the augmented reality device (100) can preferentially perform the BLE method or the UWB method over the method of exchanging signals to match the object image and the device.

[0046] The augmented reality device (100) can determine the device that the user (U) is looking at based on the location information within the real scene image (RI) of at least one device image and the viewing direction information of the user (U). For example, the augmented reality device (100) can determine the device corresponding to the object image located in the viewing direction of the user (U) as the device that the user (U) is looking at.

[0047] The augmented reality device (100) can identify whether the device being looked at by the user (U) is connected to the augmented reality device (100), and if the device being looked at by the user (U) is connected to the augmented reality device (100), the device being looked at by the user (U) can be determined as a device of interest (200).

[0048] In one embodiment, the augmented reality device (100) may determine whether to display information about a device of interest (200). For example, if the user (U) is able to obtain information about the device of interest (200), the augmented reality device (100) may decide not to display the information since there is no need to display the information redundantly. If the user (U) is unable to obtain information about the device of interest (200), the augmented reality device (100) may provide the information to the user (U) by displaying it through an augmented reality image (VI).

[0049] In one embodiment, the operation of determining whether or not to display information about a device of interest (200) by the augmented reality device (100) may be to determine to display information about the device of interest (200) if the device of interest (200) does not include a display screen. If the device of interest (200) does not include its own display screen, the user (U) cannot view information about the device. Therefore, in such a case, the augmented reality device (100) may receive information about the device of interest (200) from the device of interest (200) and display the information as an augmented reality image (VI) so that the user (U) can view it.

[0050] In one embodiment, whether the device of interest (200) includes a display screen may be included in device identification information received from the device of interest (200). For example, the augmented reality device (100) may receive device identification information and notification information from the device of interest (200). The device identification information and notification information may be received in a single step or in multiple separate steps.

[0051] In one embodiment, the notification information may be received after the augmented reality device (100) decides to display information about the device of interest (200). The device identification information is information for identifying the device of interest (200), and may include, for example, at least one of the type, color, or device ID of the device. The notification information may be related to the device of interest (200) for which a notification needs to be provided to the user. For example, the notification information may include information related to an event that occurred in the device of interest (200), such as the remaining battery level of the device of interest (200), the termination of operation of the device of interest (200), or information related to the current status of the device of interest (200).

[0052] In one embodiment, the augmented reality device (100) can identify whether the device of interest (200) includes its own display screen based on device identification information received from the device of interest (200). The augmented reality device (100) can decide to provide notification information (among information related to the device of interest (200)) to the user through an augmented reality image.

[0053] In one embodiment, the operation of determining whether the augmented reality device (100) displays information about the device of interest (200) may be, if the device of interest (200) includes a display screen, determining whether to display information about the device of interest (200) based on a facing direction of the display screen. The facing direction of the display screen indicates the direction in which the display screen is facing.

[0054] The augmented reality device (100) can detect the display screen of the device of interest (200) based on device identification information received from the device of interest (200). The augmented reality device (100) can calculate the facing direction of the display screen based on the form in which the display screen of the device of interest (200) appears on a real scene image (RI) acquired through a camera. For example, the facing angle of the display screen can be calculated based on an affine transformation value of an image corresponding to the display screen of the device of interest (200) on the real scene image (RI).

[0055] In one embodiment, the augmented reality device (100) may determine to display information about the device of interest (200) when a difference between a facing direction of the display screen of the device of interest (200) and a direction from the device of interest (200) to the augmented reality device (100) is greater than a preset threshold angle. The direction from the device of interest (200) to the augmented reality device (100) may represent a direction of a straight line connecting the center of the display screen of the device of interest (200) to the camera of the augmented reality device (100). The direction from the camera of the augmented reality device (100) to the center of the display screen of the device of interest (200) may correspond to a gaze direction of a user (U) wearing the augmented reality device (100). Accordingly, when the facing direction of the display screen of the device of interest (200) and the direction from the device of interest (200) to the augmented reality device (100) match, the user (U) wearing the augmented reality device (100) can best recognize the content displayed on the display screen of the device of interest (200), and as the difference between the two directions increases, it becomes more difficult for the user (U) to recognize the content displayed on the display screen of the device of interest (200).

[0056] In one embodiment of the present disclosure, if the difference between the facing direction of the display screen of the device of interest (200) and the direction from the device of interest (200) to the augmented reality device (100) is greater than a preset threshold angle, it may be determined that the user (U) cannot see the content displayed on the display screen of the device of interest (200). If the facing direction of the display screen of the device of interest (200) is significantly different from the direction from the device of interest (200) to the augmented reality device (100), the user (U) of the augmented reality device (100) may not be able to see the display screen of the device of interest (200), and even if the device of interest (200) displays related information on its own (for example, on the display screen of the device of interest (200), the user (U) may not be able to obtain the information. Therefore, in this case, the augmented reality device (100) may provide the user (U) with information that the user (U) cannot obtain by displaying it through an augmented reality image (VI).

[0057] In one embodiment, the operation of determining whether the augmented reality device (100) displays information about the device of interest may be, if the device of interest (200) includes a display screen, determining whether to display information about the device of interest (200) based on a distance between the device of interest (200) and the augmented reality device (100). The distance between the device of interest (200) and the augmented reality device (100) may represent (or correspond to) a straight-line distance from the center of the display screen of the device of interest (200) to the camera of the augmented reality device (100). The distance between the device of interest (200) and the augmented reality device (100) may be measured through various sensors, such as, for example, a stereo camera, a depth sensor, a depth camera, a proximity sensor, a ToF (time of flight) type sensor, or may be measured through signal exchange between the device of interest (200) and the augmented reality device (100). In one embodiment, the distance between the device of interest (200) and the augmented reality device (100) may be measured in a variety of ways, including but not limited to the examples described above.

[0058] The distance from the camera of the augmented reality device (100) to the center of the display screen of the device of interest (200) may correspond to the eye distance for a user (U) wearing the augmented reality device (100) to read the content displayed on the display screen. Accordingly, the farther the distance between the display screen of the device of interest (200) and the augmented reality device (100), the more difficult it is for the user (U) to recognize the content displayed on the display screen of the device of interest (200).

[0059] In one embodiment, if the distance between the device of interest (200) and the augmented reality device (100) is greater than a preset threshold, it may be determined that the user (U) cannot see the content displayed on the display screen of the device of interest (200). If the distance between the device of interest (200) and the augmented reality device (100) is (relatively) far, the user (U) of the augmented reality device (100) may not be able to see the display screen of the device of interest (200), and even if the device of interest (200) displays related information on its own, the user (U) may not be able to obtain the information. Therefore, in this case, the augmented reality device (100) may provide the user (U) with information that the user (U) cannot obtain by displaying it through an augmented reality image (VI). In one embodiment, the preset threshold for the distance between the device of interest (200) and the augmented reality device (100) may be determined based on user information such as eyesight.

[0060] In one embodiment, the augmented reality device (100) may display an augmented reality image (VI) including information about a device of interest (200). The augmented reality image (VI) may include a virtual image (VI2) representing information about the device of interest (200). The augmented reality image (VI) may further include an indicator (VI1) indicating the device of interest (200) corresponding to the information.

[0061] The augmented reality device (100) can determine an area in which to display an augmented reality image (VI) based on the location of the image of the device of interest (200) within the real scene image (RI). Thereafter, the augmented reality device (100) can display the augmented reality image (VI) in the determined area.

[0062] In one embodiment, referring to FIG. 1, an indicator (VI1) indicating a device of interest (200) may be displayed on the border of the device of interest (200) or may be displayed to overlap with an image of the device of interest (200). The indicator (VI1) indicating the device of interest (200) may highlight the image of the device of interest (200) through various methods and may provide the user (U) with location information about the device of interest (200).

[0063] In one embodiment, the augmented reality device (100) can acquire a real scene image (RI) through a camera and segment the acquired real scene image (RI) to identify an image of a device of interest (200). An indicator (VI1) indicating the device of interest (200) can be displayed in an area corresponding to the image of the identified device of interest (200).

[0064] In one embodiment, referring to FIG. 1, a virtual image (VI2) representing information about a device of interest (200) may be displayed near an image of the device of interest (200) or may be displayed by overlapping with an image of the device of interest (200). In one embodiment, the virtual image (VI2) representing information about the device of interest (200) may not be displayed near the image of the device of interest (200). For example, the virtual image (VI2) may be displayed in an area without other objects in the background so that the user (U) can easily recognize the information, or may be displayed enlarged in the direction of the user's (U) gaze. If the virtual image (VI2) is not displayed near the image of the device of interest (200), the augmented reality image (VI) may further include a connection line or the like indicating which device of interest (200) the virtual image (VI2) corresponds to.

[0065] FIG. 2 is a flowchart of a method for displaying information by an augmented reality device according to one embodiment of the present disclosure.

[0066] In step 210, the augmented reality device identifies a device of interest. The device of interest may represent a target object for information provision via the augmented reality device.

[0067] Among the devices connected to the AR device, a device requiring notification may be a device of interest. For example, a device of interest may include at least one of the following: a device connected to the AR device where an event occurred, a device selected by the user among the devices connected to the AR device, or a device newly connected to the AR device.

[0068] When an event requiring notification occurs on a specific device connected to an AR device, if the device includes a display, the device may display a notification on the display. In one embodiment, if the device does not include a display for displaying notifications on its own, the AR device may be used to provide the user with notifications related to the device.

[0069] When a device that was not previously connected to the AR device is newly connected to the AR device, the newly connected device may become a device of interest requiring notification through the AR device. When a device that was not previously connected to the AR device is newly paired with the AR device, the AR device may display an AR image (VI) to notify the user of the newly paired device.

[0070] In one embodiment, the device of interest may be determined by user selection. For example, if the user selects a specific device via a voice signal or gaze signal, and that device is connected to the AR device, the AR device may determine that the user wishes to receive notifications or status information about that device and provide notifications or status information about that device.

[0071] In one embodiment, the augmented reality device can identify the device the user is gazing at via an eye tracking sensor. The user's gaze direction can be determined based on eye gaze analysis, when the user's gaze remains within a certain range for a preset period of time. The device located within the user's gaze direction can be determined as the device the user is gazing at. For example, the preset time period can range from approximately 150 ms to 300 ms.

[0072] Afterwards, the AR device can identify whether the device being viewed by the user is connected to the AR device. If the device being viewed by the user is connected to the AR device, the AR device can determine that device as a device of interest.

[0073] In one embodiment, the step of identifying a device that a user is looking at through a gaze tracking sensor may include the step of obtaining a gaze direction of the user through the gaze tracking sensor, the step of obtaining a real scene image through a camera, the step of identifying at least one device from the real scene image, and the step of determining the device that the user is looking at based on a position of at least one device within the real scene image and the gaze direction of the user.

[0074] In step 220, the AR device determines whether to display information about the device of interest. For example, if the user can obtain certain information through other channels, the AR device may decide not to display that information, as there is no need to display that information redundantly. The information the AR device decides to display may include information that the user cannot obtain through other channels.

[0075] In one embodiment, an AR device may receive device identification information from a device of interest. The device identification information may include information regarding whether the device of interest includes a display screen. Based on the received device identification information, if the device of interest does not include a display screen, the AR device may determine to display notification information for the device.

[0076] In one embodiment, even if the device of interest includes its own display screen, if the augmented reality device determines that the user cannot read the information displayed on that display screen, the augmented reality device may decide to display information about the device of interest through an augmented reality image.

[0077] For example, if the facing direction of the display screen of the device of interest differs from the direction from the device of interest to the augmented reality device by a greater than preset threshold angle, it may be determined that the user cannot read the display screen of the device of interest and a notification may be displayed for the device of interest.

[0078] For example, even if the distance measured by the AR device to the device of interest is greater than a preset value, the AR device may determine that the user cannot read the display screen of the device of interest and decide to display a notification for the device of interest. In one embodiment, the preset threshold for the distance between the device of interest and the AR device may be determined based on user information such as eyesight.

[0079] In step 230, the augmented reality device displays an augmented reality image including information about the device of interest. Information related to the device of interest can be displayed as an augmented reality image through the augmented reality device.

[0080] In one embodiment, an augmented reality device can identify a device of interest from a real-world scene image acquired through a camera, and determine an area in which to display an augmented reality image based on the location of the device of interest. The area in which to display the augmented reality image can be determined as an area near (or adjacent to) the device corresponding to the corresponding information, or an area moved closer in the direction of the user's gaze from the device corresponding to the corresponding information. Thereafter, the augmented reality device can display the augmented reality image in the determined area.

[0081] FIG. 3 illustrates an operation of an augmented reality device according to one embodiment of the present disclosure to display an augmented reality image including information about a device of interest.

[0082] The operations illustrated in FIG. 3 may be performed after a device of interest has been identified. After identifying a device of interest, the AR device may determine whether to display information about the device of interest. The operation of determining whether to display information about the device of interest may be performed through steps 310 and 320.

[0083] In step 310, the AR device can identify whether information about the device of interest is already displayed. When determining whether to display information about the device of interest, the AR device can consider whether the user is in a state where the information can be obtained. The information about the device of interest can be displayed by the AR device through an AR image or through a display included in the device of interest. If the AR device is already displaying information about the device of interest through an AR image, the AR device may not need to present that information to the user again. If the device of interest includes a display and already provides notifications on its own, the AR device may not need to present that information to the user again. Therefore, if information about the device of interest is already displayed, the process proceeds to step 320 to determine again whether to display the information.

[0084] In one embodiment, if the device of interest does not include a separate display screen or does not provide notifications on its own (e.g., does not provide notifications through the screen of the device of interest), or if the augmented reality device is not currently displaying information about the device of interest, then step 330 may be proceeded to.

[0085] At step 320, the augmented reality device can identify whether information about the device of interest is already displayed but cannot be viewed by the user.

[0086] For example, even if the device of interest includes its own display screen, if it is determined that the user cannot view the information displayed on the display screen, the AR device may decide to display information about the device of interest through the AR image. For example, if the facing direction of the display screen of the device of interest differs by a greater than a preset threshold angle from the direction from the device of interest to the AR device, the AR device may determine that the user cannot view the display screen of the device of interest and decide to display a notification about the device of interest. For example, even if the distance measured by the AR device to the device of interest is greater than a preset value, the AR device may determine that the user cannot view the display screen of the device of interest and decide to display a notification about the device of interest. In one embodiment, the preset threshold for the distance between the device of interest and the AR device may be determined based on user information such as eyesight.

[0087] For example, even if the augmented reality device was already displaying information about the device of interest as an augmented reality image, if the currently displayed information is old information and the user cannot view the updated information about the device of interest, the augmented reality device may decide to display the updated information again.

[0088] In step 330, the AR device can identify information to be displayed as an AR image. The AR device can receive information from the device of interest that needs to be provided to the user. For example, information about the device of interest that needs to be provided to the user can be displayed as notification information. The notification information may be information related to the device of interest that needs to be notified to the user. For example, the notification information may include information related to an event occurring on the device of interest, such as the remaining battery level of the device of interest, the termination of the device of interest, or information related to the current status of the device of interest.

[0089] In one embodiment, an AR device can select information to display through an AR image among information related to a device of interest. The selection of information to display may be based on the priority of the information.

[0090] In step 340, the augmented reality device can recognize the distance between the user and the device of interest. Step 340 may be performed optionally. The distance between the user of the augmented reality device and the device of interest measured in step 340 may be used to determine an area in which to display an augmented reality image corresponding to the device of interest.

[0091] In step 350, the augmented reality device may determine an area in which to display the augmented reality image. The augmented reality device may determine an area in which to display the augmented reality image based on the location of the device of interest within the real scene image.

[0092] In one embodiment, the augmented reality device can identify a device of interest from a real scene image acquired through a camera, and determine an area in which to display an augmented reality image based on the location of the device of interest. The area in which to display the augmented reality image can be determined as an area near (or adjacent to) the device corresponding to the corresponding information, or an area moved closer in the direction of the user's gaze from the device corresponding to the corresponding information. For example, when the distance between the user of the augmented reality device and the device of interest is far, the area in which to display the augmented reality image can be determined as an area moved closer in the direction of the user's gaze from the device corresponding to the corresponding information. Thereafter, in step 360, the augmented reality device can display the augmented reality image in the area determined in step 350.

[0093] An augmented reality image may include a virtual image that displays information about a device of interest. The augmented reality image may further include an indicator that indicates the device of interest corresponding to the information.

[0094] In one embodiment, the indicator indicating the device of interest may be displayed within the border of the device of interest or overlapped with the image of the device of interest. The indicator indicating the device of interest may highlight the image of the device of interest through various methods and provide the user with location information about the device of interest.

[0095] In one embodiment, a virtual image representing information about a device of interest may be displayed near an image of the device of interest or may be displayed by overlapping with an image of the device of interest. In one embodiment, a virtual image representing information about a device of interest may not be displayed near an image of the device of interest (200). For example, the virtual image (VI2) may be displayed in an area without other objects in the background so that the user (U) can easily recognize the information, or may be displayed enlarged in the direction of the user's (U's) gaze. If the virtual image (VI2) is not displayed near an image of the device of interest (200), the augmented reality image (VI) may further include a connection line or the like indicating which device of interest (200) the virtual image (VI2) corresponds to.

[0096] FIG. 4 is a diagram for explaining an operation of identifying a device of interest (200) based on a selection of a user (U) according to one embodiment of the present disclosure.

[0097] In one embodiment, a device of interest (200) may be selected by a user (U) from among devices connected to an augmented reality device (100). The user (U) may select a device of which he or she wishes to know related information as a device of interest (200). The user (U) may select a specific device as a device of interest (200) by gazing at it (a), or may select the device of interest (200) via a voice signal (b). Information about the device of interest (200) selected by the user (U) may be provided to the user (U) in the form of a notification via an augmented reality image.

[0098] Referring to (a) of FIG. 4, the augmented reality device (100) can identify a device that a user (U) is looking at through a gaze tracking sensor in one embodiment. For example, the augmented reality device (100) can obtain a gaze direction of the user (U) through the gaze tracking sensor. The gaze direction of the user (U) can be obtained through eye gaze analysis. In eye gaze analysis, if the gaze of the user (U) remains in a certain range of an area for a preset period of time or longer, the device located in the area can be determined as the device that the user (U) is looking at.

[0099] The augmented reality device (100) can determine a device corresponding to an object image located in the direction of the user's (U's) gaze as the device being gazed by the user (U). If the device being gazed by the user (U) is connected to the augmented reality device (100), the device being gazed by the user (U) can be determined as a device of interest (200).

[0100] Referring to (b) of FIG. 4, a user can also select a specific device as a device of interest (200) through a voice signal.

[0101] Speech recognition refers to an operation of converting a voice signal acquired through an input device such as a microphone into a form that can be processed by an electronic device (e.g., an augmented reality device (100)). For example, speech recognition includes converting an acoustic speech signal of a voice acquired by an augmented reality device into text such as words or sentences, and may also be referred to as computer speech recognition or speech-to-text (STT). In one embodiment, the augmented reality device (100) may select a device selected by a user (U) as a device of interest (200) through speech recognition.

[0102] Natural language understanding (NLU) refers to the act of processing text or speech in natural language so that a computer can understand it. Natural language understanding can enable an augmented reality device (100) to operate according to user input given in natural language (e.g., "Tell me the status of the Galaxy Buds Pro").

[0103] For example, referring to (b) of FIG. 4, the augmented reality device (100) can determine the device of interest (200) as 'Galaxy Buds Pro' through natural language understanding from the user's natural language voice input, and determine the information to be displayed as status information about the 'Galaxy Buds Pro', such as the remaining battery level and information on the music being played.

[0104] FIG. 5 is a drawing for explaining an operation of an augmented reality device (100) according to one embodiment of the present disclosure to identify a device of interest (200a, 200b) and display an augmented reality image (VI) including information about the device of interest (200a, 200b).

[0105] FIG. 5 illustrates an embodiment in which an augmented reality device (100) according to one embodiment of the present disclosure identifies a device being looked at by a user (U) as a device of interest through a gaze tracking sensor.

[0106] First, the user (U) can focus on the first device (wireless earphones, 200a) among the devices connected to the augmented reality device (100). In this case, the device of interest is determined to be the first device (200a). The first device (200a), which is a 'wireless earphone', does not have its own display and therefore can become the target (device of interest) of information provision through the augmented reality device (100).

[0107] The augmented reality device (100) can display information about a first device (200a) determined as a device of interest through an augmented reality image (VI). In one embodiment, the augmented reality image (VI) may further include a virtual image (VI2) representing information about the device of interest (200a) and an indicator (VI1) indicating the device of interest (200a) corresponding to the information. For example, the indicator (VI1) indicating the device of interest (200a) may be displayed on the border of the device of interest (200a) to emphasize the device of interest (200a). The user can clearly recognize which information is related to which device through the virtual image (VI) including the indicator (VI1) and information (VI2).

[0108] In one embodiment, a connecting line may be further displayed between the indicator (VI1) and the information (VI2) connecting these virtual images. If the virtual image representing the information (VI2) is displayed near the corresponding device of interest, it can be clearly understood which device the information is related to. However, if the virtual image representing the information (VI2) is displayed far away from the corresponding device of interest, the connecting line connecting the information (VI2) and the indicator (VI1) of the device of interest can provide a more effective notification to the user.

[0109] In one embodiment, a user (U) may gaze at a first device (200a) and then move his / her gaze to a second device (mobile phone, 200b). If the user (U) gazes at the second device (200b) for a certain period of time, the device of interest may change to the second device (200b).

[0110] Referring to FIG. 5, the second device (200b) has its own display, but since the display screen faces downward, the user may not be able to see the content displayed on the screen of the second device (200b). In this case, the second device (200b) may become a target (device of interest) of information provision via the augmented reality device (100).

[0111] The augmented reality device (100) can display information about a second device (200b), which is a changed (new) device of interest, through an augmented reality image (VI). In one embodiment, the augmented reality image (VI) may further include a virtual image (VI4) representing information about the device of interest (200b) and an indicator (VI3) indicating the device of interest (200b) corresponding to the information. For example, the indicator (VI3) indicating the device of interest (200b) may be displayed on the border of the device of interest (200b) to highlight the device of interest (200b). A user can recognize which information is related to which device through the virtual image (VI) including the indicator (VI3) and information (VI4).

[0112] In one embodiment, the augmented reality device (100) can display information about only one device of interest at a time. In this case, when the device of interest changes, the virtual images (VI1, VI2) corresponding to the device of interest before the change are no longer displayed, and virtual images (VI3, VI4) corresponding to the new device of interest may be displayed.

[0113] FIG. 6 is a drawing for explaining an operation of an augmented reality device (100) according to one embodiment of the present disclosure to display an augmented reality image (VI) including information about a plurality of devices of interest.

[0114] In one embodiment, the augmented reality device (100) may display information (VI2, VI4) for multiple devices of interest at once, unlike the aforementioned FIG. 5. In this case, the indicator (VI1) and information (VI2) for the first device and the indicator (VI3) and information (VI4) for the second device may be distinguished from each other by using colors, hatches, etc.

[0115] For example, in the embodiment of FIG. 6, the indicator (VI1) and information (VI2) for the first device, which is an 'earphone', may be highlighted and displayed in a first color (e.g., 'yellow'), and the indicator (VI3) and information (VI4) for the second device, which is a 'mobile phone', may be highlighted and displayed in a second color (e.g., 'green'). In this case, the connection line connecting the indicator (VI1) and information (VI2) for the first device may also be displayed in yellow, and the connection line connecting the indicator (VI3) and information (VI4) for the second device may be displayed in green.

[0116] FIGS. 7A to 7D are diagrams illustrating examples of an augmented reality device according to one embodiment of the present disclosure displaying an augmented reality image including information about a device of interest.

[0117] Referring to Figure 7a, when a user looks at an air purifier that does not include its own display, the AR device can provide the user with information related to the current status of the air purifier through a virtual image. The information related to the air purifier can be displayed by connecting it to the image of the air purifier via a connection line.

[0118] Referring to FIG. 7b, when a user gazes at a speaker that does not include its own display, the AR device can provide the user with information related to the current state of the speaker (such as the current volume and information about the music being played) through a virtual image. The information related to the speaker can be displayed overlapping the speaker image or displayed near the speaker image.

[0119] Referring to Figure 7c, when a user looks at a laptop, the AR device can determine whether to provide the user with information related to the laptop. In one embodiment, if the facing direction of the laptop screen is determined to be facing in a direction that makes it difficult for the user to view the screen, the AR device can provide the user with information related to the current state of the laptop through a virtual image. The information related to the laptop can be displayed by overlapping the laptop image, for example, on a portion corresponding to the laptop screen.

[0120] In Figure 7c, an issue requiring notification (e.g., low battery level) may occur on a mobile phone that the user cannot see the screen. In this case, the AR device can provide the user with information related to the mobile phone (e.g., low battery level) through a virtual image, even if the user is looking at another device. The information related to the mobile phone can be displayed by connecting it to the mobile phone image via a connection line.

[0121] Referring to FIG. 7D , the augmented reality device may provide the user with information related to an oven based on a user's selection or an issue. In FIG. 7D , a virtual image representing information about a device of interest (an oven) may not be displayed near the image of the device of interest. For example, the virtual image containing the information may be displayed in an area free of other objects in the background so that the user can easily recognize the information, or may be displayed enlarged in the direction of the user's gaze. If, as in FIG. 7D , the virtual image containing information about the device of interest is not displayed near the image of the device of interest, the augmented reality image may further include a connecting line indicating which device of interest the virtual image corresponds to.

[0122] FIG. 8 is a block diagram of an augmented reality device (100) according to one embodiment of the present disclosure.

[0123] Referring to FIG. 8, the augmented reality device (100) may include a communication interface (110), a camera (120), a sensor (130), a processor (140), a memory (150), and an output interface (160). The communication interface (110), the camera (120), the sensor (130), the processor (140), the memory (150), and the output interface (160) may be electrically and / or physically connected to each other, respectively. In one embodiment, the components of the augmented reality device (100) are not limited to those illustrated in FIG. 8. The augmented reality device (100) may be implemented with more components than those illustrated in FIG. 8, or may be implemented with fewer components than those illustrated in FIG. 8.

[0124] In one embodiment of the present disclosure, the augmented reality device (100) may be implemented as augmented reality glasses worn on a user's head, in which case the augmented reality device (100) may further include a power supply (e.g., a battery) that supplies driving power to a communication interface (110), a camera (120), a sensor (130), a processor (140), and an output interface (160). In one embodiment, the augmented reality device (100) may not include a speaker (164).

[0125] The communication interface (110) is configured to transmit and receive data with a server or an external device (e.g., a device of interest) via a wired or wireless communication network. The communication interface (110) may perform data communication with the server or the external device using at least one of data communication methods including, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, Zigbee, Wi-Fi Direct (WFD), infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), World Interoperability for Microwave Access (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and RF communication. However, the present invention is not limited thereto, and when the augmented reality device (100) is implemented as a wearable device such as smart glasses, the communication interface (110) may transmit and receive data with a server or an external device through a network that follows a mobile communication standard such as a communication method using CDMA, WCDMA, 3G, 4G (LTE), 5G Sub 6, and / or millimeter wave (mmWave).

[0126] In one embodiment, the communication interface (110) may receive status information of the device of interest or information to be provided to the user as an augmented reality image from the device of interest under the control of the processor (140). The communication interface (110) may provide the received information of the device of interest to the processor (140).

[0127] In one embodiment, the communication interface (110) may be connected to at least one device of interest via a short-range communication method such as Bluetooth or Wi-Fi Direct, and may receive various information to be provided as notifications to the user from the device of interest.

[0128] The camera (120) is configured to acquire two-dimensional (2D) image data by photographing a real space. The camera (120) may be implemented in a small form factor so that it can be mounted on the augmented reality device (100) and may be a lightweight RGB camera that consumes low power. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the camera (120) may be implemented as any type of camera known in the art, such as an RGB-depth camera including a depth estimation function, a dynamic vision sensor camera, a stereo fish-eye camera, a grayscale camera, or an infrared camera. In one embodiment, the camera (120) may be positioned in a direction facing a user and configured to photograph the user's face.

[0129] The camera (120) may include a lens module, an image sensor, and an image processing module. The camera (120) may acquire still images or video of a real-world scene by an image sensor (e.g., a complementary metal-oxide-semiconductor (CMOS) or a charge-coupled device (CCD)). The video may include multiple image frames acquired in real time by photographing a real-world area through the camera (120). The image processing module may encode still images composed of a single image frame acquired through the image sensor or video data composed of multiple image frames and transmit the encoded data to the processor (140).

[0130] According to one embodiment of the present disclosure, capturing a real scene image may include an operation of the augmented reality device (100) to control a camera (e.g., a camera including an image sensor and a lens) (120) provided in the augmented reality device (100) to convert an optical image formed through the lens into an electrical signal to acquire an image. For example, one or more processors (140) may control the camera (120) provided in the augmented reality device (100) to capture a surrounding area of ​​the augmented reality device (100) to acquire an image (e.g., a captured image) including one or more frames. Here, the image may include a live-view image.

[0131] The sensor (130) may include sensors configured to detect real space, location, situation, or user information. In one embodiment of the present disclosure, the sensor (130) may include, but is not limited to, an eye tracking sensor, an Inertial Measurement Unit (IMU) sensor, a Global Positioning System (GPS) sensor, a Bluetooth Low Energy (BLE) sensor, an Ultra Wide Broadband (UWB) sensor, or other sensors capable of sensing various signals.

[0132] The processor (140) can execute one or more instructions of a program stored in the memory (150). The processor (140) may be composed of hardware components that perform arithmetic, logic, and input / output operations and image processing. Although FIG. 8 illustrates a single processor (140), this is not limiting and the processor (140) may be composed of one or more elements. The processor (140) may correspond to one or more processors.

[0133] The processor (140) may be composed of one processor or a plurality of processors, and may be a general-purpose processor such as one or more CPUs (Central Processing Units), APs (Application Processors), DSPs (Digital Signal Processors), a graphics-only processor such as a GPU (Graphics Processing Unit), VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU (Neural Processing Unit). The processor (140) may be controlled to process input data according to predefined operating rules or artificial intelligence models. Alternatively, when the processor (140) is an artificial intelligence-only processor, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0134] The memory (150) may be configured as at least one type of storage medium among, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk. The memory (150) may correspond to one or more memory devices or storage.

[0135] The memory (150) may store instructions related to functions and / or operations for the augmented reality device (100) to provide information about a device of interest to the user. In one embodiment, the memory (150) may store at least one of instructions, algorithms, data structures, program codes, and application programs that can be read by the processor (140). The instructions, algorithms, data structures, and program codes stored in the memory (150) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0136] The processor (140) can execute instructions or program codes stored in the memory (150) and control the overall operation of the augmented reality device (100). The processor (140) can perform operations according to one embodiment of the present disclosure. For example, the processor (140) can control the communication interface (110), the camera (120), the sensor (130), and the output interface (160) overall by executing programs stored in the memory (150).

[0137] The processor (140) may be configured with hardware components that perform arithmetic, logic, and input / output operations and signal processing. The processor (140) may be configured with at least one of, for example, a central processing unit (CPU), a microprocessor, a graphic processing unit (GRAP), an application specific integrated circuits (ASICs), a digital signal processor (DSPs), a digital signal processing device (DSPDs), a programmable logic device (PLDs), and a field programmable gate array (FPGAs), but is not limited thereto.

[0138] In one embodiment, the processor (140) may execute one or more instructions stored in the memory (150) to identify a device of interest, determine whether to display information about the device of interest, and display an augmented reality image including information about the device of interest through the display unit (162) of the output interface (160) to provide a user with a notification or information about the device of interest.

[0139] The output interface (160) is configured to output information about the device of interest as an augmented reality image (virtual image) or an audio signal under the control of the processor (140). The output interface (160) may include a display unit (162) and a speaker (164).

[0140] The display unit (162) may be composed of at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display.

[0141] In one embodiment, when the augmented reality device (100) is configured as augmented reality glasses, the display unit (162) may be configured as a lens optical system and may include a waveguide and an optical engine. The optical engine may be configured as a projector that generates light of an augmented reality image composed of text, icons, or virtual images, and projects the light onto a waveguide. The optical engine may include, for example, an image panel, an illumination optical system, a projection optical system, etc. In one embodiment, the optical engine may be disposed on a frame or a temple of the augmented reality glasses.

[0142] The speaker (164) is configured to output an acoustic signal. In one embodiment, the speaker (164) may output a voice message or notification sound to assist in providing information to a device of interest under the control of the processor (140).

[0143] FIG. 9 is a diagram illustrating a glass-type augmented reality device (900) according to one embodiment of the present disclosure.

[0144] Referring to FIG. 9, an augmented reality device (900) capable of displaying information about a device of interest through an augmented reality image is illustrated. The augmented reality device (900) is a device capable of providing augmented reality (AR) related services, and may generally include AR glasses in the shape of glasses worn on the user's face, a head mounted display (HMD) worn on the head, a virtual reality headset (VRH), or an AR helmet. In the case of a head mounted device, a display can be placed in front of the user's eyes to provide a very large screen to the user, and the screen moves according to the user's movements, so that a realistic virtual world can be provided.

[0145] In one embodiment, a user may wear an augmented reality device (900) capable of displaying visual augmented reality content. The augmented reality device (900) may include an audio module capable of providing audio augmented reality content to the user. In one embodiment, the augmented reality device (900) may include one or more cameras capable of capturing images and videos of an environment. The augmented reality device (900) may include an eye tracking system to determine the user's vergence distance. In one embodiment, the augmented reality device (900) may include a lightweight head-mounted display (HMD) (e.g., goggles, glasses, visor, etc.). In one embodiment, the augmented reality device (900) may include a device such as a lightweight portable display device or one or more laser projection glasses (e.g., glasses capable of projecting a low-powered laser onto a user's retina to project and display images or depth content to the user).

[0146] In one embodiment, the augmented reality device (900) may provide an AR service that outputs at least one virtual object to be overlapped in an area determined to be a user's field of view (FOV). For example, the area determined to be a user's field of view may be an area determined to be recognizable by a user wearing the augmented reality device (900) through the augmented reality device (900), and may be an area including all or at least a portion of the display of the augmented reality device (900). In one embodiment, the augmented reality device (900) may include a plurality of transparent members (920, 930) corresponding to each of the user's eyes.

[0147] In one embodiment, the augmented reality device (900) may include a display module (914), a camera, an audio output, and a support (921, 922).

[0148] The camera can capture images corresponding to the user's field of view or measure the distance to an object. The camera may correspond to the camera (120) of FIG. 8 described above. In one embodiment, the camera may be used for head tracking and spatial recognition. The camera may also recognize the user's movements.

[0149] In one embodiment, the camera may further include an 'ET (eye tracking) camera (912)' in addition to the camera (913) that acquires an image corresponding to the user's field of view, i.e., a real scene image, detects an object, or is used for spatial recognition. In one embodiment, the ET camera (912) may be used to detect and track the user's pupils. The ET camera (912) may be used to adjust the center of a virtual image projected on the augmented reality device (900) so that it is positioned according to the direction in which the pupils of the user wearing the augmented reality device (900) are looking. For example, the ET camera (912) may use a GS (Global shutter) camera to detect the pupil and track rapid eye movements without delay. The ET camera (912) may separately include a left-eye camera (912-1) and a right-eye camera (912-2).

[0150] In one embodiment, the display module (914) may include a first display (930) and a second display (920). The display module (914) may correspond to the display unit (162) of FIG. 8 described above. A virtual object output through the display module (914) may include information related to an application program running on the augmented reality device (900), information about a device of interest, or information related to an external object located in a real space corresponding to an area determined to be within the user's field of view. For example, the augmented reality device (900) may identify an external object included in at least a portion corresponding to an area determined to be within the user's field of view among image information related to a real space acquired through the camera (913). The augmented reality device (900) may output a virtual object related to the external object identified in at least a portion through an area determined to be within the user's field of view among the display areas of the augmented reality device (900). The external object may include an object existing in a real space.

[0151] In one embodiment, the display (920, 930) may include a light-collecting lens or a waveguide (waveguide or waveguide) in a transparent member. For example, the transparent member may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be completely transparent or translucent. In one embodiment, the transparent member may include a first transparent member (930) facing the right eye of a user wearing the augmented reality device (900) and a second transparent member (920) facing the left eye of the user. If the display is transparent, it may be positioned facing the user's eyes to display a screen.

[0152] A waveguide can transmit light generated from a light source of a display to a user's eyes. For example, the waveguide can be at least partially positioned on a portion of a transparent member (920, 930). According to one embodiment, light emitted from the display can be incident on one end of the waveguide, and the incident light can be transmitted to the user's eyes through total internal reflection within the waveguide. The waveguide can be made of a transparent material such as glass, plastic, or polymer, and can include a nano-pattern formed on one surface of an inner or outer surface, for example, a grating structure having a polygonal or curved shape. In one embodiment, the incident light can be propagated or reflected within the waveguide by the nano-pattern and provided to the user's eyes. In one embodiment, the waveguide can include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a mirror). In one embodiment, the waveguide can guide display light emitted from a light source to a user's eye using at least one diffractive element or reflective element.

[0153] In one embodiment, the display (920, 930) may include a display panel or a lens (e.g., glass). For example, the display panel may include a transparent material, such as glass or plastic. In one embodiment, the display may be formed of a transparent element, allowing a user to see through the display and perceive the actual space behind the display. The display may display a virtual object on at least a portion of the transparent element so that the user sees the virtual object superimposed on at least a portion of the actual space.

[0154] In one embodiment, the support (921, 922) may include a printed circuit board (PCB) (931-1, 931-2) for transmitting electrical signals to each component of the augmented reality device (900), a speaker (932-1, 932-2) for outputting audio signals, or a battery (933-1, 933-2) for supplying power. The speaker (932-1, 932-2) for outputting audio signals may correspond to the speaker (164) of FIG. 8 described above. For example, in a glasses-type augmented reality device (900), the support (921, 922) may be arranged on the temple portion of the glasses. The support (921, 922) may include a hinge portion (940-1, 940-2) for coupling to the main body portion of the augmented reality device (900). The speakers (932-1, 932-2) may include a first speaker (932-1) for transmitting audio signals to the user's left ear and a second speaker (932-2) for transmitting audio signals to the user's right ear.

[0155] Referring to FIG. 9, the augmented reality device (900) may include a microphone (941) for receiving a user's voice and ambient sounds. In addition, the augmented reality device (900) may include at least one illumination LED (942) to improve the accuracy of at least one camera (e.g., an ET camera (912), an outward-facing camera (913), or a recognition camera (911-1, 911-2)). For example, the illumination LED (942) may be used as an auxiliary means to improve the accuracy when capturing a user's pupil with the ET camera (912), and the illumination LED (942) may use an IR LED of an infrared wavelength rather than a visible light wavelength. For example, the illumination LED (942) may be used as an auxiliary means when capturing a user's gesture with the recognition camera (911-1, 911-2) when it is difficult to detect a subject in a dark environment.

[0156] According to one embodiment, the display module (914) may include a first light guide plate (930) corresponding to the right eye and a second light guide plate (920) corresponding to the left eye, and may provide visual information to the user through the first light guide plate (930) and the second light guide plate (920). According to one embodiment, the display module (914) may include a display panel and a lens (e.g., a glass lens, an LC lens). The display panel may include a transparent material such as glass or plastic.

[0157] In one embodiment, the display module (914) may be comprised of a transparent element, such that a user can see through the display module (914) to perceive a real space behind the display module (914) and in front of the user. The display module (914) may display a virtual object on at least a portion of the transparent element so that the user sees that the virtual object is superimposed on at least a portion of the real space.

[0158] In one embodiment, the augmented reality device (900) can identify an external object included in at least a portion of the image information related to a real space acquired through the outward-facing camera (913), which corresponds to an area determined by the user's field of view (FoV). The augmented reality device (900) can output (or display) a virtual object related to the external object identified in at least a portion of the image information through an area determined by the user's field of view among the display areas of the augmented reality device (900). The external object may include an object existing in a real space. According to various embodiments, the display area where the augmented reality device (900) displays a virtual object may include a portion of the display module (914) (for example, at least a portion of the display panel). According to one embodiment, the display area may be an area corresponding to at least a portion of the first light guide plate (930) and the second light guide plate (920).

[0159] In one embodiment, the augmented reality device (900) may use an outward-facing camera (913) to measure a distance to a physical object positioned in the front direction of the augmented reality device (900). The outward-facing camera (913) may include a high-resolution camera, such as a high-resolution (HR) camera and a photo video (PV) camera.

[0160] The augmented reality device (900) according to one embodiment of the present disclosure is not limited to the configuration described above, and may include various components in various locations and in various numbers.

[0161] The present disclosure provides a method for an augmented reality device (100) to display information about another device (e.g., a device of interest (200)). The method of operating the augmented reality device (100) may include a step of identifying a device of interest (200) for displaying information. By identifying a device of interest (200) for displaying information among other devices, the augmented reality device (100) may provide notifications to only devices that require notifications to the user (U), rather than providing notifications for all external devices to the user (U). Accordingly, excessive information may be prevented from being provided to the user (U) and data overhead may be reduced. The method of operating the augmented reality device (100) may include a step of determining whether to display information about the device of interest (200). By directly determining whether to display information about the device of interest (200), the augmented reality device (100) may provide notifications only when a specific notification is not provided to the user (U) through any other path. If a specific notification can be displayed through a display screen built into a corresponding device or provided to the user through other means, the augmented reality device (100) may not provide the same information to the user repeatedly. Accordingly, excessive information may be prevented from being provided to the user (U) and data overhead may be reduced. The method of operating the augmented reality device (100) may include a step of displaying an augmented reality image including information about a device of interest (200). The augmented reality device (100) may provide the user with a necessary notification in a timely manner by displaying the specific notification near the corresponding device of interest (200) or by overlaying it on the corresponding device of interest (200).

[0162] In one embodiment, the step of determining whether to display information about a device of interest (200) may include the steps of receiving device identification information and notification information from the device of interest (200), identifying whether the device of interest (200) includes a display screen based on the device identification information, and determining to display notification information if the device of interest (200) does not include a display screen.

[0163] In one embodiment, the step of determining whether to display information about the device of interest (200) may include, if the device of interest (200) includes a display screen, the step of determining whether to display information about the device of interest (200) based on a facing direction of the display screen.

[0164] In one embodiment, the step of determining whether to display information about the device of interest (200) based on the facing direction of the display screen may include the step of determining to display information about the device of interest (200) when a difference between the facing direction of the display screen and the direction of the augmented reality device (100) from the device of interest (200) is greater than a preset threshold angle.

[0165] In one embodiment, the step of determining whether to display information about the device of interest (200) may include, if the device of interest (200) includes a display screen, the step of measuring a distance between the augmented reality device (100) and the device of interest (200), and the step of determining whether to display information about the device of interest (200) based on the distance between the augmented reality device (100) and the device of interest (200).

[0166] In one embodiment, the step of determining whether to display information about the device of interest (200) based on the distance between the augmented reality device (100) and the device of interest (200) may include the step of determining to display information about the device of interest (200) when the distance between the augmented reality device (100) and the device of interest (200) is greater than a preset threshold value.

[0167] In one embodiment, the device of interest (200) may include at least one of the devices connected to the augmented reality device (100) in which an event occurred, a device selected by the user among the devices connected to the augmented reality device (100), or a device newly connected to the augmented reality device (100).

[0168] In one embodiment, the step of identifying a device of interest (200) may include the step of identifying a device that the user is gazing at through a gaze tracking sensor, the step of identifying whether the device that the user is gazing at is connected to the augmented reality device (100), and if the device that the user is gazing at is connected to the augmented reality device (100), the step of determining the device that the user is gazing at as the device of interest (200).

[0169] In one embodiment, the step of identifying a device that a user is looking at through a gaze tracking sensor may include the step of obtaining a gaze direction of the user through the gaze tracking sensor, the step of obtaining a real-world scene image through a camera (120), the step of identifying at least one device from the real-world scene image, and the step of determining the device that the user is looking at based on a position of at least one device within the real-world scene image and the gaze direction of the user.

[0170] In one embodiment, the step of displaying an augmented reality image including information about a device of interest (200) may include the steps of acquiring a real-world scene image through a camera (120), identifying the device of interest (200) from the real-world scene image, determining an area in which to display the augmented reality image based on a location of the device of interest (200), and displaying the augmented reality image in the determined area.

[0171] The present disclosure provides an augmented reality device that displays information about a connected device. The augmented reality device may include a camera configured to acquire a real scene image. The augmented reality device may include a display unit. The augmented reality device may include a memory that stores a program including at least one command. The augmented reality device may include at least one processor. The at least one processor may identify a device of interest for displaying information, determine whether to display information about the device of interest, and display an augmented reality image including information about the device of interest through the display unit.

[0172] In one embodiment, at least one processor may receive device identification information and notification information from a device of interest. Based on the device identification information, the at least one processor may identify whether the device of interest includes a display screen. If the device of interest does not include a display screen, the at least one processor may determine to display notification information.

[0173] In one embodiment, at least one processor may determine whether to display information about a device of interest based on a facing direction of the display screen, if the device of interest includes a display screen.

[0174] In one embodiment, at least one processor may determine to display information about the device of interest if a difference between a facing direction of the display screen and an orientation of the augmented reality device from the device of interest is greater than a preset threshold angle.

[0175] In one embodiment, at least one processor may measure the distance between the augmented reality device and the device of interest, if the device of interest includes a display screen. The at least one processor may determine whether to display information about the device of interest based on the distance between the augmented reality device and the device of interest.

[0176] In one embodiment, at least one processor may determine to display information about the device of interest if a distance between the augmented reality device and the device of interest is greater than a preset threshold.

[0177] In one embodiment, at least one processor can identify a device being looked at by the user via an eye tracking sensor. The at least one processor can identify whether the device being looked at by the user is connected to an augmented reality device. If the device being looked at by the user is connected to the augmented reality device, the at least one processor can determine the device being looked at by the user as a device of interest.

[0178] In one embodiment, at least one processor can obtain the user's gaze direction through an eye tracking sensor. At least one processor can obtain a real-world scene image through a camera. At least one processor can identify at least one device from the real-world scene image. At least one processor can identify the device the user is looking at based on the location of at least one device within the real-world scene image and the user's gaze direction.

[0179] In one embodiment, at least one processor can acquire a real-world scene image via a camera. The at least one processor can identify a device of interest from the real-world scene image. The at least one processor can determine an area in which to display an augmented reality image based on the location of the device of interest. The at least one processor can display the augmented reality image in the determined area.

[0180] The present disclosure provides a computer program product comprising a computer-readable storage medium. The storage medium may store instructions readable by an augmented reality device for executing at least one of the disclosed methods on the augmented reality device.

[0181] Thus, according to one embodiment of the present disclosure, information can be effectively provided when a critical situation occurs on a device of interest without the user being aware of the information, or when the user desires to know the device's information. Furthermore, by selectively providing notifications only to devices that require notification, the user is prevented from being provided with excessive information and data overhead can be reduced.

[0182] Various embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be formed from computer-readable program code and embodied in a computer-readable medium. In the present disclosure, “application” and “program” may refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof suitable for implementation in computer-readable program code. “Computer-readable program code” may include various types of computer code, including source code, object code, and executable code. “Computer-readable medium” may include various types of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), a hard disk drive (HDD), a compact disc (CD), a digital video disc (DVD), or various types of memory.

[0183] Additionally, the device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, a 'non-transitory storage medium' is a tangible device and may exclude wired, wireless, optical, or other communication links that transmit temporary electrical or other signals. Meanwhile, this 'non-transitory storage medium' does not distinguish between cases where data is permanently stored in the storage medium and cases where it is temporarily stored. For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored. A computer-readable medium may be any available medium that can be accessed by a computer, and may include both volatile and non-volatile media, removable and non-removable media. A computer-readable medium includes a medium on which data can be permanently stored and a medium on which data can be stored and later overwritten, such as a rewritable optical disk or an erasable memory device.

[0184] The disclosed embodiments can be implemented as a software program including commands stored on a computer-readable storage medium. A computer, as a device capable of recalling commands stored on the storage medium and performing operations according to the disclosed embodiments in accordance with the recalled commands, can include an electronic device according to the disclosed embodiments.

[0185] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may include a software program and a computer-readable storage medium having the software program stored thereon. For example, the computer program product may include a product in the form of a software program (e.g., a downloadable app) distributed electronically by a device manufacturer or through an electronic market. For electronic distribution, at least a portion of the software program may be stored in the storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic market server, or a relay server that temporarily stores the software program.

[0186] In a system comprising a server and a device, the computer program product may include a storage medium of the server or a storage medium of the device. Alternatively, if a third device (e.g., a smartphone) exists that is communicatively connected to the server or device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the server to the device or the third device, or from the third device to the device.

[0187] In this case, one of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0188] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on the server to control a device in communication with the server to perform a method according to the disclosed embodiments.

[0189] As another example, a third device may execute a computer program product to control a device in communication with the third device to perform a method according to the disclosed embodiments. When the third device executes the computer program product, the third device may download the computer program product from a server and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a preloaded state to perform the method according to the disclosed embodiments.

[0190] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0191] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. In the operating method of the augmented reality device (100), Step of identifying a device of interest (200); A step of determining whether to display information about the above device of interest (200); and A method comprising the step of displaying an augmented reality image corresponding to information about the device of interest (200).

2. In paragraph 1, The step of determining whether to display information about the above device of interest (200) is as follows: A step of receiving device identification information and notification information from the above device of interest (200); A step of identifying whether the device of interest (200) includes a display screen based on the device identification information; and A method comprising the step of determining to display the notification information based on identifying that the device of interest (200) does not include a display screen.

3. In paragraph 1 or 2, The step of determining whether to display information about the above device of interest (200) is as follows: A method comprising: determining whether to display information about the device of interest (200) based on a facing direction of the display screen, based on identifying that the device of interest (200) includes a display screen.

4. In paragraph 3, The step of determining whether to display information about the device of interest (200) based on the facing direction of the display screen is as follows: A method comprising the step of determining to display information about the device of interest (200) based on a difference between a facing direction of the display screen and a direction of the augmented reality device (100) from the device of interest (200) being greater than a preset threshold angle.

5. In any one of paragraphs 1 to 4, The step of determining whether to display information about the above device of interest (200) is as follows: A step of measuring the distance between the augmented reality device (100) and the device of interest (200) based on identifying that the device of interest (200) includes a display screen; and A method comprising the step of determining whether to display information about the device of interest (200) based on the distance between the augmented reality device (100) and the device of interest (200).

6. In paragraph 5, The step of determining whether to display information about the device of interest (200) based on the distance between the augmented reality device (100) and the device of interest (200) is as follows: A method comprising a step of determining to display information about the device of interest (200) based on a distance between the augmented reality device (100) and the device of interest (200) being greater than a preset threshold value.

7. In any one of paragraphs 1 to 6, The above device of interest (200) is a method in which at least one of the devices connected to the augmented reality device (100) in which an event has occurred, a device selected by the user among the devices connected to the augmented reality device (100), or a device newly connected to the augmented reality device (100) is included.

8. In any one of paragraphs 1 to 6, The step of identifying the above device of interest (200) is: A step of identifying the device the user is gazing at using a gaze tracking sensor; A step of identifying whether the device being watched by the user is connected to the augmented reality device (100); and A method comprising the step of determining a device being watched by the user as the device of interest (200) based on identifying that the device being watched by the user is connected to the augmented reality device (100).

9. In paragraph 8, The step of identifying the device that the user is looking at through the gaze tracking sensor is as follows: A step of obtaining the user's gaze direction through the above gaze tracking sensor; A step of acquiring a real-world scene image through a camera (120); A step of identifying at least one device from the above real scene image; and A method comprising the step of determining a device being looked at by the user based on a position of the at least one device within the real scene image and a direction in which the user is looking.

10. In any one of paragraphs 1 to 9, The step of displaying an augmented reality image including information about the above device of interest (200) is as follows: A step of acquiring a real-world scene image through a camera (120); A step of identifying the device of interest (200) from the above real scene image; A step of determining an area in which to display the augmented reality image based on the location of the device of interest (200); and A method comprising the step of displaying the augmented reality image in the determined area.

11. In an augmented reality device (100) that displays information about a connected device, Display section (162); A memory (150) storing a program including at least one instruction; and comprising at least one processor (140), The at least one processor (140) executes the at least one instruction, Identify the device of interest (200), Determine whether to display information about the above device of interest (200), An augmented reality device (100) that displays an augmented reality image corresponding to information about the device of interest (200) through the display unit (162).

12. In paragraph 11, The at least one processor (140) executes the at least one instruction, Receive device identification information and notification information from the above device of interest (200), Based on the above device identification information, identify whether the device of interest (200) includes a display screen, An augmented reality device (100) that determines to display the notification information based on identifying that the above device of interest (200) does not include a display screen.

13. In paragraph 11 or 12, The at least one processor (140) executes the at least one instruction, An augmented reality device (100) that determines whether to display information about the device of interest (200) based on a facing direction of the display screen, based on identifying that the device of interest (200) includes a display screen.

14. In any one of paragraphs 11 to 13, The at least one processor (140) executes the at least one instruction, Based on identifying that the device of interest (200) includes a display screen, the distance between the augmented reality device (100) and the device of interest (200) is measured, An augmented reality device (100) that determines whether to display information about a device of interest (200) based on the distance between the augmented reality device (100) and the device of interest (200).

15. In any one of paragraphs 11 to 14, The at least one processor (140) executes the at least one instruction, Identify the device the user is gazing at through a gaze tracking sensor, Identify whether the device being watched by the user is connected to the augmented reality device (100), An augmented reality device (100) that determines a device being watched by the user as a device of interest (200) based on identifying that the device being watched by the user is connected to the augmented reality device.

Citation Information

Patent Citations

  • Method for displaying augmented reality information and mobile terminal using this method

    KR1020120016544A

  • Portable device for displaying augmented reality image and method therefor

    KR1020150025115A

  • Electrode cell inspection apparatus and method

    KR1020220139225A

  • Geogrid comprising Pressure Sensor with excellent Durability and Response Speed

    KR1020240170643A

  • Augmented reality system and method with dynamic representation technique of augmented images

    US10665036B1