Augmented reality device for outputting login information and operating method thereof
The augmented reality device addresses login convenience and security issues by capturing website images, extracting information, and outputting credentials, enabling secure access across devices, overcoming limitations of existing simple login methods.
Patent Information
- Application Number
- PCT/KR2024/006285
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-25
- Filing Date
- 2024-05-09
- Publication Date
- 2024-11-28
AI Technical Summary
Users face inconvenience when trying to log in to websites due to forgotten or misunderstood login information, and existing simple login functions like Samsung Pass are limited to mobile devices and pose security concerns, making it difficult to use on personal PCs or other devices.
An augmented reality device that captures images of websites using a camera, extracts website information, compares it with pre-registered data, and outputs login credentials, allowing users to log in securely and conveniently across various devices.
The augmented reality device enhances user convenience by automatically providing login information, ensuring secure access to websites without exposing personal data, even on non-mobile devices, thus addressing the limitations of existing simple login methods.
Smart Images

Figure KR2024006285_28112024_PF_FP_ABST
Abstract
Description
Augmented reality device for outputting login information and method of operation thereof
[0001] The present disclosure relates to an augmented reality device that outputs login information and a method for operating the same. Specifically, the present disclosure discloses an augmented reality device that displays login information, including an ID and password for logging into a website accessed by an external device, to a user wearing the augmented reality device, and a method for operating the same.
[0002] Today, users sign up for and access countless websites. To access these websites, they must log in using login information, such as a user ID and password. However, because login information varies across numerous websites, users often forget their login information or misunderstand it, leading to inconvenient situations where they are unable to log in.
[0003] To prevent such inconveniences, simple login features like Samsung Pass, which allow users to log in using biometric authentication (such as fingerprints or irises) instead of manually entering login information like ID and password, are becoming widespread and widely used. However, the simple login feature has the drawback that it cannot be used when logging in from a personal computer or other PC, not a mobile device. Many users also avoid simple login features (such as Samsung Pass) or password auto-fill due to the psychological burden of privacy threats.
[0004] Augmented Reality (AR) is a technology that overlays virtual images onto the physical environment or real world objects of the real world. AR devices (e.g., smart glasses) that utilize AR technology are being used in everyday life for purposes such as information retrieval, route guidance, and camera shooting. In particular, smart glasses are worn as fashion items and are mainly used for outdoor activities.
[0005] One aspect of the present disclosure provides a method for an augmented reality device to provide login information. A method for operating an augmented reality device according to an embodiment of the present disclosure may include executing an AR login mode for logging into a website accessed and displayed by an external device. The method for operating an augmented reality device according to an embodiment of the present disclosure may include extracting website information, including a website name and Uniform Resource Locators (URLs), from an image captured by a camera while the AR login mode is being executed. The method for operating an augmented reality device according to an embodiment of the present disclosure may include comparing the extracted website information with website information of a registered website that is pre-registered and stored in a database of a memory, and identifying a registered website having website information that matches the extracted website information based on the comparison result. The method for operating an augmented reality device according to an embodiment of the present disclosure may include obtaining login information, including an ID and password information of the identified registered website, from a database. The method for operating an augmented reality device according to an embodiment of the present disclosure may include outputting the obtained login information.
[0006] Another aspect of the present disclosure provides an augmented reality device that provides login information. According to one embodiment of the present disclosure, the augmented reality device may include a camera that captures an image by photographing an object in a real space; a display unit including a waveguide and an optical engine; a memory (150) including a database storing at least one instruction and login information for a registered website; and at least one processor (140). The at least one processor may execute the at least one instruction to execute an AR login mode, and, while the AR login mode is executed, may capture an image by photographing a website displayed by an external device through the camera. The at least one processor may extract website information, including the website name and Uniform Resource Locators (URLs) of the website, from the acquired image by executing the at least one instruction. The at least one processor may compare the extracted website information with website information of a registered website previously stored in a database by executing the at least one instruction, and, based on the comparison result, identify a registered website having website information that matches the extracted website information. The at least one processor may obtain login information including the ID and password information of the identified registration website from a database by executing the at least one command. The at least one processor may output the obtained login information through a display unit by executing the at least one command.
[0007] Another aspect of the present disclosure provides a method for a mobile device to automatically log in to a website. In one embodiment of the present disclosure, the method may include receiving a plurality of image frames captured by a camera of an augmented reality device from a paired augmented reality device via short-range wireless communication. The method may include extracting website information, including a site name and URL of a website to be logged into, from the received plurality of image frames. The method may include comparing the extracted website information with registered website information stored in a cloud server. The method may include identifying a registered website matching the extracted website information based on the comparison result and obtaining login information, including an ID and password for the identified registered website. The method may include parsing the obtained login information to extract the ID and password. The method may include entering the extracted ID and password into the website to perform automatic login.
[0008] The present disclosure can be readily understood by the combination of the following detailed description and the accompanying drawings, wherein reference numerals refer to structural elements.
[0009] FIG. 1 is a conceptual diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to output login information for logging into a website displayed by an external device.
[0010] FIG. 2 is a flowchart illustrating a method of operating an augmented reality device according to one embodiment of the present disclosure.
[0011] FIG. 3 is a block diagram illustrating components of an augmented reality device according to one embodiment of the present disclosure.
[0012] FIG. 4A is a diagram illustrating an operation of an augmented reality device entering an augmented reality login mode (AR login mode) based on a user input according to one embodiment of the present disclosure.
[0013] FIG. 4b is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure entering an augmented reality login mode (AR login mode) based on a user's voice input.
[0014] FIG. 4c is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure entering an augmented reality login mode (AR login mode) based on a user's gaze information.
[0015] FIG. 4D is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure entering an augmented reality login mode (AR login mode) based on a hand pointing input.
[0016] FIG. 5 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to extract website information and obtain login information of a registered website matching the extracted website information.
[0017] FIG. 6 is a flowchart illustrating a method for registering login information for a new website based on the identification result of a registered website matching the extracted website information by an augmented reality device according to one embodiment of the present disclosure.
[0018] FIG. 7 is a flowchart illustrating a method for an augmented reality device to register login information for a new website according to one embodiment of the present disclosure.
[0019] FIG. 8 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to manage the storage capacity of a database during a process of registering login information for a new website.
[0020] FIG. 9 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to output login information of a website.
[0021] FIG. 10 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to output login information of a website.
[0022] FIG. 11 is a flowchart illustrating a method for an augmented reality device according to one embodiment of the present disclosure to convert login information into a voice signal and output it.
[0023] FIG. 12 is a diagram illustrating an operation of an augmented reality device according to one embodiment of the present disclosure to output login information through a TWS (True Wireless Stereo) device.
[0024] FIG. 13 is a flowchart illustrating an operation method of an augmented reality device and a mobile device according to one embodiment of the present disclosure.
[0025] FIG. 14 is a diagram illustrating the operation of an augmented reality device, a mobile device, and a cloud server according to one embodiment of the present disclosure.
[0026] FIG. 15 is a flowchart illustrating an operation method of an augmented reality device, a mobile device, and a cloud server according to one embodiment of the present disclosure.
[0027] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.
[0028] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.
[0029] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.
[0030] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.
[0031] In this disclosure, the term "processor" may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits, including at least one processor. In the context of "at least one processor," one or more processors may be configured to individually and / or collectively perform the various functions described herein in a distributed fashion. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform multiple functions. However, these terms encompass, without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor may perform all of the functions. Furthermore, the at least one processor may include a combination of processors that perform various functions of the disclosed functions in a distributed manner. At least one processor may execute at least one instruction or program code to achieve or perform various functions.
[0032] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.
[0033] In this disclosure, 'Augmented Reality' means displaying a virtual image together within a physical environment space of the real world or displaying a real object and a virtual image together.
[0034] In the present disclosure, an 'augmented reality device' is a device capable of expressing augmented reality, and may be, for example, not only augmented reality glasses in the shape of glasses worn by a user on the face, but also a head-mounted display apparatus (HMD: Head Mounted Display Apparatus) worn on the head, or an augmented reality helmet.
[0035] In the present disclosure, an "external device" refers to a device used by a user wearing an augmented reality device. The external device may be, for example, any one of a PC, a laptop computer, a tablet PC, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a navigation system, an MP3 player, or a camcorder. In one embodiment of the present disclosure, the external device may include a mobile device such as a smartphone.
[0036] In the present disclosure, functions related to 'artificial intelligence' are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.
[0037] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is trained using a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.
[0038] In the present disclosure, the 'artificial intelligence model' may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs neural network operations through operations between the operation results of the previous layer and the plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the plurality of weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network model may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.
[0039] In the present disclosure, "vision recognition" refers to image signal processing that inputs an image into an artificial intelligence model and, through inference using the artificial intelligence model, recognizes (detection) an object from the input image, classifies the object into a specific category, or segments the object. In one embodiment of the present disclosure, vision recognition may refer to image processing that uses an artificial intelligence model to recognize a website login area from an image captured by a camera, or recognizes a user's hand.
[0040] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.
[0041] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.
[0042] FIG. 1 is a conceptual diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to output login information (40) for logging into a website (20) displayed by an external device (200).
[0043] The augmented reality device (100) is a device capable of expressing augmented reality, and may be configured as, for example, augmented reality glasses in the shape of glasses that the user (10) wears on the face. In FIG. 1, the augmented reality device (100) is illustrated as augmented reality glasses, but is not limited thereto. For example, the augmented reality device (100) may be implemented as a head-mounted display apparatus (HMD) worn on the head, an augmented reality helmet, or the like.
[0044] An external device (200) is a device used by a user (10) wearing an augmented reality device (100), which can access a website (20) and display the screen of the website (20). Referring to FIG. 1, the external device (200) is illustrated as a PC, but is not limited thereto. The external device (200) may be, for example, any one of a laptop computer, a tablet PC, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, or a camcorder. In one embodiment of the present disclosure, the external device (200) may include a mobile device such as a smart phone.
[0045] Referring to FIG. 1, the augmented reality device (100) may include a camera (110). In FIG. 1, only the minimum components for explaining the function and / or operation of the augmented reality device (100) are illustrated, and the components included in the augmented reality device (100) are not limited to those illustrated in FIG. 1. The components of the augmented reality device (100) will be described in detail in FIG. 3.
[0046] In the embodiment illustrated in FIG. 1, the augmented reality device (100) executes an augmented reality login mode (AR login mode) (operation ①), captures an image of a website (20) displayed by an external device (200) through a camera (110) (operation ②), extracts website information (30) from the image, identifies a registered website that matches the extracted website information among registered websites previously stored in a registered website database (158) (operation ③), and outputs login information (40) of the identified registered website (operation ④). Hereinafter, the functions and / or operations of the augmented reality device (100) will be described in detail with reference to FIGS. 1 and 2 together.
[0047] FIG. 2 is a flowchart illustrating an operation method of an augmented reality device (100) according to one embodiment of the present disclosure.
[0048] In step S210, the augmented reality device (100) is connected to an external device (200, see FIG. 1) and executes an augmented reality login mode (AR login mode) to log in to a displayed website (20, see FIG. 1). In the present disclosure, the 'augmented reality login mode' means an operation mode of capturing an image of an external device (200) displaying a website (20) using a camera (110, see FIG. 1), extracting information of the website (20) from the image, and outputting login information (40) matching the extracted information of the website (20). Referring also to FIG. 1, the augmented reality device (100) may receive a user input of pressing a button mapped to a function of executing the augmented reality login mode, and execute the augmented reality login mode when a user input of pressing the button is received.
[0049] However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the augmented reality device (100) may receive a voice input (e.g., "Enter the augmented reality login mode~") for executing the augmented reality login mode, and execute the augmented reality login mode based on the received voice input. In one embodiment of the present disclosure, the augmented reality device (100) may recognize a hand pointing input of a user pointing to an icon for executing the augmented reality login mode or pointing to a login area (26) of a website (20) displayed on an external device (200), and execute the augmented reality login mode when the hand pointing input is recognized. In one embodiment of the present disclosure, the augmented reality device (100) may include a gaze tracking sensor (120, see FIG. 3), and may detect a gaze direction of the user using the gaze tracking sensor (120), and may execute the augmented reality login mode when a gaze point by the detected gaze direction is located in the login area (26) of the website (20). A specific embodiment in which the augmented reality device (100) executes the augmented reality login mode will be described in detail in FIGS. 4a to 4d.
[0050] In step S220 of FIG. 2, as the augmented reality login mode is executed, the augmented reality device (100) extracts website information (30) including a site name and URL from an image captured by a camera (110, see FIG. 1). Referring also to FIG. 1, the augmented reality device (100) can capture an image of a website (20) accessed and displayed by an external device (200) using the camera (110). The augmented reality device (100) can extract a site name area (22), a URL (Uniform Resource Locators) area (24), and a login area (26) of the website (20) from the captured image. The augmented reality device (100) can perform vision recognition using an artificial intelligence model to detect the site name area, URL area, and login area (26) of the website (20) from the image. In one embodiment of the present disclosure, the artificial intelligence model may be a deep neural network model trained to recognize the shapes of the site name area, URL area, and login area from an image through a supervised learning method that applies an image as input and applies the shapes of a standardized site name area, URL area, and login area as ground truth. The deep neural network model may be, for example, a convolutional neural network (CNN) model, but is not limited thereto.
[0051] The augmented reality device (100) can extract website information (30, see FIG. 1) including a site name and a URL (Uniform Resource Locator) from the detected site name area and URL area. In one embodiment of the present disclosure, the augmented reality device (100) can perform STR (Scene Text Recognition) to extract text representing the site name from an image included in the detected site name area (22). In one embodiment of the present disclosure, the augmented reality device (100) can perform OCR (Optical Character Recognition) to extract text representing the URL from an image included in the URL area (24). A specific embodiment in which the augmented reality device (100) extracts website information (30) from an image captured by the camera (110) will be described in detail with reference to FIG. 5.
[0052] In step S230 of FIG. 2, the augmented reality device (100) compares the extracted website information (30, see FIG. 1) with the website information of registered websites previously stored in the database to identify a registered website that matches the extracted website information. In the present disclosure, the term "registered website" refers to a website that has been registered by pre-storing login information including an ID and password for logging into the website by user input in the database. Referring also to FIG. 1, the augmented reality device (100) can compare the extracted website information (30) with the website information of registered websites previously stored in the registered website database (158), and identify a registered website that has information that matches the extracted website information. In one embodiment of the present disclosure, the augmented reality device (100) can perform a string search to compare the extracted website information with the website information of registered websites stored in the registered website database (158). The augmented reality device (100) can identify a registered website having matching website information based on the comparison result.
[0053] In step S240 of FIG. 2, the augmented reality device (100) obtains login information (40) including the ID and password of the identified registration website. Referring also to FIG. 1, the augmented reality device (100) can obtain the ID 'gilee' and password information 'XXXX201@' of the identified registration website.
[0054] In step S250 of FIG. 2, the augmented reality device (100) outputs the acquired login information (40). Referring to the embodiment illustrated in FIG. 1, the augmented reality device (100) may overlay and display the login information (40) on the login area (22) of the website (20) displayed by the external device (200) by projecting at least one of letters, numbers, special symbols, and combinations thereof representing the ID and password included in the login information (40) onto a waveguide. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the augmented reality device (100) may also overlay and display the login information (40) on an arbitrary area of a real object viewed by the user. In one embodiment of the present disclosure, the augmented reality device (100) may perform TTS (Text-to-Speech) to convert the ID and password in text form into a voice signal, and output the converted voice signal through a speaker. In one embodiment of the present disclosure, the augmented reality device (100) may output a voice signal indicating login information (40) through a paired TWS (True Wireless Stereo) device. A specific embodiment in which the augmented reality device (100) outputs login information (40) will be described in detail with reference to FIGS. 9 to 12.
[0055] In FIGS. 1 and 2, the external device (200) is illustrated and described as displaying a screen of a website (20), and the augmented reality device (100) outputs login information (40) for logging into the website (20), but is not limited thereto. In the present disclosure, the external device (200) may display an execution screen of an Internet banking, public certificate, or application, and the augmented reality device (100) may output login information for logging into the Internet banking, public certificate, or application in the same manner as described above.
[0056] Today, users (10) sign up for and use numerous websites, internet banking, public certificates, applications, etc. However, when users (10) attempt to log in to websites, internet banking, public certificates, or applications, they may fail to log in due to forgetting or incorrect login information, which can cause inconvenience. To prevent such inconveniences, simple login features such as Samsung Pass, which support logging in through biometric authentication (e.g., fingerprints or irises) instead of directly entering login information such as ID and password, are being popularized and used. However, the simple login feature has the limitation that it cannot be used when logging in from a personal PC or other PC, not a mobile device. In addition, web browsers such as Chrome and Edge provide an auto-complete password storage feature, but it can be difficult to confirm accurate login information at times when complete and accurate login information is required, such as when reinstalling the operating system, using a new PC, or using a new mobile device. Many users do not use simple login features (e.g., Samsung Pass) or password auto-fill features due to the psychological burden of personal information security threats.
[0057] The purpose of the present disclosure is to provide an augmented reality device (100) and its operation method to automatically output login information when accessing a website, internet banking, public certificate, or application through an external device (200), thereby eliminating inconvenience in daily life and improving user convenience.
[0058] The augmented reality device (100) according to the embodiment illustrated in FIGS. 1 and 2 extracts website information (30) including the site name and URL of a website (20), identifies a registered website matching the extracted website information (30), and outputs login information (40) of the identified registered website, thereby preventing inconveniences in daily life, such as login failure due to not remembering login information, and improving user convenience. In addition, the present disclosure provides a technical effect that personal information can be prevented from being exposed to the outside because login information (40) is displayed only to a user (10) wearing the augmented reality device (100), and thus provides a high level of security. The function and / or operating method of the augmented reality device (100) according to one embodiment of the present disclosure is not limited to outputting login information of a website (20), Internet banking, a public certificate, or an application, and can be utilized to output information that the user (10) is curious about or needs to remember, thereby providing a technical effect that can provide information simply and quickly.
[0059] FIG. 3 is a block diagram illustrating components of an augmented reality device (100) according to one embodiment of the present disclosure.
[0060] Referring to FIG. 3, the augmented reality device (100) may include a camera (110), an eye tracking sensor (120), a microphone (130), a processor (140), a memory (150), a communication interface (160), a display unit (170), and a speaker (180). The camera (110), the eye tracking sensor (120), the microphone (130), the processor (140), the memory (150), the communication interface (160), the display unit (170), and the speaker (180) may each be electrically and / or physically connected to each other. Only components for explaining the operation of the augmented reality device (100) are illustrated in FIG. 3, and the components included in the augmented reality device (100) are not limited to those illustrated in FIG. 3. In one embodiment of the present disclosure, the augmented reality device (100) may further include an IMU sensor (Inertial Measurement Unit). In one embodiment of the present disclosure, the augmented reality device (100) is implemented as a portable device, and in this case, the augmented reality device (100) may further include a battery that supplies driving power to a camera (110), an eye tracking sensor (120), a microphone (130), a processor (140), a communication interface (160), a display unit (170), and a speaker (180).
[0061] In one embodiment of the present disclosure, the augmented reality device (100) may not include a microphone (130) or a speaker (180).
[0062] The camera (110) is configured to acquire an image of an object by photographing the object in a real space. The camera (110) may include a lens module, an image sensor, and an image processing module. The camera (110) and the second camera (114) may acquire a still image or video of the object by an image sensor (e.g., CMOS or CCD). The video may include a plurality of image frames continuously acquired by photographing the object through the camera (110). The image processing module may encode a still image composed of a single image frame acquired through the image sensor or video data composed of a plurality of image frames and transmit the encoded data to the processor (140).
[0063] In one embodiment of the present disclosure, a camera (110) can capture an external device (200, see FIG. 1) located in a real space. The external device (200) displays an execution screen of a website, Internet banking, public certificate, or application, and the camera (110) can capture the screen displayed by the external device (200) to acquire an image. The camera (110) can provide image data of the captured image to the processor (140).
[0064] The gaze tracking sensor (120) is a device configured to detect the gaze direction of a user's eyes and track the gaze direction. The gaze tracking sensor (120) can detect the gaze direction of the user by detecting an image of the human eye or pupil, or by detecting the direction or amount of reflected light such as near-infrared light reflected from the cornea. In one embodiment of the present disclosure, the gaze tracking sensor (120) includes a left-eye gaze tracking sensor and a right-eye gaze tracking sensor, and can detect the gaze direction of the user's left eye and the gaze direction of the user's right eye, respectively. Detecting the gaze direction of the user may include an operation of obtaining gaze information related to the user's gaze. In one embodiment of the present disclosure, the gaze tracking sensor (120) can detect a gaze point where the gaze direction of the user's left eye detected by the left-eye gaze tracking sensor and the gaze direction of the user's right eye detected by the right-eye gaze tracking sensor converge, and obtain three-dimensional position coordinate value information of the gaze point.
[0065] A microphone (130) is a device configured to receive sound from a real space or an external device and convert the received sound into an acoustic signal. In one embodiment of the present disclosure, the microphone (130) can receive a user's voice and convert the received voice into an acoustic signal.
[0066] The processor (140) can execute one or more instructions of a program stored in the memory (150). The processor (140) may be configured with hardware components that perform arithmetic, logic, and input / output operations and image processing. Although the processor (140) is illustrated as a single element in FIG. 3, it is not limited thereto. In one embodiment of the present disclosure, the processor (140) may be configured with one or more multiple elements. One or more processors included in the processor (140) may be circuitry such as a system on chip (SoC), an integrated circuit (IC), etc. The processor (140) may be a general-purpose processor such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), a graphics-only processor such as a graphics processing unit (GPU), a vision processing unit (VPU), or an artificial intelligence-only processor such as a neural processing unit (NPU). The processor (140) can be controlled to process input data according to a predefined operating rule or artificial intelligence model by executing at least one instruction or program code stored in the memory (150). Alternatively, if the processor (140) is an artificial intelligence-only processor, the artificial intelligence-only processor can be designed with a hardware structure specialized for processing a specific artificial intelligence model.
[0067] The memory (150) may be configured as at least one type of storage medium, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk.
[0068] The memory (150) may store instructions related to the functions and / or operations of the augmented reality device (100) acquiring login information of a website displayed by an external device and outputting the login information. In one embodiment of the present disclosure, the memory (150) may store at least one of instructions, an algorithm, a data structure, a program code, and an application program that can be read by the processor (140). The instructions, algorithms, data structures, and program codes stored in the memory (150) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.
[0069] In the following embodiments, the processor (140) may be implemented by executing instructions or program codes stored in the memory (150).
[0070] The processor (140) may execute an augmented reality login mode (AR login mode) based on a user input. In one embodiment of the present disclosure, the processor (140) may execute the AR login mode upon receiving a user input of pressing a button mapped to a function of executing the AR login mode among the buttons of the AR device (100). In one embodiment of the present disclosure, the microphone (130) receives a voice input for executing the AR login mode, for example, a voice input such as “execute the AR login mode~”, and the processor (140) may convert the voice input received through the microphone (130) into text and execute the AR login mode based on the converted text. In one embodiment of the present disclosure, the camera (110) may capture an image of a user’s hand, and the processor (140) may recognize an area pointed by the user’s hand from the image, and execute the AR login mode if the recognized area is located in a login area of a website. In one embodiment of the present disclosure, an icon for executing an augmented reality login mode is displayed through the display unit (170), and when a user's hand points to the icon, the processor (140) can execute the augmented reality login mode. In one embodiment of the present disclosure, the gaze tracking sensor (120) detects the gaze direction of the user's left eye and the gaze direction of the right eye, detects the position of a gaze point where the gaze direction of the left eye and the gaze direction of the right eye converge, and the processor (140) can execute the augmented reality login mode when the position of the gaze point is located on a login area of a website.
[0071] As the augmented reality login mode is executed, the processor (140) can control the camera (110) to capture an image by taking a picture of a website displayed by an external device.
[0072] The website information extraction module (152) is configured with commands or program codes related to the function and / or operation of extracting the site name and Uniform Resource Locators (URLs) of a website from an image. The processor (140) may execute the commands or program codes of the website information extraction module (152) to extract website information from an image acquired through the camera (110). In one embodiment of the present disclosure, the website information extraction module (152) may include an artificial intelligence model that performs vision recognition to detect a standardized site name area and URL area from an image. The artificial intelligence model may be a deep neural network model trained to recognize the shape of the site name area and URL area from an image through a supervised learning method that applies an image as an input and applies the shape of the standardized site name area and URL area as a ground truth. The processor (140) may input an image into the deep neural network model and perform inference using the deep neural network model to detect the site name area and URL area from the image. In one embodiment of the present disclosure, the processor (140) may perform STR (Scene Text Recognition) to extract text representing a site name from an image included in a detected site name area. In one embodiment of the present disclosure, the processor (140) may perform OCR (Optical Character Recognition) to extract text representing a URL from an image included in a URL area.
[0073] The registration website identification module (154) is configured with commands or program codes related to a function and / or operation for comparing the extracted website information with the website information of the registration website previously stored in the registration website database (158) to identify the matching registration website. In the present disclosure, the term 'registration website' refers to a website in which login information including an ID and password for logging into the website by user input is pre-stored in the database and registered. The processor (140) executes the commands or program codes of the registration website identification module (154) to compare the website information extracted from the image, i.e., the site name and URL, with the site names and URLs of the registration websites pre-stored in the registration website database (158), and to identify the registration website having information matching the extracted website information. In one embodiment of the present disclosure, the processor (140) performs a string search to compare text included in the extracted website information with text of a registered website stored in a registered website database (158), and can identify a registered website having website information that matches the comparison result.
[0074] If a matching registration website is identified as a result of the comparison, the processor (140) can obtain login information including the ID and password of the identified registration website from the registration website database (158).
[0075] If a matching registered website is not identified as a result of the comparison, the processor (140) may perform a registration step for a new website. The new website registration module (156) is configured with commands or program codes related to the function and / or operation of registering and storing login information of an unregistered website that is not registered in the registration website database (158). The processor (140) may execute the commands or program codes of the new website registration module (156) to store an ID and password for logging into the new website together with the website information of the new website in the registration website database (158). In one embodiment of the present disclosure, the processor (140) may control the camera (110) to capture a login screen of the new website displayed by an external device, thereby obtaining a plurality of image frames, and extract the ID and password of the new website from the obtained plurality of image frames. The processor (140) may store the extracted ID and password of the new website together with the site name and URL of the new website in the registration website database (158). A specific embodiment in which the processor (140) registers login information for a new website will be described in detail in FIGS. 7 and 8.
[0076] In one embodiment of the present disclosure, the processor (140) may capture a login process on a new website, acquire a plurality of image frames, and store them in a temporary buffer within the memory (150). If the login process on the new website fails, the processor (140) may delete the plurality of image frames stored in the temporary buffer. The processor (140) may store the plurality of image frames in a storage space within the memory (150) only if the login process on the new website is successful. A specific embodiment of how the processor (140) manages the storage space of the memory during the registration process on the new website will be described in detail with reference to FIG. 8.
[0077] The registration website database (158) is a database that stores login information for registered websites, such as ID and password information. In one embodiment of the present disclosure, the registration website database (158) may store login information in text format. The login information stored in the registration website database (158) may be stored along with the site name and URL of the relevant website.
[0078] The registration website database (158) may be composed of non-volatile memory. Non-volatile memory refers to a storage medium that stores and maintains information even when power is not supplied, and can use the stored information again when power is supplied. The non-volatile memory may be composed of at least one of, for example, flash memory, a hard disk, a solid state drive (SSD), a multimedia card micro type, an external memory of a card type (e.g., SD or XD memory, etc.), a read only memory (ROM), a magnetic disk, or an optical disk. In FIG. 3, the registration website database (158) is illustrated as being included in the memory (150), but the present disclosure is not limited to the example illustrated in the drawing. In one embodiment of the present disclosure, the registration website database (158) may not be included in the memory (150) and may be implemented as a separate component.
[0079] However, this is not limited thereto, and in one embodiment of the present disclosure, the registration website database (158) may be stored in a web storage or cloud server that is accessible via a network and performs a storage function. In this case, the augmented reality device (100) may communicate with the web storage or cloud server through a communication interface (160) and perform data transmission and reception. The processor (140) may receive login information for the registration website from the web storage or cloud server through the communication interface (160).
[0080] The processor (140) can output login information through the display unit (170). In one embodiment of the present disclosure, the processor (140) can display the login information by overlaying it on the login area of a website displayed by an external device by projecting at least one of letters, numbers, special symbols, and combinations thereof representing an ID and password included in the login information onto the waveguide of the display unit (170). However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the processor (140) can also display the login information on any area of a real object viewed by a user through the augmented reality device (100).
[0081] In one embodiment of the present disclosure, the memory (150) further includes a Text-to-Speech (TTS) module, and the processor (140) can convert text-based login information into a voice signal by executing commands or program codes included in the TTS module. The processor (140) can control the speaker (180) to output the converted voice signal.
[0082] The communication interface (160) is a device configured to perform data communication with an external device or a cloud server. In one embodiment of the present disclosure, the communication interface (160) may include a short-range wireless communication module and a mobile communication module.
[0083] A short-range wireless communication module is configured to connect to an external device using a wireless communication network and perform data transmission and reception. The short-range wireless communication module may be configured as at least one hardware module of, for example, a WiFi, a Wi-Fi Direct communication unit, a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, an NFC (Near Field Communication unit), a Zigbee communication unit, an Ant+ communication unit, or a μWave communication unit, but is not limited thereto. In one embodiment of the present disclosure, the communication interface (160) may be paired with an external device (e.g., a mobile device) via Bluetooth communication and may perform data transmission and reception with the external device using Bluetooth communication. However, it is not limited thereto, and the communication interface (160) can perform data transmission and reception with an external device using at least one short-range wireless communication network among WiFi, Wi-Fi Direct, BLE (Bluetooth Low Energy), NFC (Near Field Communication unit), Zigbee, Ant+, or μWave.
[0084] A mobile communication module is a communication module configured to transmit and receive wireless signals with at least one of a base station, an external device, or a server on a mobile communication network. The mobile communication module may transmit and receive data using, for example, at least one of 5G mmWave communication, 5G Sub 6 communication, LTE (Long Term Evolution) communication, or 3G mobile communication. In one embodiment of the present disclosure, the mobile communication module may transmit and receive data with a server under the control of a processor (140).
[0085] The display unit (170) is configured to display login information. When the augmented reality device (100) is configured as augmented reality glasses, the display unit (170) may be configured as a lens optical system and may include a waveguide and an optical engine. The optical engine may be configured as a projector that generates light of a graphical UI composed of text, icons, virtual images, etc. and projects the light onto a waveguide. The optical engine may include, for example, an image panel, an illumination optical system, a projection optical system, etc. In one embodiment of the present disclosure, the optical engine may be disposed on the frame or temples of the augmented reality glasses. In one embodiment of the present disclosure, the optical engine may project a virtual UI including at least one of letters, numbers, special symbols, and combinations thereof representing an ID and password included in the login information onto the waveguide under the control of the processor (140), thereby overlaying and displaying the login information on a login area of a website displayed by an external device.
[0086] However, it is not limited thereto, and the display unit (170) may be configured as at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display.
[0087] The speaker (180) is configured to output an audio signal. In one embodiment of the present disclosure, the speaker (180) can output a voice message obtained by converting login information into a voice signal under the control of the processor (140).
[0088] FIG. 4A is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure entering an augmented reality login mode (AR login mode) based on a user input.
[0089] Referring to FIG. 4A, the augmented reality device (100) may include a button (102). If the augmented reality device (100) is implemented as an augmented reality glass, the button (102) may be a hardware button that protrudes outward and is placed on the temple of the augmented reality glasses. However, the present invention is not limited thereto, and the button (102) may also be implemented as a graphical user interface (UI) displayed on a touchscreen to receive touch input. In this case, the touchscreen may be formed on the temple of the glasses.
[0090] The button (102) may be pre-mapped to a function that executes an augmented reality login mode. The augmented reality device (100) may receive a user input of pressing the button (102) and execute the augmented reality login mode upon receiving the user input.
[0091] FIG. 4b is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure entering an augmented reality login mode (AR login mode) based on a voice input of a user (10).
[0092] Referring to FIG. 4b, a user (10) may make an utterance such as “Please activate AR login mode~”, and the augmented reality device (100) may receive the user’s utterance through a microphone (130, see FIG. 3) and convert it into an acoustic signal. The memory (150, see FIG. 3) of the augmented reality device (100) may store an ASR (Automatic Speech Recognition) model and an NLU (Natural Language Understanding) model, and the processor (140, see FIG. 3) may execute commands or program codes of the ASR model to convert the acoustic signal acquired through the microphone (130) into text. The processor (140) may recognize the user’s intent from the text by executing commands or program codes of the NLU model. The processor (140) may execute the augmented reality login mode based on the recognized intent.
[0093] FIG. 4c is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure entering an augmented reality login mode (AR login mode) based on a user's gaze information.
[0094] Referring to FIG. 4C, the augmented reality device (100) may include a gaze tracking sensor (120L, 120R). The gaze tracking sensor (120L, 120R) may detect a user's gaze direction by detecting an image of a human eye or pupil, or by detecting a direction or amount of reflected light such as near-infrared light reflected from the cornea. In one embodiment of the present disclosure, the gaze tracking sensor (120L, 120R) may include a left-eye gaze tracking sensor (120L) that detects a gaze direction of the user's left eye, and a right-eye gaze tracking sensor (120R) that detects a gaze direction of the user's right eye. The augmented reality device (100) may detect a gaze point (Gaze Point) where the gaze direction of the user's left eye detected by the left-eye gaze tracking sensor (120L) and the gaze direction of the user's right eye detected by the right-eye gaze tracking sensor (120R) converge, and may obtain position information of the gaze point (G). The augmented reality device (100) can execute the augmented reality login mode when the position of the acquired gaze point (G) is located on the login area (410) of the website (400) displayed by the external device (200). In one embodiment of the present disclosure, the augmented reality device (100) can execute the augmented reality login mode when the gaze point (G) remains on the login area (410) for a preset period of time (e.g., 10 seconds) or longer.
[0095] FIG. 4d is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure entering an augmented reality login mode (AR login mode) based on a hand pointing input.
[0096] Referring to FIG. 4D, the augmented reality device (100) can acquire an image by taking a picture of a website (400) displayed by an external device (200) using a camera (110). The augmented reality device (100) can perform vision recognition using an artificial intelligence model to recognize a user's hand from the image and identify the location of the area pointed by the recognized hand. In the embodiment illustrated in FIG. 4D, the augmented reality device (100) can execute an augmented reality login mode when the user's hand points to the login area (410) of the website (400) displayed by the external device (200).
[0097] Although not illustrated in FIG. 4d, the augmented reality device (100) can project and display a virtual UI (e.g., an icon) for executing an augmented reality login mode on the waveguide. The augmented reality device (100) can recognize a user's hand pointing to the virtual UI using an artificial intelligence model and execute the augmented reality login mode based on the recognition result.
[0098] In the embodiments illustrated in FIGS. 4A to 4D, when the augmented reality login mode is executed, the augmented reality device (100) may capture an image by taking a picture of a screen of a website displayed by an external device (200) using a camera (110, see FIG. 3). In one embodiment of the present disclosure, the augmented reality device (100) may also capture a video including a plurality of image frames that are sequential over time using the camera (110).
[0099] FIG. 5 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure extracting website information of a registration website (540) and obtaining login information (560) of the registration website that matches the extracted website information.
[0100] Referring to FIG. 5, a screen of a website (500) may be displayed on an external device (200). The augmented reality device (100) may capture an image of the website (500) displayed by the external device (200) through a camera (110, see FIG. 3) when the augmented reality login mode is executed, and may detect a site name area (510) and a URL area (520) from the acquired image (Operation ①). In one embodiment of the present disclosure, the augmented reality device (100) may include an artificial intelligence model that performs vision recognition to detect a standardized site name area (510) and a URL area (520) from an image. The artificial intelligence model may be a deep neural network model trained to recognize the shape of the site name area (510) and the URL area (520) from an image through a supervised learning method that applies an image as an input and applies the shape of the standardized site name area and the URL area as a ground truth. The deep neural network model may be, for example, a convolutional neural network (CNN) model, but is not limited thereto. The deep neural network model may be implemented as, for example, at least one of a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and deep Q-networks. The processor (140, see FIG. 3) of the augmented reality device (100) inputs an image into the deep neural network model and performs inference using the deep neural network model to detect a site name area (510) and a URL area (520) from the image.
[0101] The processor (140) of the augmented reality device (100) can perform STR (Scene Text Recognition) to extract text representing the site name from an image included in the detected site name area (510) (Operation ②). In the embodiment illustrated in FIG. 5, the processor (140) can perform STR to extract the text 'SAMSUNG', which is the site name of the website (500).
[0102] The processor (140) can perform OCR (Optical Character Recognition) to extract text representing a URL from an image included in the URL area (520) (Operation ③). In the embodiment illustrated in FIG. 5, the processor (140) can perform OCR to extract the text 'www.samsung.com', which is the URL of the website (500). The processor (140) can obtain website information (530) including the site name and URL of the website (500).
[0103] The augmented reality device (100) can compare the extracted website information (530) with the website information of a registered website (540) previously stored in the memory (150, see FIG. 3) to search for a matching registered website (operation ④). In the present disclosure, the 'registration website (540)' refers to a website in which login information, including an ID and password for logging into the website by user input, is pre-stored in a database and registered. For example, the registration website (540) may include at least one of Naver, Woori Bank, and Samsung, but is not limited thereto. In one embodiment of the present disclosure, the registration website (540) may be stored in the registration website database (158, see FIG. 3) of the memory (150). The processor (140) of the augmented reality device (100) can search for a registered website that includes website information that matches the extracted website information (530) by performing a string search and comparing the text constituting the site name and URL included in the extracted website information (530) with the text constituting the registered website (540). In the embodiment illustrated in FIG. 5, the processor (140) can identify 'Samsung' as the registered website that matches the extracted website information (530) as a result of the string search.
[0104] The augmented reality device (100) can obtain login information (550) of a matching registration website (operation ⑤). The processor (140) of the augmented reality device (100) can obtain login information (550) including an ID (e.g., 'joy1234') and a password (e.g., 'friends88S') of a registration website (e.g., 'Samsung') searched from a registration website database (158).
[0105] FIG. 6 is a flowchart illustrating a method for registering login information for a new website based on the identification result of a registered website matching the extracted website information by an augmented reality device (100) according to one embodiment of the present disclosure.
[0106] Steps S610 and S620 illustrated in FIG. 6 are steps that specify the operation of step S230 illustrated in FIG. 2. After step S620 is performed, step S240 illustrated in FIG. 2 may be performed.
[0107] In step S610, the augmented reality device (100) compares the extracted website information with the website information of the registered website previously stored in the database. Referring to the embodiment illustrated in FIG. 5, the processor (140, see FIG. 3) of the augmented reality device (100) may compare the extracted website information (530, see FIG. 5) with the website information of the registered website (540, see FIG. 5) previously stored in the memory (150, see FIG. 3) to search for a matching registered website. In one embodiment of the present disclosure, the processor (140) may perform a string search to compare the text constituting the extracted website information (530) with the text constituting the website information of the registered websites (540) previously stored in the registered website database (158, see FIG. 3).
[0108] In step S620, the augmented reality device (100) determines whether a registered website matching the extracted website information is identified. Referring to the embodiment illustrated in FIG. 5, the processor (140) of the augmented reality device (100) can determine whether a registered website having website information matching the extracted website information (530) is identified as a result of a string search of the registered website database (158).
[0109] When a matching registration website is identified (step S240), the augmented reality device (100) obtains login information including the ID and password of the identified registration website.
[0110] If a matching registration website is not identified (step S630), the augmented reality device (100) outputs a pop-up message or a TTS voice message asking whether to register a new website. In one embodiment of the present disclosure, the augmented reality device (100) may display a pop-up message asking the user whether to register login information for the new website. The pop-up message may be composed of the text, for example, "Would you like to register login information for the new website?" In one embodiment of the present disclosure, the augmented reality device (100) may convert the text asking the user whether to register login information for the new website into a voice message through a TTS (Text-to-Speech) module, and output the resulting voice message through a speaker (180, see FIG. 3).
[0111] In step S640, the augmented reality device (100) determines whether to register a new website. In one embodiment of the present disclosure, the augmented reality device (100) may receive a user's hand pointing input or voice input, and determine whether to register login information for the new website based on the received hand pointing input or voice input.
[0112] When a hand pointing input indicating a virtual image (graphic UI) indicating 'confirmation' from a user is received or a voice input (e.g., "yes") is received (step S650), the augmented reality device (100) registers login information for a new website.
[0113] If a hand pointing input indicating a virtual image (graphical UI) indicating "rejection" is received from the user, or a voice input (e.g., "no") is received, the new website login information is not registered and the application is terminated. If no input is received from the user, the new website login information is not registered and the application is terminated.
[0114] The step (S650) of registering login information for a new website is described in detail in FIGS. 7 and 8.
[0115] FIG. 7 is a flowchart illustrating a method for registering login information for a new website by an augmented reality device (100) according to one embodiment of the present disclosure. Steps S710 to S760 illustrated in FIG. 7 are steps that embody the operation of step S650 illustrated in FIG. 6.
[0116] FIG. 8 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to manage the storage capacity of a database (158) during a process of registering login information for a new website.
[0117] Hereinafter, an embodiment in which an augmented reality device (100) registers login information for a new website will be described with reference to FIGS. 7 and 8 together.
[0118] In step S710 of FIG. 7, the augmented reality device (100) captures a login screen to obtain a plurality of image frames as the gaze point or hand pointing area according to the user's gaze information is located on the login area of the new website. In one embodiment of the present disclosure, the augmented reality device (100) can detect the gaze direction of the user's left eye and the gaze direction of the right eye using a gaze tracking sensor (120, see FIG. 3), and obtain position information of a gaze point (G, see FIG. 4c) where the gaze direction of the left eye and the gaze direction of the right eye converge. The augmented reality device (100) can capture a screen of the website (400) using a camera (110, see FIG. 3) to obtain a plurality of image frames as the position of the gaze point (G) is located on the login area (410, see FIG. 4c) of the website (400, see FIG. 4c) displayed by an external device (200, see FIG. 1 and FIG. 4c). In one embodiment of the present disclosure, the augmented reality device (100) can capture a screen of a website (400) to obtain a plurality of image frames when the position of the gaze point (G) remains on the login area (410) for a preset period of time (e.g., 10 seconds). In one embodiment of the present disclosure, the augmented reality device (100) can recognize a hand pointing input pointing to the login area (410) of the website (400) displayed by an external device (200), and capture a screen of the website (400) using the camera (110) as the hand pointing input is recognized, thereby obtaining a plurality of image frames.
[0119] The augmented reality device (100) can capture the entire screen of a website (400) displayed on an external device (200) within the field of view (FOV) of the camera (110) to obtain multiple image frames. The multiple image frames are image frames obtained by continuously capturing the process of a user entering an ID and password from beginning to end over time, and can constitute a video file.
[0120] In step S720, the augmented reality device (100) stores a plurality of image frames captured by the camera (110) in a temporary buffer. Referring also to FIG. 8, the augmented reality device (100) may store a video file (800) composed of a plurality of image frames in a temporary buffer (temp) within the database (158).
[0121] In step S730, the augmented reality device (100) determines whether the user's login is successful. In one embodiment of the present disclosure, the processor (140, see FIG. 3) of the augmented reality device (100) performs vision recognition using an artificial intelligence model to recognize a login area from a plurality of image frames, and can determine that the login is successful if the status of the login area changes. In one embodiment of the present disclosure, the processor (140) can recognize a screen of a website from a plurality of image frames, and can determine that the login is successful if the entire screen of the recognized website changes or a pop-up window disappears. In one embodiment of the present disclosure, the processor (140) can determine that the login is unsuccessful if there is no change in the login area or the screen of the website does not change.
[0122] If it is determined that the login is successful (step S740), the augmented reality device (100) stores a plurality of image frames in a storage space within the memory. Referring to FIG. 8, the database (158) within the memory (150, see FIG. 3) includes a temporary buffer (temp) and a plurality of repositories (158-1 to 158-n), and if the augmented reality device (100) determines that the user's login is successful, it can move the video file (800) from the temporary buffer (temp) to one of the plurality of repositories (158-1 to 158-n) (in the embodiment illustrated in FIG. 8, the 'second storage (158-2)') and store it.
[0123] In step S750, the augmented reality device (100) extracts login information including an ID and password for a new website from a plurality of image frames. In one embodiment of the present disclosure, the processor (140) of the augmented reality device (100) may perform vision recognition using an artificial intelligence model to detect a login area from the plurality of image frames, and perform OCR (Optical Character Recognition) to recognize text constituting the ID and password entered in the login area. The processor (140) may extract the recognized text as login information.
[0124] In step S760, the augmented reality device (100) stores the extracted login information in a database along with the site name and URL of the new website.
[0125] If it is determined that login has failed in step S730 (step S770), the augmented reality device (100) deletes multiple image frames stored in a temporary buffer (temp, see FIG. 8).
[0126] In general, a video file (800, see FIG. 8) composed of multiple image frames has a large capacity. The augmented reality device (100) according to the embodiment illustrated in FIGS. 7 and 8 stores the video file (800) in the internal memory (150) only when the user successfully logs in to a new website, and deletes the video file (800) temporarily stored in a temporary buffer (temp, see FIG. 8) when the login fails, thereby efficiently managing the capacity of the memory (150) within the augmented reality device (100).
[0127] FIG. 9 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to output login information of a website.
[0128] Referring to FIG. 9, an external device (200) may access a website (900) and display a screen of the website (900), and an augmented reality device (100) may overlay and display login information (920) on a login area (910) of the website (900) displayed by the external device (200). When the augmented reality device (100) is implemented as an augmented reality glass, the augmented reality device (100) may include a lens optical system including a waveguide and an optical engine as components. The optical engine may be configured as a projector that generates light of a graphical UI composed of text, icons, virtual images, etc. and projects the light onto a waveguide. The optical engine may include, for example, an image panel, an illumination optical system, a projection optical system, etc. In one embodiment of the present disclosure, the optical engine may be disposed on the frame or temples of the augmented reality glasses. In one embodiment of the present disclosure, the processor (140, see FIG. 3) of the augmented reality device (100) can overlay and display the login information (920) on the login area (910) of the website (900) displayed by the external device (200) by projecting a virtual UI including at least one of letters, numbers, special symbols, and combinations thereof representing login information (920) onto the waveguide.
[0129] In one embodiment of the present disclosure, the processor (140) of the augmented reality device (100) may display some of the ID and password included in the login information (920) as '*'. The augmented reality device (100) does not display all of the login information (920) but instead displays some of it as '*', thereby providing a technical effect that can prevent personal information leakage even in an exceptional situation where a person other than the user is peeking at the augmented reality device (100).
[0130] FIG. 10 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to output login information (1020) of a website (1000).
[0131] Referring to FIG. 10, an external device (200) can access a website (1000) and display the screen of the website (1000), and an augmented reality device (100) can display login information (1020) of the website (1000) displayed by the external device (200). The augmented reality device (100) can be implemented as augmented reality glasses and include a lens optical system including a waveguide and an optical engine as components. Since the optical engine is the same as that described in FIG. 9, a redundant description will be omitted. In the embodiment illustrated in FIG. 10, the processor (140, see FIG. 3) of the augmented reality device (100) can display login information (1020) on an arbitrary area of a real object (e.g., 'external device (200)') viewed by the user through the augmented reality device (100) by projecting a virtual UI including at least one of letters, numbers, special symbols, and combinations thereof representing login information (1020) onto a waveguide.
[0132] FIG. 11 is a flowchart illustrating a method in which an augmented reality device (100) according to one embodiment of the present disclosure converts login information into a voice signal and outputs it.
[0133] Steps S1110 and S1120 illustrated in FIG. 11 are steps that specify the operation of step S250 illustrated in FIG. 2.
[0134] In step S1110, the augmented reality device (100) performs TTS (Text-to-Speech) to convert login information in text form into a voice signal. In one embodiment of the present disclosure, the memory (150, see FIG. 3) may include a TTS module that converts text into a voice signal. The processor (140, see FIG. 3) of the augmented reality device (100) may convert an ID and password in text form into a voice signal by executing commands or program codes included in the TTS module, thereby obtaining a voice message.
[0135] In step S1120, the augmented reality device (100) outputs the converted voice signal through a speaker. In one embodiment of the present disclosure, the processor (140) of the augmented reality device (100) can output a voice message obtained through conversion using TTS through a speaker (180, see FIG. 3).
[0136] FIG. 12 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to output login information through a TWS device (1200).
[0137] Referring to FIG. 12, the augmented reality device (100) can be wirelessly paired with the TWS device (1200) (Operation ①). The TWS (True Wireless Stereo) device (1200) refers to a completely wireless earphone with no wire between the left receiver and the right receiver. The TWS device (1200) includes a battery in each of the left receiver and the right receiver, and the battery can be charged through a charging case (not shown). The augmented reality device (100) can be paired with the TWS device (1200) through a short-range wireless communication method. For example, the augmented reality device (100) and the TWS device (1200) can be paired through Bluetooth.
[0138] The augmented reality device (100) can convert login information into a voice signal using a TTS module (Operation ②). A detailed description of the method by which the augmented reality device (100) converts login information into a voice signal is omitted.
[0139] The augmented reality device (100) can transmit a voice signal to the TWS device (1200) (Operation ③). In one embodiment of the present disclosure, the augmented reality device (100) can transmit data of the voice signal to the TWS device (1200) via Bluetooth.
[0140] The TWS device (1200) can output a voice signal (operation ④).
[0141] FIG. 13 is a flowchart illustrating an operation method of an augmented reality device (100) and a mobile device (1300) according to one embodiment of the present disclosure.
[0142] In step S1310, the augmented reality device (100) recognizes the mobile device (1300) from an image captured by the camera (110, see FIG. 3). In one embodiment of the present disclosure, a user wearing the augmented reality device (100) looks at the mobile device (1300) through the augmented reality device (100), and the augmented reality device (100) captures a real object including the mobile device (1300) through the camera (110) to obtain an image. The augmented reality device (100) can perform vision recognition using an artificial intelligence model to recognize the mobile device (1300) from the image.
[0143] The mobile device (1300) may be, for example, a smart phone. However, it is not limited thereto, and the mobile device (1300) may be any one of a tablet PC, an e-book reader, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, or a camcorder.
[0144] In step S1320, the augmented reality device (100) performs pairing with the mobile device (1300). In one embodiment of the present disclosure, the augmented reality device (100) can wirelessly pair with the mobile device (1300) using Bluetooth or WiFi Direct.
[0145] In step S1330, the augmented reality device (100) extracts website information including the site name and URL of the website displayed on the mobile device (1300) from the image. In step S1340, the augmented reality device (100) compares the extracted website information with the registration website information stored in the database in the memory (150, see FIG. 3) to identify the matching registration website. In step S1350, the augmented reality device (100) obtains login information of the identified registration website. Steps S1330 to S1350 are identical to steps S220 to S240 illustrated and described in FIG. 2, and therefore, redundant descriptions thereof will be omitted.
[0146] In step S1360, the augmented reality device (100) stores the acquired login information in text form.
[0147] In step S1370, the augmented reality device (100) transmits a text file containing login information to the mobile device (1300). In one embodiment of the present disclosure, the augmented reality device (100) may transmit the text data containing login information to the mobile device (1300) via Bluetooth or Wi-Fi Direct.
[0148] In step S1380, the mobile device (1300) parses the text file to extract the ID and password. In one embodiment of the present disclosure, the mobile device (1300) may also extract website information including the site name and URL from the text file.
[0149] In step S1390, the mobile device (1300) performs automatic login by entering the extracted ID and password into the website. In one embodiment of the present disclosure, the mobile device (1300) can compare the website information extracted from the text file with the information of the currently accessed website to determine whether there is a match. If the website is determined to be a match, the mobile device (1300) can automatically log in by entering the ID and password extracted from the text file into the website.
[0150] The embodiment illustrated in FIG. 13 relates to a function and / or operation in which an augmented reality device (100) that operates in a stand-alone manner, including a processor, obtains login information regarding a website and transmits the information to a mobile device (1300), and the mobile device (1300) automatically logs in to the website using the login information transmitted from the augmented reality device (100). The embodiment illustrated in FIG. 13 can be equally applied not only to automatically logging in to a website, but also to logging in to Internet banking, a public certificate, or an application. The mobile device (1300) according to one embodiment of the present disclosure receives login information regarding a website from the augmented reality device (100), performs automatic log-in, and then stores the login information, thereby allowing the user to automatically log in later even when not wearing the augmented reality device (100).
[0151] FIG. 14 is a diagram illustrating the operation of an augmented reality device (101), a mobile device (1400), and a cloud server (1500) according to one embodiment of the present disclosure.
[0152] Referring to FIG. 14, the augmented reality device (101) may include a camera (110). The augmented reality device (101) illustrated in FIG. 14 is a device that operates in a phone-tethered manner tethered with a mobile device (1400), and has an index indicating performance such as an operation speed or processing speed of an internal processor (or an application processor) that is relatively low compared to the processor (140, see FIG. 3) of the augmented reality device (101) illustrated in FIGS. 1 to 13, so that most functions and / or operations are performed by the mobile device (1400), and only an operation of outputting the performance result can be performed. The augmented reality device (101) and the cloud server (1500) are synchronized, and login information that a user used to log in through the mobile device (1400) while wearing the augmented reality device (101) in the past can be stored in a database of the cloud server (1500).
[0153] In the embodiment illustrated in FIG. 14, the augmented reality device (101) can capture a website displayed by an external device using a camera (110) to obtain a plurality of image frames and transmit the plurality of image frames to a mobile device (1400) (Operation ①). The mobile device (1400) can extract website information from the plurality of image frames (Operation ②) and transmit the extracted website information to a cloud server (1500) (Operation ③). The cloud server (1500) can identify a registered website matching the received website information (Operation ④) and transmit login information of the identified registered website to the mobile device (1400) (Operation ⑤). The mobile device (1400) can perform automatic login using the login information received from the cloud server (1500) (Operation ⑥).
[0154] The operations (operations ① to ⑥) of the augmented reality device (101), mobile device (1400), and cloud server (1500) illustrated in FIG. 14 will be described in detail with reference to FIG. 15.
[0155] FIG. 15 is a flowchart illustrating an operation method of an augmented reality device (101), a mobile device (1400), and a cloud server (1500) according to one embodiment of the present disclosure.
[0156] In step S1510, the augmented reality device (101) performs pairing with the mobile device (1400). In one embodiment of the present disclosure, the augmented reality device (101) may perform pairing with the mobile device (1400) via a short-range wireless communication method, for example, Bluetooth or WiFi-Direct.
[0157] In step S1520, the augmented reality device (101) captures a website displayed by the mobile device (1400) through a camera to obtain a plurality of image frames. The plurality of image frames are acquired continuously over time and can constitute video data. In one embodiment of the present disclosure, the augmented reality device (101) can capture a plurality of image frames by capturing not only a website accessed and displayed by the mobile device (1400), but also an execution screen of an Internet banking service, a public certificate, or an application.
[0158] In step S1530, the augmented reality device (101) transmits a plurality of image frames to the mobile device (1400). In one embodiment of the present disclosure, the augmented reality device (101) transmits a plurality of image frames acquired in real time to the mobile device (1400), and the plurality of image frames can be streamed in real time in the form of a video on the mobile device (1400).
[0159] In step S1540, the mobile device (1400) extracts website information including the site name and URL of the website from a plurality of image frames. In one embodiment of the present disclosure, the mobile device (1400) may perform vision recognition using an artificial intelligence model to detect the site name area and URL area of the website from a plurality of image frames acquired from the augmented reality device (101). In one embodiment of the present disclosure, the artificial intelligence model may be a deep neural network model trained to recognize the shape of the site name area and URL area from an image through a supervised learning method that applies an image as an input and applies the shape of a standardized site name area and URL area as a ground truth. The deep neural network model may be, for example, a convolutional neural network (CNN) model, but is not limited thereto.
[0160] The mobile device (1400) can extract website information including the site name and URL from the detected site name area and URL area. In one embodiment of the present disclosure, the mobile device (1400) can perform STR (Scene Text Recognition) to extract text representing the site name from an image area included in the detected site name area. In one embodiment of the present disclosure, the mobile device (1400) can perform OCR (Optical Character Recognition) to extract text representing the URL from an image area included in the URL area.
[0161] In step S1550, the mobile device (1400) accesses the database (1510) of the cloud server (1500) and compares the received website information with the registration website information stored in the database (1510). The cloud server (1500) may include a database (1510) containing login information regarding the registration website. The term "registration website" refers to a registered website where login information, including an ID and password for logging into the website, is pre-stored.
[0162] In step S1560, the mobile device (1400) identifies a matching registration website as a result of the comparison, and obtains login information of the identified registration website from the cloud server (1500). The mobile device (1400) may compare website information with website information of the registration website previously stored in the database (1510). In one embodiment of the present disclosure, the mobile device (1400) may perform a string search on the database (1510) of the cloud server (1500) to compare the website information with website information of the registration website stored in the database. If a matching registration website is identified as a result of the comparison, the mobile device (1400) may obtain login information including an ID and password of the identified registration website from the database (1510).
[0163] In step S1570, the mobile device (1400) parses the login information to extract the ID and password. In one embodiment of the present disclosure, the mobile device (1400) may receive the login information in the form of a text file from the cloud server (1500) and extract website information, including the site name and URL, from the received text file.
[0164] In step S1580, the mobile device (1400) performs automatic login by entering the extracted ID and password into the website. In one embodiment of the present disclosure, the mobile device (1400) can compare the website information extracted from the text file with the information of the currently accessed website to determine whether there is a match. If the website is determined to be a match, the mobile device (1400) can automatically log in by entering the ID and password extracted from the text file into the website.
[0165] The embodiment illustrated in FIGS. 14 and 15 is an embodiment in which an augmented reality device (101) including an internal processor (or application processor) with relatively low performance compared to FIG. 13 operates in a phone-tethered manner in which it is paired with a mobile device (1400). The augmented reality device (101) only streams a video file composed of a plurality of image frames to the mobile device (1400), and the operations of identifying a registration website and obtaining login information can be performed through a cloud server (1500). In addition, the mobile device (1400) can obtain login information of the registration website from the cloud server (1500) and perform automatic login. Automatic login allows a user to log in without having to perform cumbersome actions such as directly entering an ID and password, thereby providing an effect of improved user convenience.
[0166] In the phone-tethered method, if website information is registered on the cloud server (1500), the user can automatically log into the website using only the mobile device (1400) without having to wear the augmented reality device (101). Furthermore, the embodiments illustrated in FIGS. 14 and 15 can be equally applied not only to automatically logging into a website, but also to logging into internet banking, public certificates, or applications.
[0167] One aspect of the present disclosure provides a method for an augmented reality device (100) to provide login information. The method for operating the augmented reality device (100) according to one embodiment of the present disclosure may include a step (S210) of executing an augmented reality login mode (AR login mode) for logging into a website accessed and displayed by an external device (200). The method for operating the augmented reality device (100) according to one embodiment of the present disclosure may include a step (S220) of extracting website information including a site name and URL (Uniform Resource Locators) of the website from an image captured by a camera as the augmented reality login mode is executed. The method for operating the augmented reality device (100) according to one embodiment of the present disclosure may include a step (S230) of comparing the extracted website information with website information of a registered website that is registered in advance and stored in a database of a memory, and identifying a registered website having website information that matches the extracted website information based on the comparison result. The method of operating an augmented reality device (100) according to one embodiment of the present disclosure may include a step (S240) of obtaining login information including ID and password information of an identified registration website from a database. The method of operating an augmented reality device (100) according to one embodiment of the present disclosure may include a step (S250) of outputting the obtained login information.
[0168] In one embodiment of the present disclosure, the step (S210) of executing the augmented reality login mode may include: receiving a user input of pressing a button mapped to a function of executing the augmented reality login mode or receiving a voice input from a user for executing the augmented reality login mode; and executing the augmented reality login mode based on the received user input or voice input.
[0169] In one embodiment of the present disclosure, the step (S210) of executing the augmented reality login mode may include a step of executing the augmented reality login mode by recognizing a hand pointing input pointing to a login area of a website from an image captured through a camera.
[0170] In one embodiment of the present disclosure, the step (S210) of executing the augmented reality login mode may include the step of detecting a gaze point where the gaze direction of the user's left eye and the gaze direction of the user's right eye converge using a gaze tracking sensor (120, see FIG. 3); and the step of executing the augmented reality login mode when the detected gaze point is located on a login area of a website displayed by an external device (200).
[0171] In one embodiment of the present disclosure, the method of operating the augmented reality device (100) may further include a step (S650) of registering the ID and password of the website as login information of a new website, if a registered website matching the extracted website information is not identified as a result of comparing the extracted website information with the website information of the registered website.
[0172] In one embodiment of the present disclosure, the step (S650) of registering login information of the new website may include a step (S710) of capturing a login screen of the website by using a camera (110, see FIG. 3) according to a position of a gaze point according to a direction of the user's gaze detected by an eye tracking sensor (120) or an area pointed by a user's hand recognized from an image, and positioned on a login area of the new website displayed by an external device (200), thereby obtaining a plurality of image frames. The step (S650) of registering login information of the new website may include a step (S750) of extracting login information including an ID and password of the new website from the obtained plurality of image frames. The step (S650) of registering login information of the new website may include a step (S760) of storing the extracted login information together with the site name and URL of the new website in the database.
[0173] In one embodiment of the present disclosure, in the step of outputting the login information (S250), the augmented reality device (100) can display the login information by overlaying it on the login area of the website displayed by the external device (200) by projecting at least one of letters, numbers, special symbols, and combinations thereof representing the ID and password included in the login information onto a waveguide.
[0174] In one embodiment of the present disclosure, in the step (S250) of outputting the login information, the augmented reality device (100) projects at least one of letters, numbers, special symbols, and combinations thereof representing the ID and password included in the login information onto a waveguide, thereby displaying the login information on an arbitrary area of a real object viewed by the user through the augmented reality device (100).
[0175] In one embodiment of the present disclosure, the step (S250) of outputting the login information may include a step (S1110) of converting the text-type ID and password included in the login information into a voice signal by performing TTS (Text-to-Speech) and a step (S1120) of outputting the converted voice signal through a speaker (180).
[0176] In one embodiment of the present disclosure, the operating method of the augmented reality device (100) may further include a step of recognizing the external device (200) from an image captured by the camera (110) (S1310), and a step of pairing with the recognized external device (200) via a short-range wireless communication method (S1320). The operating method of the augmented reality device (100) may further include a step of storing the acquired login information as a text file and transmitting the stored text file to the external device (200) via a short-range wireless communication method (S1370). The external device (200) may parse text included in the text file received from the augmented reality device (100) to obtain login information including an ID and a password, and input the obtained login information into the website to perform automatic login (S1390).
[0177] One aspect of the present disclosure provides an augmented reality device (100) that provides login information. According to one embodiment of the present disclosure, the augmented reality device (100) may include a camera (110) that captures an image by photographing an object in a real space; a display unit (170) including a waveguide and an optical engine; a memory (150) including a database that stores at least one instruction and login information of a registered website; and at least one processor (140). The at least one processor (140) may execute at least one instruction to execute an augmented reality login mode, and as the augmented reality login mode is executed, may capture a website displayed by an external device (200) through the camera (110) to acquire an image. The at least one processor (140) may extract website information including the website name and URL (Uniform Resource Locators) of the website from the acquired image by executing at least one instruction. The at least one processor (140) may compare the extracted website information with the website information of the registered website previously stored in the database by executing at least one command, and identify the registered website having the website information matching the extracted website information based on the comparison result. The at least one processor (140) may obtain login information including the ID and password information of the identified registered website from the database by executing at least one command. The at least one processor (140) may output the obtained login information through the display unit (170) by executing at least one command.
[0178] In one embodiment of the present disclosure, the augmented reality device (100) may further include a button positioned on the temple of the glasses and protruding outward. The button is mapped to a function for executing an augmented reality login mode, and the at least one processor (140) may receive a user input for pressing the button. The at least one processor (140) may execute the augmented reality login mode based on the received user input.
[0179] In one embodiment of the present disclosure, the at least one processor (140) can recognize a hand pointing input pointing to a login area of a website from an image captured through a camera (110) by executing at least one command, and execute an augmented reality login mode based on the recognized hand pointing input.
[0180] In one embodiment of the present disclosure, the augmented reality device (100) may further include a gaze tracking sensor (120) that detects a gaze direction of a user's left eye and a gaze direction of a user's right eye. The at least one processor (140) may execute at least one command to detect a gaze point where the gaze direction of the user's left eye and the gaze direction of the user's right eye, which are detected using the gaze tracking sensor (120), converge, and execute an augmented reality login mode when the detected gaze point is located on a login area of a website displayed by an external device (200).
[0181] In one embodiment of the present disclosure, the at least one processor (140) executes at least one command so that, if a registered website matching the extracted website information is not identified as a result of comparing the extracted website information with the website information of the registered website, the ID and password of the website can be registered as login information of the new website.
[0182] In one embodiment of the present disclosure, the at least one processor (140) may capture a login screen of a website through the camera (110) and obtain a plurality of image frames by executing at least one command, such that the location of the gaze point according to the user's gaze direction or the area pointed by the user's hand recognized from the image is located on the login area of the new website displayed by the external device (200). The at least one processor (140) may extract login information including an ID and password of the new website from the obtained plurality of image frames by executing at least one command. The at least one processor (140) may store the extracted login information together with the site name and URL of the new website in a database by executing at least one command.
[0183] In one embodiment of the present disclosure, the at least one processor (140) controls the optical engine to project at least one of letters, numbers, special symbols, and combinations thereof representing an ID and password included in the login information onto the waveguide by executing at least one command, thereby overlaying and displaying the login information on the login area of the website displayed by the external device (200).
[0184] In one embodiment of the present disclosure, the at least one processor (140) controls the optical engine to project at least one of letters, numbers, special symbols, and combinations thereof representing an ID and password included in the login information onto a waveguide by executing at least one command, thereby displaying the login information on an arbitrary area of a real object viewed by a user through the augmented reality device (100).
[0185] In one embodiment of the present disclosure, the augmented reality device (100) may further include a speaker (180) that outputs an audio signal. The at least one processor (140) may perform TTS (Text-to-Speech) to convert a text-based ID and password included in the login information into a voice signal, and output the converted voice signal through the speaker (180).
[0186] One aspect of the present disclosure provides a method for a mobile device (1400) to automatically log in to a website. In one embodiment of the present disclosure, the method for operating the mobile device (1400) may include a step (S1530) of receiving a plurality of image frames captured by a camera of an augmented reality device (101) from an augmented reality device (101) paired via short-range wireless communication. The method for operating the mobile device (1400) may include a step (S1540) of extracting website information including a site name and URL of a website to be logged in from the received plurality of image frames. The method for operating the mobile device (1400) may include a step (S1550) of comparing the extracted website information with registered website information stored in a cloud server. The method for operating the mobile device (1400) may include a step (S1560) of identifying a registered website matching the extracted website information based on the comparison result, and obtaining login information including an ID and password information of the identified registered website. The operating method of the mobile device (1400) may include a step (S1570) of extracting an ID and password by parsing the acquired login information. The operating method of the mobile device (1400) may include a step (S1580) of entering the extracted ID and password into a website to execute automatic login.
[0187] The program executed by the augmented reality device (100) described in the present disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.
[0188] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.
[0189] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The media may be readable by a computer, stored in a memory, and executed by a processor.
[0190] A computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium does not contain signals and is tangible, but does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.
[0191] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.
[0192] A computer program product may include a software program, a computer-readable storage medium having the software program stored thereon. For example, the computer program product may be provided by a manufacturer of an augmented reality device (100) or an electronic market (e.g., Samsung Galaxy Store). TM ) may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically. For electronic distribution, at least a portion of the software program may be stored in a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server of the augmented reality device (100), a server of an electronic market, or a relay server that temporarily stores the software program.
[0193] The computer program product may include a storage medium of the server or the storage medium of the augmented reality device (100) in a system comprising an augmented reality device (100) and / or a server (e.g., a cloud server (1500, see FIGS. 14 and 15)). Alternatively, if there is a third device (e.g., a mobile device (1400, see FIGS. 14 and 15)) that is communicatively connected to the augmented reality device (100), the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the augmented reality device (100) to the third device, or from the third device to an electronic device.
[0194] In this case, either the augmented reality device (100) or the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, at least one of the augmented reality device (100) and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.
[0195] For example, the augmented reality device (100) may execute a computer program product stored in a memory (150, see FIG. 3) to control another electronic device that is in communication with the augmented reality device (100) to perform a method according to the disclosed embodiments.
[0196] For example, a third device may execute a computer program product to control an electronic device in communication with the third device to perform a method according to the disclosed embodiment.
[0197] When a third device executes a computer program product, the third device may download the computer program product from the augmented reality device (100) and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the disclosed embodiments.
[0198] Although the embodiments described above have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.
Claims
1. In a method in which an augmented reality device (100) provides login information, Step (S210) of executing an augmented reality login mode (AR login mode) for logging into a website accessed and displayed by an external device (200); Step (S220) of extracting website information including the site name and URL (Uniform Resource Locators) of the website from an image captured through a camera as the augmented reality login mode is executed; A step (S230) of comparing the extracted website information with the website information of a registered website that has been pre-registered and stored in a database of a memory, and identifying a registered website having website information that matches the extracted website information based on the comparison result; Step (S240) of obtaining login information including ID and password information of the above-identified registration website from the database; and Step (S250) of outputting the acquired login information; A method comprising:
2. In paragraph 1, The step (S210) of executing the above augmented reality login mode is: A step of detecting a gaze point where the gaze direction of the user's left eye and the gaze direction of the user's right eye converge using a gaze tracking sensor (120); and A step of executing the augmented reality login mode when the detected gaze point is located on the login area of the website displayed by the external device (200); A method comprising:
3. In paragraph 1 or 2, If a registered website matching the extracted website information is not identified as a result of comparing the extracted website information with the website information of the registered website, a step (S650) of registering the ID and password of the website as login information of a new website; A method further comprising:
4. In paragraph 3, The step (S650) of registering login information for the above new website is as follows: A step (S710) of capturing a login screen of the website by using the camera so that the location of the gaze point according to the direction of the user's gaze detected by the gaze tracking sensor (120) or the area pointed by the user's hand recognized from the image is located on the login area of the new website displayed by the external device (200); A step (S750) of extracting login information including the ID and password of the new website from the acquired plurality of image frames; and A step (S760) of storing the extracted login information together with the site name and URL of the new website in the database; A method comprising:
5. In any one of clauses 1 to 4, The step (S250) of printing the above login information is as follows: A method for overlaying and displaying the login information on the login area of a website displayed by the external device (200) by projecting at least one of letters, numbers, special characters, and combinations thereof representing the ID and password included in the login information onto a waveguide.
6. In any one of clauses 1 to 5, The step (S250) of printing the above login information is as follows: A method for displaying the login information on an arbitrary area of a real object viewed by a user through the augmented reality device (100) by projecting at least one of letters, numbers, special characters, and combinations thereof representing the ID and password included in the login information onto a waveguide.
7. In any one of clauses 1 to 4, The step (S250) of printing the above login information is as follows: Step (S1110) of converting the ID and password in text form included in the login information into a voice signal by performing TTS (Text-to-Speech); and Step (S1120) of outputting the converted voice signal through a speaker (180); A method comprising:
8. In any one of clauses 1 to 7, Step (S1310) of recognizing the external device (200) from an image captured through the camera; Step (S1320) of pairing with the above-mentioned recognized external device (200) through a short-range wireless communication method; and A step (S1370) of saving the acquired login information as a text file and transmitting the saved text file to an external device (200) via the short-range wireless communication method; Including more, A method in which the external device (200) parses text included in a text file received from the augmented reality device (100) to obtain the login information including an ID and password, and inputs the obtained login information into the website to perform automatic login (S1390).
9. In an augmented reality device (100) that provides login information, A camera (110) that captures images by photographing objects in real space; A display unit (170) including a waveguide and an optical engine; A memory (150) comprising a database storing at least one instruction and login information of a registration website; and comprising at least one processor (140); The at least one processor (140) executes the at least one instruction: A method of executing an augmented reality login mode (AR login mode), and obtaining an image by photographing a website displayed by an external device (200) through the camera (110) while the augmented reality login mode is executed, extracting website information including a site name and URL (Uniform Resource Locators) of the website from the obtained image, comparing the extracted website information with the website information of the registered website previously stored in the database, identifying a registered website having website information matching the extracted website information according to the comparison result, and obtaining login information including ID and password information of the identified registered website from the database, An augmented reality device (100) that outputs the acquired login information through the display unit (170).
10. In paragraph 9, A gaze tracking sensor (120) that detects the gaze direction of the user's left eye and the gaze direction of the user's right eye; Including more, The at least one processor (140) executes the at least one instruction: An augmented reality device (100) that detects a gaze point where the gaze direction of the user's left eye and the gaze direction of the user's right eye converge using the gaze tracking sensor (120), and executes the augmented reality login mode when the detected gaze point is located on the login area of a website displayed by the external device (200).
11. In either of paragraphs 9 or 10, The at least one processor (140) executes the at least one instruction: An augmented reality device (100) that registers the ID and password of the website as login information for a new website when a registered website matching the extracted website information is not identified as a result of comparing the extracted website information with the website information of the registered website.
12. In paragraph 11, The at least one processor (140) executes the at least one instruction: As the location of the gaze point according to the user's gaze direction or the area pointed by the user's hand recognized from the image is located on the login area of the new website displayed by the external device (200), the login screen of the website is captured through the camera (110) to obtain a plurality of image frames, Extracting login information including the ID and password of the new website from the plurality of image frames obtained above, An augmented reality device (100) that stores the extracted login information together with the site name and URL of the new website in the database.
13. In any one of paragraphs 9 to 12, The at least one processor (140) executes the at least one instruction: An augmented reality device (100) that overlays and displays the login information on a login area of a website displayed by the external device (200) by controlling the optical engine to project at least one of letters, numbers, special symbols, and combinations thereof representing an ID and password included in the login information onto the waveguide.
14. In any one of paragraphs 9 to 13, The at least one processor (140) executes the at least one instruction: An augmented reality device (100) that projects at least one of letters, numbers, special symbols, and combinations thereof representing an ID and password included in the above login information onto a waveguide, thereby displaying the login information on an arbitrary area of a real object viewed by a user through the augmented reality device (100).
15. In any one of paragraphs 9 to 14, A speaker (180) that outputs an audio signal; Including more, The at least one processor (140) executes the at least one instruction: An augmented reality device (100) that converts text-based ID and passwords included in the login information into voice signals by performing TTS (Text-to-Speech) and outputs the converted voice signals through the speaker (180).
Citation Information
Patent Citations
Method for assisting in input to application providing content using head-mounted device, program for sing computer to implement the same, and content display device
JP2018063540A
Method for operating display device
KR1020150006705A
System of rhythm game and providing method thereof
KR1020230148658A
Biometric Authentication Substitute For Passwords On A Wearable Computing Device
US20140337634A1
Personal information display device and processing method therefor
US20220222971A1
