Augmented reality implementation device and user authentication method using same

Abstract

This augmented reality implementation device comprises: an optical lens provided to transmit image light displaying a virtual image to an eyeball of a user; a display module provided to emit the image light displaying the virtual image toward the optical lens; a camera module for acquiring captured data; a memory for storing a program including one or more instructions; and a processor provided to execute the one or more instructions of the program stored in the memory. The processor determines whether a user authentication request using biometric features is inputted, generates a control signal for controlling the camera module so as to operate the camera module, if the user authentication request using the biometric features is inputted, identifies the biometric features of an authentication object from the captured data generated by the operation of the camera module, determines whether the biometric features match pre-stored biometric authentication data, and generates an authentication completion signal using the biometric features if the biometric features match the biometric authentication data.

Description

Augmented reality implementation device and user authentication method using the same

[0001] The present invention relates to an augmented reality implementation device and a user authentication method using the same. More specifically, the present invention relates to an augmented reality implementation device that performs user authentication using the user's biometric characteristics and gaze trajectory when a user authentication request is input from an external server or the like, and a user authentication method using the same.

[0002] This application claims priority to Korean Patent Application No. 10-2024-0051040, filed April 16, 2024, the entire contents of which are incorporated herein by reference.

[0003] Previously, when performing banking payment services using a smartphone, user authentication was performed by taking a picture of the user's face using the camera installed in the smartphone, or by the user drawing an authentication pattern on the smartphone screen with their finger.

[0004] Recently, active development has been underway on AR glasses, which are worn on the user's face. Commercialization of AR glasses in this form is also underway. These glasses can communicate with banking servers and perform banking payment services. However, because they are worn on the user's face, these glasses face the problem of not being able to utilize the user authentication methods typically used on smartphones.

[0005] An object of the present invention is to provide an augmented reality implementation device capable of performing user authentication.

[0006] Another object of the present invention is to provide a user authentication method using the augmented reality implementation device described above.

[0007] According to exemplary embodiments for achieving the above object of the present invention, an augmented reality implementation device includes an optical lens configured to transmit image light displaying a virtual image to a user's eye, a display module configured to emit image light displaying the virtual image toward the optical lens, a camera module configured to acquire photographed data, a memory configured to store a program including one or more commands, and a processor configured to execute one or more commands of the program stored in the memory, wherein the processor determines whether a user authentication request using a biometric characteristic is input, and when a user authentication request using the biometric characteristic is input, generates a control signal for controlling the camera module to operate the camera module, identifies a biometric characteristic of an authentication object in photographed data generated by the operation of the camera module, determines whether the biometric characteristic matches pre-stored biometric authentication data, and when the biometric characteristic matches the biometric authentication data, generates an authentication completion signal using the biometric characteristic.

[0008] In exemplary embodiments, the augmented reality implementation device further includes a communication module communicatively connected to a server, and the processor can transmit an authentication completion signal using the biometric characteristic to the communication module, thereby causing the communication module to transmit the authentication completion signal using the biometric characteristic to an external server.

[0009] In exemplary embodiments, the processor may determine whether a user authentication request using an authentication pattern is input, and if a user authentication request using an authentication pattern is input, generate authentication pattern image data representing a virtual authentication pattern image, and cause the display module to emit image light displaying the virtual authentication pattern image toward the optical lens based on the authentication pattern image data.

[0010] In exemplary embodiments, the augmented reality implementation device further includes a gaze tracking sensor provided to obtain information on a user's gaze direction, and the processor, when a user authentication request using the authentication pattern is input, generates a control signal for controlling the gaze tracking sensor to operate the gaze tracking sensor, calculates a user's gaze point trajectory based on the user's gaze direction information generated by the operation of the gaze tracking sensor, determines whether the gaze point trajectory matches previously stored authentication pattern data, and, when the gaze point trajectory matches the authentication pattern data, generates an authentication completion signal using the authentication pattern.

[0011] In exemplary embodiments, the augmented reality implementation device further includes a communication module communicatively connected to a server, and the processor can transmit an authentication completion signal using the authentication pattern to the communication module, thereby causing the communication module to transmit the authentication completion signal using the authentication pattern to an external server.

[0012] In a user authentication method using an augmented reality implementation device according to exemplary embodiments for achieving another object of the present invention, it is determined whether a user authentication request using a biometric characteristic is input. If a user authentication request using the biometric characteristic is input, a control signal for controlling a camera module is generated to operate the camera module. The biometric characteristic of the authentication object is identified from the captured data generated by the operation of the camera module. It is determined whether the biometric characteristic matches previously stored biometric authentication data. If the biometric characteristic matches the biometric authentication data, an authentication completion signal using the biometric characteristic is generated.

[0013] In exemplary embodiments, the user authentication method using the augmented reality implementation device may further include transmitting an authentication completion signal using the biometric characteristic to a communication module, so that the communication module transmits the authentication completion signal using the biometric characteristic to an external server.

[0014] In exemplary embodiments, the user authentication method using the augmented reality implementation device may further include determining whether a user authentication request using an authentication pattern is input, and, if a user authentication request using an authentication pattern is input, generating authentication pattern image data representing a virtual authentication pattern image, and causing the display module to emit image light displaying the virtual authentication pattern image toward the optical lens based on the authentication pattern image data.

[0015] In exemplary embodiments, the user authentication method using the augmented reality implementation device may further include, when a user authentication request using the authentication pattern is input, generating a control signal for controlling a gaze tracking sensor to operate the gaze tracking sensor, calculating a user's gaze point trajectory based on the user's gaze direction information generated by the operation of the gaze tracking sensor, determining whether the gaze point trajectory matches previously stored authentication pattern data, and generating an authentication completion signal using the authentication pattern when the gaze point trajectory matches the authentication pattern data.

[0016] In exemplary embodiments, the user authentication method using the augmented reality implementation device may further include transmitting an authentication completion signal using the authentication pattern to a communication module, so that the communication module transmits the authentication completion signal using the authentication pattern to an external server.

[0017] According to exemplary embodiments, an augmented reality implementation device includes an optical lens configured to transmit image light displaying a virtual image to a user's eyes, a display module configured to emit image light displaying the virtual image toward the optical lens, a camera module configured to acquire photographed data, a memory configured to store a program including one or more commands, and a processor configured to execute one or more commands of the program stored in the memory, wherein the processor determines whether a user authentication request using a biometric characteristic is input, and when a user authentication request using the biometric characteristic is input, generates a control signal for controlling the camera module to operate the camera module, identifies a biometric characteristic of an authentication object in photographed data generated by the operation of the camera module, determines whether the biometric characteristic matches pre-stored biometric authentication data, and when the biometric characteristic matches the biometric authentication data, generates an authentication completion signal using the biometric characteristic.

[0018] Accordingly, when the user wears the augmented reality implementation device, user authentication can be easily performed through the user's biometric characteristics with strong security.

[0019] In addition, when the user wears the augmented reality implementation device, the user can receive a virtual authentication pattern image through the augmented reality implementation device and easily perform the user authentication through eye movement.

[0020] However, the effects of the present invention are not limited to the above-mentioned effects, and may be expanded in various ways without departing from the spirit and scope of the present invention.

[0021] FIG. 1 is a drawing showing an interface for performing user authentication using an augmented reality implementation device according to exemplary embodiments.

[0022] FIG. 2 is a perspective view showing the structure of an augmented reality implementation device according to exemplary embodiments.

[0023] FIG. 3 is a block diagram showing the configuration of an augmented reality implementation device according to exemplary embodiments.

[0024] Fig. 4 is a perspective view showing the detailed configuration of the optical lens of Fig. 3.

[0025] FIG. 5 is a diagram illustrating a principle of transmitting image light for displaying real object image light and virtual image to a user's eye by an optical lens and a display module according to exemplary embodiments.

[0026] FIGS. 6 to 12 are drawings showing augmented reality screens provided to a user through an augmented reality implementation device according to exemplary embodiments.

[0027] FIGS. 13 to 21 are flowcharts illustrating a user authentication method using an augmented reality implementation device according to exemplary embodiments.

[0028] [Explanation of symbols]

[0029] 10: Augmented reality implementation device

[0030] 20: Server

[0031] 100: Optical Lens

[0032] 200: Display module

[0033] 300: Camera module

[0034] 400: Eye tracking sensor

[0035] 500: Communication module

[0036] 600: Processor

[0037] 700: Memory

[0038] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present invention. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names, but rather based on their meanings and the overall content of the present invention.

[0039] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0040] Throughout the present invention, when a part is said to "include" a certain component, unless otherwise specifically stated, this does not exclude other components but rather implies the inclusion of other components. Furthermore, terms such as "part," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0041] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system, in conjunction with other devices or components, is "capable of." For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) that can perform the operations by executing one or more software programs stored in memory.

[0042] In addition, when a component is referred to as being "connected" or "connected" to another component in the present invention, it should be understood that the component may be directly connected or directly connected to the other component, but may also be connected or connected via another component in between, unless there is a specific description to the contrary.

[0043] In the present invention, 'Augmented Reality (AR)' means displaying a virtual image within a physical environment space of the real world.

[0044] In addition, the term 'Augmented Reality Implementation Device' refers to a device capable of implementing augmented reality, and generally includes an augmented reality glasses device (Augmented Reality Glasses) in the form of glasses worn on the user's face, a head-mounted display device (HMD: Head Mounted Display Apparatus) worn on the head, and an augmented reality helmet (Augmented Reality Helmet) worn on the head.

[0045] Meanwhile, 'real world object' refers to objects that actually exist in the real world. For example, a real world object may be various objects such as electronic devices that the user recognizes through an augmented reality implementation device, or a part of the user's body such as a hand.

[0046] In addition, a 'virtual image' is an image generated by an augmented reality implementation device and may include both static and dynamic images. Such a virtual image may be recognized by the user together with a real object by the augmented reality implementation device. The virtual image may be an image that includes information about a service provided to the user. In addition, the virtual image may be an image that indicates information about the operation of the augmented reality implementation device, a control menu for the augmented reality implementation device, etc. In addition, the virtual image may be an image that includes a specific pattern shape for user authentication. The virtual image may include, for example, at least one selected from letters, numbers, symbols, icons, images, and animations.

[0047] Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those skilled in the art can easily implement them. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein.

[0048] Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

[0049] FIG. 1 is a drawing showing an interface for performing user authentication using an augmented reality implementation device according to exemplary embodiments.

[0050] Referring to FIG. 1, a user can be provided with an augmented reality environment in which a virtual image is displayed in a real scene containing real objects through an augmented reality implementation device (10). The user can receive information regarding services provided from an external server, etc. through a virtual service content image (VSI) displayed by the augmented reality implementation device (10).

[0051] At this time, if there is a request for user authentication through the augmented reality implementation device (10), the user can take a picture of a part of the user's body while wearing the augmented reality implementation device (10), and perform user authentication through biometric characteristics identified from the pictured part of the user's body.

[0052] In addition, when there is a request for user authentication through an augmented reality implementation device (10), the user can be provided with a virtual authentication pattern image (VOI) displayed by the augmented reality implementation device (10), and perform the user authentication by moving the gaze along the authentication pattern included in the virtual authentication pattern image (VOI).

[0053] Here, a user authentication request via the augmented reality implementation device (10) may occur when the augmented reality implementation device (10) receives the user authentication request from an external server, etc. For example, the augmented reality implementation device (10) may receive payment service information from an external bank's payment system server, and the user authentication request may occur based on the received payment service information.

[0054] In addition, a user authentication request through the augmented reality implementation device (10) may occur when an authentication code is photographed through the augmented reality implementation device (10) and the authentication code is identified. For example, when the user photographs the authentication code, such as a QR code displayed on a monitor or printed material, while wearing the augmented reality implementation device (10), and the augmented reality implementation device (10) identifies the authentication code in the photographed data, the user authentication request may occur based on this.

[0055] In addition, a user authentication request through the augmented reality implementation device (10) may occur when the user first wears the augmented reality implementation device (10). For example, when the user wears the augmented reality implementation device (10) and the augmented reality implementation device (10) identifies that the user is wearing the augmented reality implementation device (10) through a wearing detection sensor, the user authentication request may occur based on the wearing information.

[0056] In addition, a user authentication request through the augmented reality implementation device (10) may occur when the augmented reality implementation device (10) determines that the user is exposed to a dangerous state by a previously stored danger detection program while the user is wearing the augmented reality implementation device (10). For example, when the user is exposed to a specific dangerous state while wearing the augmented reality implementation device (10) and the augmented reality implementation device (10) recognizes that the user is exposed to the dangerous state through a danger detection sensor, the user authentication request may occur based on the dangerous state exposure recognition information.

[0057] The augmented reality implementation device (10) can identify the user authentication request and determine whether a user authentication request using biometric characteristics is input. If a user authentication request using biometric characteristics is input, the augmented reality implementation device (10) can photograph a part of the user's body with a camera module. The augmented reality implementation device (10) can identify the user's biometric characteristics from the photographed data, and perform user authentication by determining whether the identified biometric characteristics match pre-stored biometric authentication data. For example, the augmented reality implementation device (10) can photograph the user's palm with the camera module, identify a vein pattern of the user's palm from the photographed data, and perform user authentication by determining whether the identified vein pattern matches the pre-stored user vein pattern data. In addition, the augmented reality implementation device (10) can photograph the user's eye with an eye tracking sensor, identify the user's iris pattern from the photographed data, and perform user authentication by determining whether the identified iris pattern matches pre-stored user iris pattern data.

[0058] In addition, the augmented reality implementation device (10) can identify the user authentication request and determine whether a user authentication request using an authentication pattern is input. If a user authentication request using the authentication pattern is input, the augmented reality implementation device (10) can display a virtual authentication pattern image (VOI) including the authentication pattern and provide it to the user. The augmented reality implementation device (10) can acquire a user's gaze point trajectory based on the user's gaze direction information acquired through the gaze tracking sensor, and can perform user authentication by determining whether the acquired gaze point trajectory matches with previously stored authentication pattern data.

[0059] FIG. 2 is a perspective view showing the structure of an augmented reality implementation device according to exemplary embodiments.

[0060] In exemplary embodiments, the augmented reality implementation device (10) may be an augmented reality glasses (Augmented Reality Class) in the form of glasses worn on the user's face. In addition, in another embodiment, the augmented reality implementation device (10) may be a head-mounted display apparatus (HMD) or an augmented reality helmet worn on the user's head.

[0061] Referring to FIG. 2, when the augmented reality implementation device (10) is the augmented reality glass in the form of the glasses, the augmented reality implementation device (10) may include a frame portion (51), a fixing portion (53), an optical lens (100), a display module (200), a camera module (300), a gaze tracking sensor (400), a communication module (500), a processor (600), and a memory (700).

[0062] The frame (51) may be provided in a form that completely surrounds the periphery of the lens, similar to a typical pair of glasses. In addition, the frame (51) may be provided in a form that surrounds only a portion of the lens. The fixed portion (53) may be provided in the form of a pair of glasses arms. In addition, the fixed portion (53) may be rotatably coupled to the frame (51). In this way, the fixed portion (53) may be provided to be folded relative to the frame (51) by being rotatably coupled to the frame (51).

[0063] An optical lens (100), a display module (200), a camera module (300), a gaze tracking sensor (400), a communication module (500), a processor (600), and a memory (700) may be placed in at least one selected from among the frame portion (51) and the fixed portion (53).

[0064] In exemplary embodiments, at least a portion of the outer edge of the optical lens (100) may be coupled to the frame portion (51). In exemplary embodiments, only an upper portion of the outer edge of the optical lens (100) may be coupled to the frame portion (51). The optical lens (100) may have a left-eye optical lens (100L) and a right-eye optical lens (100R), and the left-eye optical lens (100L) and the right-eye optical lens (100R) may each be configured identically. The left-eye optical lens (100L) may be coupled to the left portion of the frame portion (51), and the right-eye optical lens (100R) may be coupled to the right portion of the frame portion (51). However, this is exemplary, and the frame portion (51) may be configured in two pieces to correspond to the left-eye optical lens (100L) and the right-eye optical lens (100R), respectively. In this case, the left eye optical lens (100L) and the right eye optical lens (100R) can be combined into separate frame parts (51) to form an augmented reality implementation device (10).

[0065] In exemplary embodiments, the display module (200) may be disposed inside the frame portion (51). The display module (200) may be disposed on the upper portion of the frame portion (51) and may be configured to emit image light (VL) (see FIG. 3) that displays a virtual image in a direction from the upper portion to the lower portion of the optical lens (100). The display module (200) may have a left-eye display module (200L) and a right-eye display module (200R), and the left-eye display module (200L) and the right-eye display module (200R) may each be configured identically. The left-eye display module (200L) may be disposed on the left side of the frame portion (51), and the right-eye display module (200R) may be disposed on the right side of the frame portion (51).

[0066] In exemplary embodiments, the camera module (300) may be disposed in a form embedded in a portion of the frame portion (51). For example, the camera module (300) may be disposed in a form embedded in the nose bridge portion of the frame portion (51) and may be provided to capture real objects placed in the front space of the user.

[0067] In exemplary embodiments, the gaze tracking sensor (400), the communication module (500), the processor (600), and the memory (700) may be arranged in at least one selected from the frame portion (51) and the fixed portion (53).

[0068] The detailed configuration and function of the optical lens (100), display module (200), camera module (300), eye tracking sensor (400), communication module (500), processor (600), and memory (700) constituting the augmented reality implementation device (10) will be described with reference to FIG. 3.

[0069] Below, the configuration and function of the augmented reality implementation device (10) are described in detail.

[0070] Fig. 3 is a block diagram showing the configuration of an augmented reality implementation device according to exemplary embodiments. Fig. 4 is a perspective view showing a detailed configuration of an optical lens of Fig. 3. Fig. 5 is a diagram showing the principle of transmitting image light for displaying a real object image light and a virtual image to a user's eyes by an optical lens and a display module according to exemplary embodiments. Figs. 6 to 12 are diagrams showing augmented reality screens provided to a user through an augmented reality implementation device according to exemplary embodiments.

[0071] Referring to FIGS. 3 to 12, the augmented reality implementation device (10) may include an optical lens (100), a display module (200), a camera module (300), an eye tracking sensor (400), a communication module (500), a processor (600), and a memory (700). However, the components of the augmented reality implementation device (10) according to the present invention are not limited to those shown in FIG. 3, and for example, the augmented reality implementation device (10) may further include a battery module (not shown) provided to supply power to electronic components.

[0072] First, the optical lens (100) is provided to transmit real object image light (RL) emitted from a real object and image light (VL) displaying a virtual image emitted from a display module (200) to the user's eye (E).

[0073] As illustrated in FIG. 4, the optical lens (100) may have an overall shape such as a circle or a rectangle, and the optical lens (100) may have a first surface (101) and a second surface (103) that are arranged to face each other.

[0074] Here, when the user wears the augmented reality implementation device (10) on the face, the first side (101) of the optical lens (100) may be the side placed on the real object side, and the second side (103) may be the side placed on the user's eye (E) side.

[0075] Meanwhile, the optical lens (100) may include an optical element (110).

[0076] The optical lens (100) may be made of a transparent glass material or plastic material and may be provided to transmit real object image light (RL) emitted from a real object to the user's eye (E). In addition, the optical lens (100) may be provided to transmit image light (VL) displaying a virtual image transmitted from an optical element (110) to the user's eye (E).

[0077] The optical element (110) may be used without limitation as long as it is a reflective element capable of reflecting image light. For example, the optical element (110) may be a mirror with a reflectivity of 100% that reflects all image light, but may also be provided in the form of a semi-mirror that has transparency that transmits only a portion of the image light. In addition, the optical element (110) may be selected from various optical refractive elements or diffractive elements, such as a diffractive optical element and a holographic optical element. The optical element (110) may be configured by a combination of at least one selected from among a reflective element, a refractive element, and a diffractive element.

[0078] Meanwhile, the optical element (110) may be disposed embedded within the optical lens (100). The optical element (110) may be provided to reflect image light (VL) displaying a virtual image emitted from the display module (200) and transmit it to the user's eye (E).

[0079] Here, the optical element (110) may include a first optical element (111) and a second optical element (113).

[0080] In exemplary embodiments, the first optical element (111) may have a bar shape having a predetermined width and length. In exemplary embodiments, the width of the first optical element (111) may be formed to be 8 mm or less, more preferably 4 mm or less.

[0081] The first optical element (111) can be embedded in the lower part of the inside of the optical lens (100) and arranged along the longitudinal direction of the optical lens (100).

[0082] In exemplary embodiments, the first optical element (111) may be arranged such that the reflective surface faces the first surface (101) of the optical lens (100), and the first optical element (111) may be provided to reflect image light (VL) representing a virtual image incident from the first surface (101) of the optical lens (100) and emit the image light toward the first surface (101) of the optical lens (100).

[0083] Next, the second optical element (113) may have a bar-shaped, circular, or oval shape. In addition, there may be a plurality of second optical elements (113). In addition, the second optical element (113) may be formed in the shape of a mirror bar.

[0084] In exemplary embodiments, the size of each of the plurality of second optical elements (113) may be formed to be 8 mm or less, more preferably 4 mm or less. Here, the size of the second optical element (113) may mean the shortest distance between any two points on the edge boundary of the second optical element (113).

[0085] A plurality of second optical elements (113) may be disposed embedded within the optical lens (100), and the plurality of second optical elements (113) may be disposed on top of the first optical element (111), but may be disposed at a predetermined distance from the first optical element (111). At this time, when the second optical element (113) is formed in the form of a mirror bar, the second optical element (113) may be disposed along the same longitudinal direction as the longitudinal direction of the first optical element (111) described above.

[0086] At this time, each of the plurality of second optical elements (113) may be arranged so that the reflective surface faces the second surface (103) of the optical lens (100), and each of the plurality of second optical elements (113) may be provided to reflect image light (VL) displaying a virtual image incident from the first optical element (111) or the first surface (101) of the optical lens (100) and emit it toward the user's eye (E).

[0087] The process in which the image light (VL) displaying the virtual image emitted from the display module (200) is transmitted to the user's eye (E) through the first optical element (111) and the second optical element (113) will be described in detail with reference to FIG. 5.

[0088] Next, the display module (200) can generate image light (VL) that displays a virtual image, and can be equipped to emit the generated image light toward the optical lens (100).

[0089] In exemplary embodiments, the display module (200) may be positioned on the upper portion of the optical lens (100) in a form that is mounted on a holder (not shown) mounted on the upper portion of the optical lens (100). Alternatively, the display module (200) may be positioned inside the frame portion (51) in a form that is directly coupled to the frame portion (51) (see FIG. 2).

[0090] For example, the display module (200) may include an LED (Light Emitting Diodes) display device, an OLED (Organic Light Emitting Diodes) display device, and a micro display device.

[0091] The display module (200) can receive image data representing a virtual image from the processor (600), generate image light (VL) displaying the virtual image based on the received image data, and emit the image light (VL) displaying the generated virtual image toward the optical lens (100).

[0092] In exemplary embodiments, the processor (600) may provide image data including pixel values ​​for a plurality of pixels representing a virtual image to the display module (200), and the display module (200) may generate image light representing the virtual image by controlling a light source according to the pixel values ​​of each of the plurality of pixels.

[0093] A specific method in which the display module (200) emits image light (VL) to display a virtual image on the optical lens (100) will be described in detail with reference to FIG. 5.

[0094] As illustrated in FIG. 5, the display module (200) may be disposed on the upper portion of the optical lens (100). The display module (200) may emit image light (VL) displaying a virtual image toward the first surface (101) of the optical lens (100). The image light (VL) displaying a virtual image emitted toward the first surface (101) of the optical lens (100) may be totally reflected by the first surface (101) of the optical lens (100) and transmitted to the first optical element (111). At this time, the first optical element (111) may be disposed so as to reflect the image light (VL) displaying a virtual image that is totally reflected by the first surface (101) of the optical lens (100) back toward the first surface (101) of the optical lens (100). Accordingly, the image light (VL) displaying the virtual image transmitted to the first optical element (111) can be reflected by the first optical element (111) and emitted toward the first surface (101) of the optical lens (100).

[0095] Thereafter, the image light (VL) displaying the virtual image emitted toward the first surface (101) of the optical lens (100) may be totally reflected by the first surface (101) of the optical lens (100) and transmitted to the second optical element (113). At this time, the second optical element (113) may be arranged so as to reflect the image light (VL) displaying the virtual image that is totally reflected by the first surface (101) and incident toward the user's eye (E). Accordingly, the image light (VL) displaying the virtual image transmitted to the second optical element (113) may be reflected by the second optical element (113) and transmitted toward the user's eye (E). The user may recognize the virtual image through the image light (VL) displaying the virtual image that is incident to the eye (E).

[0096] Here, the size of the second optical element (113) can be formed to be 8 mm or less or 4 mm or less, which is smaller than the size of a human pupil. Accordingly, the depth of field for light incident on the user's eye (E) through the second optical element (113) can be greatly deep. Accordingly, when the user changes the focal length for the real world while gazing at the real world, a pinhole effect can be generated that causes the user to perceive that the virtual image is always in focus regardless of the change in the focal length.

[0097] Next, the camera module (300) may be equipped to acquire shooting data by capturing real objects in the real world and transmit the acquired shooting data to the processor (600).

[0098] In exemplary embodiments, the camera module (300) can acquire photographing data by photographing an authentication object (CO), which is a real object, and provide the acquired photographing data to the processor (600).

[0099] In addition, the camera module (300) can obtain shooting data by taking a picture of a manipulation object (HO), which is a real object, and provide the obtained shooting data to the processor (600).

[0100] For example, the camera module (300) may include an image sensor such as a complementary metal-oxide semiconductor (CMOS), a charge-coupled device (CCD), or an active pixel sensor. The camera module (300) may include a rectilinear lens, a concave lens, a convex lens, a wide-angle lens, or the like. In addition, the camera module (300) may be analog or digital. In addition, the camera module (300) may include a biometric scanner such as a blood vessel scanner.

[0101] Next, the gaze tracking sensor (400) may be a device that tracks the gaze direction of the user's eye (E). In exemplary embodiments, the gaze tracking sensor (400) may include a light irradiation unit and a light detection unit. The gaze tracking sensor (400) may be equipped to detect the user's gaze direction by detecting an image of a human pupil through the light irradiation unit and the light detection unit, or by detecting the direction or amount of reflected light such as near-infrared light reflected from the cornea. For example, the light irradiation unit may include an infrared LED that irradiates infrared light to the user's eye (E), and the light detection unit may include an infrared camera that detects reflected light reflected from the user's eye (E).

[0102] In exemplary embodiments, the gaze tracking sensor (400) may be placed on the frame portion (51). For example, the gaze tracking sensor (400) may be placed on the inner side of the frame portion (51) facing the user's eye (E).

[0103] The gaze tracking sensor (400) may include a left-eye gaze tracking sensor and a right-eye gaze tracking sensor, and the left-eye gaze tracking sensor and the right-eye gaze tracking sensor may detect the gaze direction of the user's left eye and the gaze direction of the user's right eye, respectively.

[0104] The gaze tracking sensor (400) can generate gaze point data including position coordinate values ​​for the user's gaze point based on the detected gaze direction of the user's left eye and the gaze direction of the user's right eye. The 'gaze point' can refer to a point where the gaze direction of the left eye and the gaze direction of the right eye converge according to binocular disparity. The gaze tracking sensor (400) can transmit the generated gaze point data to the processor (600).

[0105] In contrast, the gaze tracking sensor (400) can transmit the gaze direction information of the user's left eye and the gaze direction information of the user's right eye to the processor (600), thereby enabling the processor (600) to generate gaze point data based on the gaze direction information of the user's left eye and the gaze direction information of the user's right eye.

[0106] Next, the communication module (500) can support the establishment of a wired or wireless communication channel between the augmented reality implementation device (10) and an external server (20), etc., and the performance of reception through the established communication channel.

[0107] In exemplary embodiments, the communication module (500) may also be provided to receive service content information from an external server (20) or the like through wired communication or wireless communication, and transmit the received service content information to the processor (600). In addition, the communication module (500) may be provided to receive the user authentication request from the external server (20) or the like through the wired communication or the wireless communication, and transmit the received user authentication request to the processor (600). In addition, the communication module (500) may be provided to receive an authentication completion signal from the processor (600), and transmit the authentication completion signal to the external server (20) or the like through the wired communication or the wireless communication.

[0108] In exemplary embodiments, the communication module (500) may include a wireless communication module or a wired communication module. The communication module (500) may communicate with an external server (20) or the like through at least one network using the wireless communication module or the wired communication module. For example, the wireless communication module may include a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module. The wired communication module may include a local area network (LAN) communication module or a power line communication module.

[0109] Next, the processor (600) can execute one or more instructions or program codes stored in the memory (700) and perform functions or operations corresponding to the instructions or program codes.

[0110] The processor (600) may be electrically or physically connected to the display module (200), the camera module (300), the gaze tracking sensor (400), and the communication module (500). The processor (600) may control the overall operation of the display module (200), the camera module (300), the gaze tracking sensor (400), and the communication module (500) by executing one or more commands or program codes stored in the memory (700).

[0111] A processor (600) may refer to a data processing device built into hardware that has a physically structured circuit to perform functions expressed in instructions or program codes. For example, the processor (600) may include, but is not limited to, a central processing unit (CPU), a microprocessor, a graphic processing unit (GRAP), an application processor (AP), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), and a field programmable gate array (FPGA).

[0112] Although the processor (600) is illustrated as being configured as a single device in FIG. 3, it is not limited thereto. In exemplary embodiments, the processor (600) may be configured as one or more devices.

[0113] Next, the memory (700) can store instructions and program codes that can be read by the processor (600). For example, the memory (700) can include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a Mask ROM, a Flash ROM, etc.), a hard disk drive (HDD), or a solid state drive (SSD).

[0114] The commands or program codes stored in the memory (700) can be implemented in a programming or scripting language such as C, C++, Java, assembler, etc.

[0115] The memory (700) can store at least one software module including the above commands or the above program code. Each software module is executed by the processor (600) to cause the augmented reality implementation device (10) to perform a predetermined operation or function. In exemplary embodiments, the memory (700) can store a service information identification module (710), an image data generation module (720), a control command generation module (730), a biometric characteristic authentication module (740), an authentication pattern authentication module (750), and a user request identification module (760), but is not limited thereto, and some of these may be stored or other software modules may be further stored.

[0116] As described above, the processor (600) can implement the following embodiments by executing software modules stored in the memory (700).

[0117] First, the processor (600) can execute a service identification module (710) stored in the memory (700) to determine whether at least one of a user authentication request using a biometric characteristic and a user authentication request using an authentication pattern is input.

[0118] In one embodiment, the communication module (500) can receive service information from an external server (20), etc. The processor (600) can identify the service information received from the external server (20), etc., by executing commands or program codes included in the service information identification module (710). The processor (600) can determine whether at least one selected from among a user authentication request using the biometric characteristic and a user authentication request using the authentication pattern is input through the identification of the service information.

[0119] For example, the external server (20) may be a payment system server of a bank. The communication module (500) may receive payment service information including a user authentication request from the payment system server of the bank. The processor (600) may identify the payment service information received from the communication module (500) and determine whether a user authentication request for user login is received. In this case, the user authentication request for user login may be a user authentication request using biometric characteristics. In addition, the processor (600) may identify the payment service information received from the communication module (500) and determine whether a user authentication request for payment is received. In this case, the user authentication request for payment may be a user authentication request using an authentication pattern.

[0120] In one embodiment, the camera module (300) can capture an authentication code. The processor (600) can identify the authentication code from the captured data received from the camera module (300) by executing commands or program codes included in the service information identification module (710). The processor (600) can determine whether at least one of a user authentication request using the biometric characteristic and a user authentication request using the authentication pattern generated through the identification of the authentication code is input.

[0121] For example, the user can take a picture of an authentication code, such as a QR code, displayed on a monitor or printed matter through the camera module (300) while wearing the augmented reality implementation device (10). The augmented reality implementation device (10) can receive the picture data from the camera module (300) and identify the authentication code from the picture data. The processor (600) can transmit authentication code identification information to an external service providing server through the communication module (500), and the external service providing server that has received the authentication code identification information can transmit the user authentication request corresponding to the authentication code identification information to the augmented reality implementation device (10). The processor (600) can receive the user authentication request from the communication module (500). Here, the user authentication request may be a user authentication request using the biometric characteristic or a user authentication request using the authentication pattern.

[0122] In one embodiment, a wearing detection sensor provided in an augmented reality implementation device (10) can detect that the augmented reality implementation device (10) is in contact with the user's face. The processor (600) can execute commands or program codes included in a service information identification module (710) to identify whether the user is wearing the augmented reality implementation device (10) based on detection information received from the wearing detection sensor. The processor (600) can determine whether at least one selected from among a user authentication request using the biometric characteristic generated through wearing identification and a user authentication request using the authentication pattern is input.

[0123] For example, when the user wears the augmented reality implementation device (10), detection information regarding the user's wearing of the augmented reality implementation device (10) may be generated by a wearing detection sensor, such as a contact sensor, provided in the augmented reality implementation device (10). The augmented reality implementation device (10) may receive the detection information from the wearing detection sensor and identify the user's wearing of the augmented reality implementation device (10) through the detection information. When the user's wearing of the augmented reality implementation device (10) is identified, a user authentication request for verifying the actual owner, which is previously stored in the memory (700), may be input to the processor (600). Here, the user authentication request may be a user authentication request using the biometric characteristic or a user authentication request using the authentication pattern.

[0124] In one embodiment, a risk detection sensor provided in an augmented reality implementation device (10) can detect the user's physical condition and the user's surrounding environment. The processor (600) can execute commands or program codes included in a service information identification module (710) to identify whether the user is exposed to a risk state based on detection information received from the risk detection sensor. The processor (600) can determine whether at least one selected from among a user authentication request using the biometric characteristic generated through the identification of exposure to a risk state and a user authentication request using the authentication pattern is input.

[0125] For example, when the user wears the augmented reality implementation device (10), detection information regarding the user's exposure to a dangerous state may be generated by wearing detection sensors such as a heart rate sensor and a temperature sensor provided in the augmented reality implementation device (10). The augmented reality implementation device (10) may receive the detection information from the danger detection sensor and identify the user's exposure to a dangerous state through the detection information. When the user's exposure to a dangerous state is identified, a user authentication request for confirming the user state previously stored in the memory (700) may be input to the processor (600). Here, the user authentication request may be a user authentication request using the biometric characteristic or a user authentication request using the authentication pattern.

[0126] As described above, when the processor (600) determines that a user authentication request using a biometric characteristic has been received, the processor (600) can execute the image data generation module (720) and the control command generation module (730) stored in the memory (700).

[0127] In addition, when the processor (600) determines that a user authentication request using an authentication pattern has been received, the processor (600) can execute the image data generation module (720) and the control command generation module (730) stored in the memory (700).

[0128] Meanwhile, the processor (600) can determine whether service content information is input by executing the service identification module (710) stored in the memory (700).

[0129] In one embodiment, the communication module (500) can receive service content information from an external server (20), etc. The processor (600) can execute commands or program codes included in the service information identification module (710) to determine whether the service content information is input.

[0130] For example, the external server (20) may be a bank's payment system server. The communication module (500) may receive payment service information, including service content information regarding user account information, from the bank's payment system server. The processor (600) may identify the payment service information received from the communication module (500) and determine whether service content information regarding user account information is being input.

[0131] As described above, when the processor (600) determines that service content information has been input, the processor (600) can execute at least one selected from among the image data generation module (720) and the control command generation module (730) stored in the memory (700) and the user request identification module (760).

[0132] Next, the processor (600) can execute an image data generation module (720) stored in the memory (700) to generate image data representing a virtual image regarding service information.

[0133] The image data generation module (720) may include instructions or program codes relating to an algorithm for generating image data including pixel values ​​for a plurality of pixels. The processor (600) may generate image data representing a virtual image of service information by executing the instructions or program codes included in the image data generation module (720).

[0134] The processor (600) may generate second data based on the first data by executing commands or program codes included in the image data generation module (720). For example, the first data may include a user authentication request using a biometric characteristic, a user authentication request using an authentication pattern, gaze point data, service content information, identification data for an operation object (HO), and location data for the operation object (HO). The second data may include image data for expressing various arrangements, shapes, colors, texts, etc. of a virtual image related to the service information.

[0135] In exemplary embodiments, the processor (600) may generate authentication request image data representing a virtual authentication request image (VCI) based on a user authentication request using a biometric characteristic.

[0136] Here, the virtual authentication request image (VCI) represented by the authentication request image data may include various shapes and texts for conveying information about user authentication using biometric characteristics to the user.

[0137] As illustrated in FIG. 6, the virtual authentication request image (VCI) represented by the authentication request image data may include text for an authentication request using biometric characteristics. Furthermore, the virtual authentication request image (VCI) represented by the authentication request image data may include an authentication object guide image (OG) corresponding to the shape of the authentication object (CO). For example, the authentication object guide image (OG) may be in the form of a figure corresponding to the shape of the user's hand or arm, which is the authentication object (CO).

[0138] In this way, by displaying the authentication object guide image (OG) to the user, the user can place the authentication object (CO), such as a hand or arm, in a position where the camera module (300) can easily capture the image.

[0139] Next, the processor (600) can generate authentication pattern image data representing a virtual authentication pattern image (VOI) based on a user authentication request using an authentication pattern.

[0140] Here, the virtual authentication pattern image (VOI) expressed by the authentication pattern image data may include various shapes and texts to convey information about user authentication using the authentication pattern to the user.

[0141] As illustrated in FIG. 7, the virtual authentication pattern image (VOI) represented by the authentication pattern image data may include text for an authentication request using the authentication pattern. Furthermore, the virtual authentication pattern image (VOI) represented by the authentication pattern image data may include a plurality of path point images (PP). For example, each of the plurality of path point images (PP) may be circular in shape.

[0142] Here, the processor (600) may generate gaze point image data representing a virtual gaze point image (VPI) based on the gaze point data acquired by the gaze tracking sensor (400) or the gaze point data acquired by executing the authentication pattern authentication module (750) described below. Alternatively, the processor (600) may also generate gaze point image data representing a virtual gaze point image (VPI) based on the gaze point data input from the gaze tracking sensor (400).

[0143] Here, the virtual gaze point image (VPI) represented by the gaze point image data may be displayed at the point where the user is looking. As illustrated in Fig. 7, the virtual gaze point image (VPI) represented by the gaze point image data may be circular in shape.

[0144] Additionally, the processor (600) may generate gaze point image data representing a virtual gaze point image (VPI) moving in response to the movement of the user's gaze based on the gaze point data. Accordingly, when the user's gaze point location changes, the virtual gaze point image (VPI) may also be displayed at the changed gaze point location of the user.

[0145] Meanwhile, the processor (600) can generate gaze point trajectory image data representing a virtual gaze point trajectory image (VTI) based on the gaze point data.

[0146] Specifically, the processor (600) can generate the gaze point trajectory image data representing that the virtual gaze point trajectory image (VTI) is generated in response to the movement path of the user's gaze point. As illustrated in FIG. 7, the virtual gaze point trajectory image (VTI) represented by the gaze point trajectory image data may be in the form of a line formed in response to the movement path of the user's gaze point, or may be in the form of a figure having a constant area.

[0147] Meanwhile, the processor (600) can simultaneously generate the gaze point image data representing a virtual gaze point image (VPI) and the gaze point trajectory image data representing a virtual gaze point trajectory image (VTI) based on the gaze point data.

[0148] By displaying a virtual gaze point image (VPI) and a virtual gaze trajectory image (VTI), users can easily identify where their gaze is located. Furthermore, users can easily identify the shape of their gaze trajectory.

[0149] Next, the processor (600) can generate service content image data representing a virtual service content image (VSI) based on the service content information.

[0150] Here, the virtual service content image (VSI) expressed by the service content image data may include various shapes and texts to convey information about the service to the user.

[0151] For example, as illustrated in FIG. 8, a virtual service content image (VSI) represented by service content image data may include text about the service content to be performed. Furthermore, the virtual service content image (VSI) represented by the service content image data may include multiple service menu images (SM).

[0152] Meanwhile, the processor (600) can generate service content image data representing that a virtual service content image (VSI) is positioned adjacent to an identified operation object (HO) based on the acquired operation object location data by executing a user request identification module (760) described later.

[0153] In exemplary embodiments, the processor (600) may generate service content image data representing that a portion of a virtual service content image (VSI) is positioned within an area of ​​an identified manipulation object (HO). Alternatively, the processor (600) may generate service content image data representing that a virtual service content image (VSI) is positioned at a predetermined distance from the identified manipulation object (HO) (e.g., at a location within reach of a user's hand).

[0154] Next, the processor (600) can generate instruction image data representing a virtual instruction image (VII) based on the identification data for the obtained operation object (HO) by executing the user request identification module (760) described below.

[0155] Here, the virtual instruction image (VII) represented by the instruction image data may have various shapes and colors, and may also include text. In exemplary embodiments, the processor (600) may generate instruction image data representing that the virtual instruction image (VII) includes an instruction line image (VI-1) and a cursor image (VI-2).

[0156] For example, as illustrated in Fig. 9, the guide line image (VI-1) may be expressed in the form of a line having a preset length and thickness. In addition, the cursor image (VI-2) may be expressed in the form of various shapes having a preset area.

[0157] Meanwhile, the processor (600) can generate a pointer image data that represents that the pointer image (VI-1) is positioned to extend in a specific direction in a part of the manipulation object (HO) and that the cursor image (VI-2) is positioned at the end of the pointer image (VI-1) based on the manipulation object position data obtained by executing the user request identification module (760) described below.

[0158] For example, as illustrated in FIG. 9, the guide line image (VI-1) may be positioned in a direction extending from the user's fingertip to the finger placement, and the cursor image (VI-2) may be expressed as being positioned at the end of the guide line image (VI-1).

[0159] In exemplary embodiments, the processor (600) may generate instruction image data representing that a virtual instruction image (VII) moves in response to the movement of an identified operation object (HO) based on the operation object position data.

[0160] For example, as illustrated in FIG. 9, when a manipulation object (HO) is first identified, the processor (600) can generate guidance image data representing that a guidance line image (VI-1) is positioned to extend in a specific direction in a portion of the manipulation object (HO) and a cursor image (VI-2) is positioned at an end of the guidance line image (VI-1). Thereafter, the processor (600) can generate guidance image data representing that the guidance line image (VI-1) and the cursor image (VI-2) move in response to the movement of the identified manipulation object (HO), based on the manipulation object position data.

[0161] Meanwhile, the processor (600) can generate instruction image data representing that a virtual instruction image (VII) is arranged to extend in the direction of a virtual service content image (VSI) based on the manipulation object position data.

[0162] For example, when a manipulation object (HO) is first identified as illustrated in FIG. 10, the processor (600) may generate guidance image data representing that a guidance line image (VI-1) extends in a direction of a virtual service content image (VSI) in a portion of the manipulation object (HO), an end of the guidance line image (VI-1) is positioned adjacent to the virtual service content image (VSI), and a cursor image (VI-2) is positioned at the end of the guidance line image (VI-1). At this time, the processor (600) may generate guidance image data representing that an end of the guidance line image (VI-1) is positioned at a position spaced apart from the virtual service content image (VSI) by a certain distance, and a cursor image (VI-2) is positioned at a position spaced apart from the virtual service content image (VSI) by a certain distance. Alternatively, the processor (600) may generate pointer image data that represents that an end of the pointer image (VI-1) is positioned on a virtual service content image (VSI), and that the cursor image (VI-2) is positioned within an area of ​​the virtual service content image (VSI).

[0163] Meanwhile, the processor (600) can calculate a distance (D) between the virtual service content image (VSI) and the identified operation object (HO) based on the display position and operation object position data of the virtual service content image (VSI) described above, and generate instruction image data expressing different virtual instruction images (VII) according to the calculated distance (D).

[0164] In exemplary embodiments, the processor (600) may generate a virtual pointer image (VII) representing only a cursor image (VI-2) when the distance (D) between the displayed virtual service content image (VSI) and the identified operation object (HO) is less than or equal to a preset distance. Accordingly, as illustrated in FIG. 11, when the distance (D) between the displayed virtual service content image (VSI) and the identified operation object (HO) is less than or equal to a preset distance, only the cursor image (VI-2) may be displayed.

[0165] In exemplary embodiments, the processor (600) may generate a virtual pointer image (VII) including both a pointer image (VI-1) and a cursor image (VI-2) when the distance (D) between the displayed virtual service content image (VSI) and the identified operation object (HO) exceeds a preset distance. Accordingly, as illustrated in FIG. 12, when the distance (D) between the displayed virtual service content image (VSI) and the identified operation object (HO) exceeds a preset distance, both the pointer image (VI-1) and the cursor image (VI-2) may be displayed. Through this, even when the displayed virtual service content image (VSI) is located far away from the user, the user can accurately identify the contact point of the service content image (VSI) that the user is pointing to through the pointer image (VI-1).

[0166] Meanwhile, as described above, the processor (600) may generate service content image data representing that a virtual service content image (VSI) is positioned adjacent to an identified manipulation object (HO) based on the manipulation object position data acquired by executing the user request identification module (760) described below. In this way, when the virtual service content image (VSI) is expressed as being positioned adjacent to the identified manipulation object (HO), and the distance (D) between the displayed virtual service content image (VSI) and the identified manipulation object (HO) is less than or equal to a preset distance, the processor (600) may generate instruction image data representing a virtual instruction image (VII) that includes only a cursor image (VI-2).

[0167] In this way, by displaying a virtual instruction image (VII) to the user, the user can perform a service request by focusing his / her gaze on the virtual instruction image (VII) rather than on the manipulation object (HO) such as the hand.

[0168] When the processor (600) generates the authentication request image data, the authentication pattern image data, the gaze point image data, the gaze point trajectory image data, the service content image data, and the instruction image data as described above, the processor (600) can transmit the generated image data to the display module (200).

[0169] The processor (600) can transmit the generated authentication request image data to the display module (200), so that the display module (200) can emit image light that displays a virtual authentication request image (VCI) based on the authentication request image data. In addition, the processor (600) can transmit the generated authentication pattern image data to the display module (200), so that the display module (200) can emit image light that displays a virtual authentication pattern image (VOI) based on the authentication pattern image data. In addition, the processor (600) can transmit the generated gaze point image data to the display module (200), so that the display module (200) can emit image light that displays a virtual gaze point image (VPI) based on the gaze point image data. In addition, the processor (600) can transmit the generated gaze point trajectory image data to the display module (200), so that the display module (200) can emit image light that displays a virtual gaze point trajectory image (VTI) based on the gaze point trajectory image data. In addition, the processor (600) can transmit the generated service content image data to the display module (200), so that the display module (200) can emit image light that displays a virtual service content image (VSI) based on the service content image data. In addition, the processor (600) can transmit the generated instruction image data to the display module (200), so that the display module (200) can emit image light that displays a virtual instruction image (VII) based on the instruction image data.

[0170] Next, the processor (600) may execute a control command generation module (730) stored in the memory (700) to generate a control signal for controlling the operation of at least one selected from among the camera module (300) and the gaze tracking sensor (400) according to the input user authentication request.

[0171] The control command generation module (730) may include instructions or program codes regarding an algorithm for generating a control signal for controlling the operation of a specific electronic device. The processor (600) may generate a control signal for controlling the operation of at least one selected from among the camera module (300) and the gaze tracking sensor (400) based on an input user authentication request by executing the instructions or program codes included in the control command generation module (730).

[0172] In exemplary embodiments, when a user authentication request using a biometric characteristic is received, the processor (600) may generate a control signal that causes the camera module (300) to operate. The processor (600) may transmit the generated control signal to the camera module (300), and the camera module (300) may initiate a photographing operation according to the control signal and generate photographing data.

[0173] For example, when the processor (600) receives a user authentication request using a biometric characteristic, the camera module (300) can start a photographing operation by the control signal and photograph an authentication object (CO), such as the user's hand.

[0174] In addition, when the processor (600) receives a user authentication request using an authentication pattern, it can generate a control signal that causes the gaze tracking sensor (400) to operate. The processor (600) can transmit the generated control signal to the gaze tracking sensor (400), and the gaze tracking sensor (400) can start a sensing operation according to the control signal to obtain information on the user's gaze direction.

[0175] For example, when the processor (600) receives a user authentication request using an authentication pattern, the gaze tracking sensor (400) can initiate a sensing operation by the control signal to obtain the user's gaze direction information. After obtaining the user's gaze direction information, the gaze tracking sensor (400) can generate the gaze point data for the user based on the user's gaze direction information.

[0176] In addition, the processor (600) can generate a control signal for controlling the operation of at least one selected from among the camera module (300) and the gaze tracking sensor (400) based on the input service content information by executing commands or program codes included in the control command generation module (730).

[0177] In exemplary embodiments, when service content information is input, the processor (600) may generate a control signal that causes the camera module (300) to operate. The processor (600) may transmit the generated control signal to the camera module (300), and the camera module (300) may initiate a shooting operation according to the control signal and generate shooting data.

[0178] For example, when the processor (600) receives service content information, the camera module (300) can start a shooting operation by the control signal and can shoot an operation object (HO), such as the user's hand.

[0179] In addition, when the processor (600) receives service content information, it can generate a control signal that causes the gaze tracking sensor (400) to operate. The processor (600) can transmit the generated control signal to the gaze tracking sensor (400), and the gaze tracking sensor (400) can start a sensing operation according to the control signal to obtain the user's gaze direction information.

[0180] For example, when the processor (600) receives service content information, the gaze tracking sensor (400) can initiate a sensing operation by the control signal to obtain the user's gaze direction information. After obtaining the user's gaze direction information, the gaze tracking sensor (400) can generate the gaze point data for the user based on the user's gaze direction information.

[0181] Meanwhile, the processor (600) can transmit a control signal to the camera module (300) to cause the camera module (300) to operate, and then receive shooting data from the camera module (300). When the processor (600) receives shooting data from the camera module (300), it can execute at least one selected from the biometric characteristic authentication module (740) and the user request identification module (760) stored in the memory (700).

[0182] In addition, the processor (600) may transmit a control signal to the gaze tracking sensor (400) to cause the gaze tracking sensor (400) to operate, and then receive the user's gaze direction information or the gaze point data from the gaze tracking sensor (400). When the processor (600) receives the user's gaze direction information or the gaze point data from the gaze tracking sensor (400), the processor (600) may execute at least one selected from the image data generation module (720) and the authentication pattern authentication module (750) stored in the memory (700).

[0183] Next, the processor (600) can execute the biometric authentication module (740) stored in the memory (700) to identify the biometric characteristics of the authentication object (CO) from the photographed data input from the camera module (300). In addition, the processor (600) can determine whether the identified biometric characteristics match the previously stored biometric authentication data. If the processor (600) determines that the identified biometric characteristics match the previously stored biometric authentication data, the processor (600) can generate an authentication completion signal using the biometric characteristics.

[0184] The biometric authentication module (740) may include commands or program codes related to known image processing algorithms, such as object detection algorithms. In addition, the biometric authentication module (740) may include commands or program codes related to known biometric recognition algorithms, such as vein pattern recognition algorithms and iris pattern recognition algorithms.

[0185] The processor (600) can identify an authentication object (CO) from the photographed data input from the camera module (300) by executing commands or program codes included in the biometric authentication module (740). In addition, the processor (600) can identify the biometric characteristics of the authentication object (CO) based on the identified biometric characteristics from the photographed data input from the camera module (300), and if it is determined that the identified biometric characteristics match the previously stored biometric authentication data, it can generate an authentication completion signal using the biometric characteristics.

[0186] For example, the authentication object (CO) identified by the processor (600) may be the user's hand or the user's arm. In addition, the biometric characteristic of the authentication object (CO) identified by the processor (600) may be a vein pattern. The processor (600) may determine whether the identified biometric characteristic, that is, the vein pattern, matches biometric authentication data regarding pre-stored vein patterns. If the processor (600) determines that the identified vein pattern matches the pre-stored vein pattern data, the processor (600) may generate an authentication completion signal using the biometric characteristic.

[0187] In addition, the processor (600) can identify an authentication object (CO) from the photographed data input from the gaze tracking sensor (400) by executing commands or program codes included in the biometric authentication module (740). In addition, the processor (600) can identify the biometric characteristics of the authentication object (CO) based on the authentication object (CO) identified from the photographed data input from the gaze tracking sensor (400), and if it is determined that the identified biometric characteristics match the previously stored biometric authentication data, it can generate an authentication completion signal using the biometric characteristics.

[0188] For example, the authentication object (CO) identified by the processor (600) from the photographed data input from the gaze tracking sensor (400) may be the iris of the user's eye. In addition, the biometric characteristic of the authentication object (CO) identified by the processor (600) may be an iris pattern. The processor (600) may determine whether the identified biometric characteristic, that is, the iris pattern, matches biometric authentication data regarding a previously stored iris pattern. If the processor (600) determines that the identified iris pattern matches the previously stored iris pattern data, the processor (600) may generate an authentication completion signal using the biometric characteristic.

[0189] Meanwhile, the processor (600) may generate an authentication completion signal using a biometric characteristic and then transmit the authentication completion signal using the biometric characteristic to the communication module (500). The communication module (500) may transmit the received authentication completion signal using the biometric characteristic to an external server (20), etc. For example, if the completion of user authentication is performed at the request of a bank's payment system server, the authentication completion signal may be transmitted to the bank's payment system server through the communication module (500). By transmitting the authentication completion signal to the bank's payment system server, the next step of the service provided by the bank's payment system server may proceed.

[0190] In addition, when an authentication completion signal using biometric characteristics is generated, the processor (600) can transmit a notification signal to an external agency terminal or an external user terminal via the communication module (500). For example, when the completion of user authentication is performed because the user is determined to be in a dangerous state, the processor (600) can transmit a rescue request signal via the communication module (500) to a government agency terminal such as a fire station or police station or a terminal corresponding to a previously stored contact.

[0191] Next, the processor (600) may execute the authentication pattern authentication module (750) stored in the memory (700) to calculate the user's gaze point trajectory based on the user's gaze direction information input from the gaze tracking sensor (400). In addition, the processor (600) may determine whether the calculated gaze point trajectory matches the previously stored authentication pattern data. If the processor (600) determines that the calculated gaze point trajectory matches the previously stored authentication pattern data, the processor (600) may generate an authentication completion signal using the authentication pattern.

[0192] The authentication pattern authentication module (750) may include instructions or program codes for an algorithm that calculates information about the user's gaze point based on the user's gaze direction data. The processor (600) may generate the gaze point data from the user's gaze direction information input from the gaze tracking sensor (400) by executing the instructions or program codes included in the authentication pattern authentication module (750). Alternatively, the processor (600) may receive the gaze point data generated by the gaze tracking sensor (400). The processor (600) may calculate the user's gaze point trajectory based on the generated gaze point data, and if it is determined that the calculated gaze point trajectory matches the previously stored authentication pattern data, it may generate an authentication completion signal using the authentication pattern.

[0193] For example, the processor (600) can determine whether the shape of the gaze point trajectory calculated based on the gaze point data matches the path shape expressed by the authentication pattern data previously stored in an external server (20), etc. If the shape of the user's gaze point trajectory matches the path shape expressed by the authentication pattern data, the processor (600) can generate an authentication completion signal using the authentication pattern.

[0194] Here, the processor (600) can perform user authentication using the authentication pattern described above after the user makes a pattern input completion gesture. The processor (600) determines whether a pattern input completion gesture is identified from the user, and if a pattern input completion gesture is identified from the user, can perform user authentication using the authentication pattern. That is, the processor (600) calculates a gaze point trajectory of the user based on gaze point data up to the point at which the user makes the pattern input completion gesture, and if it is determined that the calculated gaze point trajectory matches the previously stored authentication pattern data, it can generate an authentication completion signal using the authentication pattern. At this time, the user's pattern input completion gesture may be a motion related to the user's eyes. For example, the user's pattern input completion gesture may be a state in which the user moves his gaze, inputs an authentication pattern, and then closes his eyes for several seconds, repeatedly, several times. The processor (600) can determine whether the user makes a pattern input completion gesture through the gaze tracking sensor (400). When the processor (600) determines that the user has made a pattern input completion gesture, it can calculate the user's gaze point trajectory based on gaze point data up to the point at which the user has made the pattern input completion gesture, and determine whether the calculated gaze point trajectory matches the previously stored authentication pattern data.

[0195] Meanwhile, the processor (600) can generate an authentication completion signal using an authentication pattern and then transmit the authentication completion signal using the authentication pattern to the communication module (500). The communication module (500) can transmit the authentication completion signal using the received authentication pattern to an external server (20), etc. For example, if the completion of user authentication is performed at the request of the bank's payment system server, the authentication completion signal can be transmitted to the bank's payment system server through the communication module (500). By transmitting the authentication completion signal to the bank's payment system server, the next step of the service provided by the bank's payment system server can proceed.

[0196] In addition, when an authentication completion signal using an authentication pattern is generated, the processor (600) can transmit a notification signal to an external agency terminal or an external user terminal via the communication module (500). For example, when the completion of user authentication is performed because the user is determined to be in a dangerous state, the processor (600) can transmit a rescue request signal to a government agency terminal such as a fire station or police station or a terminal corresponding to a previously stored contact via the communication module (500).

[0197] Next, the processor (600) can execute a user request identification module (760) stored in the memory (700) to determine whether a manipulation object (HO), which is a real object, is identified in the shooting data received from the camera module (300).

[0198] The user request identification module (760) may include commands or program codes related to a known image processing algorithm, such as an object detection algorithm. The processor (600) may determine whether the manipulation object (HO) is identified in the photographed data received from the camera module (300) by executing the commands or program codes included in the user request identification module (760). If the processor (600) identifies the manipulation object (HO) from the photographed data received, it may generate identification data for the corresponding manipulation object (HO).

[0199] In exemplary embodiments, the manipulation object (HO) identified by the processor (600) from the shooting data received from the camera module (300) may be the user's hand.

[0200] Next, the processor (600) can obtain the location information of the identified manipulation object (HO) by executing the commands or program codes included in the user request identification module (760), and can generate manipulation object location data including the location coordinate values ​​of the identified manipulation object (HO). If the identified manipulation object (HO) moves, the processor (600) can generate manipulation object location data including the location coordinate values ​​corresponding to the movement of the identified manipulation object (HO).

[0201] As described above, when the processor (600) identifies a manipulation object (HO) in the input shooting data, the processor (600) can execute the image data generation module (720) described above.

[0202] Meanwhile, the processor (600) can execute commands or program codes included in the user request identification module (760) to determine whether the manipulation gesture of the manipulation object (HO) is identified in the shooting data received from the camera module (300).

[0203] When the processor (600) identifies a manipulation gesture from the input photographed data, it can generate identification data for the manipulation gesture. Here, after identifying the manipulation object (HO), the processor (600) can determine whether a manipulation gesture of the manipulation object (HO) is identified from the input photographed data.

[0204] In exemplary embodiments, the manipulation gestures identified by the processor (600) may include hand gestures expressed in the shape of the user's hand, hand gestures expressed in the trajectory drawn by the center point of the user's hand, and hand gestures expressed in the combination of the shape of the user's hand and the trajectory drawn by the center point of the hand.

[0205] For example, the manipulation gestures identified by the processor (600) may include a 'tap' in which a finger is tapped once, a 'two finger tap' in which a finger is tapped once, a 'double tap' in which a finger is tapped twice, a 'press' in which a finger is pressed firmly, a 'drag' in which a finger is pressed firmly and then moved in a specific direction, a 'pinch out' and a 'pinch in' in which two fingers are moved in different directions, a fist clenched N or more times (where N is a natural number greater than or equal to 1), etc.

[0206] As described above, when the processor (600) identifies the manipulation gesture from the photographed data, the processor (600) can generate a user request signal based on the identification data for the manipulation gesture.

[0207] Meanwhile, the processor (600) can generate a user request signal when it identifies an operation gesture while the virtual instruction image (VII) is placed on the virtual service content image (VSI).

[0208] For example, if the processor (600) identifies an operation gesture while the cursor image (VI-2) is placed on a virtual service content image (VSI), the processor (600) may generate a user request signal. In this case, if the virtual service content image (VSI) includes a plurality of service menu images (SM), the processor (600) may generate a user request signal if the processor (600) identifies an operation gesture while the cursor image (VI-2) is placed on any one of the plurality of service menu images (SM).

[0209] Meanwhile, the processor (600) can determine whether a manipulation gesture of a manipulation object (HO) is identified in the photographed data input from the gaze tracking sensor (400) by executing commands or program codes included in the user request identification module (760). If the processor (600) identifies a manipulation gesture of a manipulation object (HO) in the photographed data input from the gaze tracking sensor (400), it can generate identification data for the corresponding manipulation gesture.

[0210] In exemplary embodiments, the manipulation object (HO) identified by the processor (600) from the captured data received from the gaze tracking sensor (400) may be the user's eye. Additionally, the manipulation gesture of the manipulation object (HO) identified by the processor (600) from the captured data received from the gaze tracking sensor (400) may include the user's eye blinking motion.

[0211] For example, the manipulation gesture identified by the processor (600) may include a motion in which the user blinks his eyes several times, a motion in which the user closes his eyes for several seconds and then opens them, etc.

[0212] When the processor (600) identifies a manipulation gesture of a manipulation object (HO) from the photographed data received from the gaze tracking sensor (400), the processor (600) can generate a user request signal based on the identification data for the manipulation gesture.

[0213] As described above, the user can make a user request for the provided service content information through the viewing point.

[0214] Meanwhile, as described above, the processor (600) can generate gaze point image data representing a virtual gaze point image (VPI) based on gaze point data. Accordingly, a virtual service content image (VSI) and a virtual gaze point image (VPI) can be displayed together to the user.

[0215] At this time, the processor (600) can generate a user request signal when it identifies an operation gesture while the gaze point image (VPI) is placed on the virtual service content image (VSI).

[0216] For example, if the processor (600) identifies the manipulation gesture while the gaze point image (VPI) is positioned on the virtual service content image (VSI), the processor (600) may generate a user request signal. In this case, if the virtual service content image (VSI) includes a plurality of service menu images (SM), the processor (600) may generate the user request signal while identifying the manipulation gesture while the gaze point image (VPI) is positioned on any one of the plurality of service menu images (SM).

[0217] Meanwhile, the processor (600) can transmit the generated user request signal to the communication module (500), so that the communication module (500) can transmit the user request signal to an external server (20), etc.

[0218] For example, if the completion of a user request is performed at the request of a bank's payment system server, a user request signal may be transmitted to the bank's payment system server via the communication module (500). By transmitting the user request signal to the bank's payment system server, the next step of the service provided by the bank's payment system server may be performed.

[0219] Below, a method for performing user authentication using the augmented reality implementation device of Fig. 3 is described.

[0220] Fig. 13 is a flowchart illustrating a user authentication method using an augmented reality implementation device according to exemplary embodiments.

[0221] Referring to FIGS. 1 to 13, first, the processor (600) of the augmented reality implementation device (10) determines whether a user authentication request using a biometric characteristic is input (S1310), and if a user authentication request using a biometric characteristic is input, the processor generates a control signal for controlling the camera module (300) to cause the camera module (300) to operate (S1320).

[0222] Specifically, the processor (600) can determine whether a user authentication request using a biometric characteristic is input by executing commands or program codes included in the service information identification module (710) to identify service information input from the communication module (500). In addition, the processor (600) can determine whether a user authentication request using a biometric characteristic is input by executing commands or program codes included in the service information identification module (710) to identify the authentication code from the photographed data received from the camera module (300). In addition, the processor (600) can determine whether a user authentication request using a biometric characteristic is input by executing commands or program codes included in the service information identification module (710) to identify the user's wearing of the augmented reality implementation device (10) from the detection information received from the wearing detection sensor. In addition, the processor (600) can determine whether a user authentication request using biometric characteristics is input by executing commands or program codes included in the service information identification module (710) to identify whether the user is exposed to a risk state from the detection information received from the risk detection sensor.

[0223] When the processor (600) determines that a user authentication request using a biometric characteristic has been input, the processor (600) can execute commands or program codes included in the control command generation module (730), and the processor (600) can generate a control signal for controlling the operation of the camera module (300) based on the input user authentication request using a biometric characteristic. The processor (600) can transmit the generated control signal to the camera module (300), and the camera module (300) can start a shooting operation according to the control signal and generate shooting data.

[0224] Meanwhile, after the camera module (300) operates according to the control signal transmitted from the processor (600), the processor (600) can receive shooting data from the camera module (300). When the processor (600) receives shooting data from the camera module (300), it can execute the biometric characteristic authentication module (740) stored in the memory (700).

[0225] Next, the processor (600) of the augmented reality implementation device (10) identifies the biometric characteristics of the authentication object (CO) from the captured data generated by the operation of the camera module (300) (S1330), determines whether the biometric characteristics match the previously stored biometric authentication data (S1340), and if the biometric characteristics match the biometric authentication data, generates an authentication completion signal using the biometric characteristics (S1350).

[0226] Specifically, the processor (600) can identify an authentication object (CO) from the photographed data input from the camera module (300) by executing commands or program codes included in the biometric authentication module (740). In addition, the processor (600) can identify a biometric characteristic of the authentication object (CO) based on the identified authentication object (CO), and if it is determined that the identified biometric characteristic matches pre-stored biometric authentication data, it can generate an authentication completion signal using the biometric characteristic.

[0227] For example, the authentication object (CO) identified by the processor (600) from the photographed data input from the camera module (300) may be the user's hand or the user's arm. In addition, the biometric characteristic of the authentication object (CO) identified by the processor (600) may be a vein pattern. The processor (600) may determine whether the identified biometric characteristic, that is, the vein pattern, matches biometric authentication data regarding pre-stored vein patterns. If the processor (600) determines that the identified vein pattern matches the pre-stored vein pattern data, the processor (600) may generate an authentication completion signal using the biometric characteristic.

[0228] Meanwhile, the processor (600) of the augmented reality implementation device (10) generates an authentication completion signal using a biometric characteristic (S1350), and then transmits the generated authentication completion signal using a biometric characteristic to the communication module (500), so that the communication module (500) can transmit the authentication completion signal using a biometric characteristic to an external server (20) (S1360).

[0229] In addition, when an authentication completion signal using biometric characteristics is generated, the processor (600) may transmit a notification signal to an external agency terminal or an external user terminal via the communication module (500). For example, when user authentication is completed because the user is determined to be in a dangerous state, the processor (600) may transmit a rescue request signal to a government agency terminal such as a fire station or police station or a terminal corresponding to a previously stored contact via the communication module (500).

[0230] Meanwhile, in another embodiment, the processor (600) of the augmented reality implementation device (10) determines whether a user authentication request using a biometric characteristic is input, and if a user authentication request using a biometric characteristic is input, generates a control signal for controlling the gaze tracking sensor (400) to cause the gaze tracking sensor (400) to operate.

[0231] When the processor (600) determines that a user authentication request using a biometric characteristic has been input, the processor (600) can execute commands or program codes included in the control command generation module (730), and the processor (600) can generate a control signal for controlling the operation of the gaze tracking sensor (400) based on the input user authentication request using a biometric characteristic. The processor (600) can transmit the generated control signal to the gaze tracking sensor (400), and the gaze tracking sensor (400) can start a shooting operation according to the control signal to generate shooting data.

[0232] Meanwhile, after the gaze tracking sensor (400) operates according to the control signal transmitted from the processor (600), the processor (600) can receive photographing data from the gaze tracking sensor (400). When the processor (600) receives photographing data from the gaze tracking sensor (400), the processor (600) can execute the biometric characteristic authentication module (740) stored in the memory (700).

[0233] Next, the processor (600) of the augmented reality implementation device (10) can identify the biometric characteristics of the authentication object (CO) from the captured data generated by the operation of the gaze tracking sensor (400), determine whether the biometric characteristics match the previously stored biometric authentication data, and if the biometric characteristics match the biometric authentication data, generate an authentication completion signal using the biometric characteristics.

[0234] Specifically, the processor (600) can identify an authentication object (CO) from the photographed data input from the camera module (300) by executing commands or program codes included in the biometric authentication module (740). In addition, the processor (600) can identify a biometric characteristic of the authentication object (CO) based on the identified authentication object (CO), and if it is determined that the identified biometric characteristic matches pre-stored biometric authentication data, it can generate an authentication completion signal using the biometric characteristic.

[0235] For example, the authentication object (CO) identified by the processor (600) from the photographed data input from the gaze tracking sensor (400) may be the iris of the user's eye. In addition, the biometric characteristic of the authentication object (CO) identified by the processor (600) may be an iris pattern. The processor (600) may determine whether the identified biometric characteristic, that is, the iris pattern, matches biometric authentication data regarding a previously stored iris pattern. If the processor (600) determines that the identified iris pattern matches the previously stored iris pattern data, the processor (600) may generate an authentication completion signal using the biometric characteristic.

[0236] Fig. 14 is a flowchart illustrating a user authentication method using an augmented reality implementation device according to exemplary embodiments.

[0237] Referring to FIG. 14, the processor (600) of the augmented reality implementation device (10) determines whether a user authentication request using a biometric characteristic is input (S1310), and if a user authentication request using a biometric characteristic is input, generates authentication request image data representing a virtual authentication request image (VCI) (S1410), and causes the display module (200) to emit image light displaying the virtual authentication request image (VCI) toward the optical lens (100) based on the authentication request image data (S1420).

[0238] Specifically, the processor (600) can generate authentication request image data representing a virtual authentication request image (VCI) based on a user authentication request using a biometric characteristic by executing commands or program code included in the image data generation module (720).

[0239] The processor (600) can transmit the generated authentication request image data to the display module (200), and the display module (200) can emit image light toward the optical lens (100) to display a virtual authentication request image (VCI) based on the authentication request image data.

[0240] Meanwhile, a virtual authentication request image (VCI) represented by authentication request image data may include an authentication object guide image (OG).

[0241] Fig. 15 is a flowchart illustrating a user authentication method using an augmented reality implementation device according to exemplary embodiments.

[0242] Referring to FIG. 15, the processor (600) of the augmented reality implementation device (10) determines whether a user authentication request using an authentication pattern is input (S1510), and if a user authentication request using the authentication pattern is input, generates authentication pattern image data representing a virtual authentication pattern image (VOI) (S1520), and causes the display module (200) to emit image light displaying the virtual authentication pattern image (VOI) toward the optical lens (100) based on the authentication pattern image data (S1530).

[0243] Specifically, the processor (600) can execute commands or program codes included in the service information identification module (710), and determine whether a user authentication request using the authentication pattern is input by identifying the service information received from the communication module (500). In addition, the processor (600) can execute commands or program codes included in the service information identification module (710) to identify the authentication code from the photographed data received from the camera module (300), thereby determining whether a user authentication request using a biometric characteristic is input. In addition, the processor (600) can execute commands or program codes included in the service information identification module (710) to identify the user's wearing of the augmented reality implementation device (10) from the detection information received from the wearing detection sensor, thereby determining whether a user authentication request using a biometric characteristic is input. In addition, the processor (600) can determine whether a user authentication request using biometric characteristics is input by executing commands or program codes included in the service information identification module (710) to identify whether the user is exposed to a risk state from the detection information received from the risk detection sensor.

[0244] When the processor (600) determines that a user authentication request using the authentication pattern has been input, the processor (600) can execute commands or program codes included in the image data generation module (720) and generate authentication pattern image data representing a virtual authentication pattern image (VOI) based on the user authentication request using the authentication pattern.

[0245] The processor (600) can transmit the generated authentication pattern image data to the display module (200), and the display module (200) can emit image light toward the optical lens (100) to display a virtual authentication pattern image (VOI) based on the authentication pattern image data.

[0246] Fig. 16 is a flowchart illustrating a user authentication method using an augmented reality implementation device according to exemplary embodiments.

[0247] Referring to FIG. 16, the processor (600) of the augmented reality implementation device (10) determines whether a user authentication request using an authentication pattern is input (S1510), and if a user authentication request using an authentication pattern is input, generates a control signal for controlling the gaze tracking sensor (400) to operate the gaze tracking sensor (400) (S1610), generates user gaze point data based on the user's gaze direction information generated by the operation of the gaze tracking sensor (400) (S1620), generates gaze point image data for expressing a virtual gaze point image (VPI) based on the gaze point data (S1630), and causes the display module (200) to emit image light displaying the virtual gaze point image (VPI) based on the gaze point image data toward the optical lens (100) (S1640).

[0248] In contrast, the processor (600) of the augmented reality implementation device (10) may generate gaze point image data for expressing a gaze point point image based on the gaze point data received from the gaze tracking sensor (400).

[0249] Specifically, when the processor (600) determines that a user authentication request using an authentication pattern has been input, the processor (600) may execute commands or program codes included in the image data generation module (720), and the processor (600) may generate gaze point image data representing a virtual gaze point image (VPI) based on the gaze point data. Alternatively, the processor (600) may generate gaze point image data representing a virtual gaze point image (VPI) based on the gaze point data received from the gaze tracking sensor (400). In addition, the processor (600) may generate gaze point image data representing that the virtual gaze point image (VPI) moves in response to the movement of the user's gaze based on the gaze point data.

[0250] The processor (600) can transmit the generated gaze point image data to the display module (200), and the display module (200) can emit image light toward the optical lens (100) to display a virtual gaze point image (VPI) based on the gaze point image data.

[0251] Fig. 17 is a flowchart illustrating a user authentication method using an augmented reality implementation device according to exemplary embodiments.

[0252] Referring to FIG. 17, the processor (600) of the augmented reality implementation device (10) determines whether a user authentication request using an authentication pattern is input (S1510), and if a user authentication request using an authentication pattern is input, generates a control signal for controlling the gaze tracking sensor (400) to operate the gaze tracking sensor (400) (S1710), calculates a user's gaze point trajectory based on the user's gaze direction information generated by the operation of the gaze tracking sensor (400) (S1720), generates gaze point trajectory image data for expressing a virtual gaze point trajectory image (VTI) based on the gaze point trajectory (S1730), and causes the display module (200) to emit image light toward the optical lens (100) to display the virtual gaze point trajectory image (VTI) based on the gaze point trajectory image data (S1740).

[0253] Specifically, when the processor (600) determines that a user authentication request using an authentication pattern has been input, the processor (600) can execute commands or program codes included in the image data generation module (720) and generate gaze point trajectory image data representing a virtual gaze point trajectory image (VTI).

[0254] The processor (600) can transmit the generated gaze point trajectory image data to the display module (200), and the display module (200) can emit image light toward the optical lens (100) to display a virtual gaze point trajectory image (VTI) based on the gaze point trajectory image data.

[0255] Fig. 18 is a flowchart illustrating a user authentication method using an augmented reality implementation device according to exemplary embodiments.

[0256] Referring to FIG. 18, the processor (600) of the augmented reality implementation device (10) determines whether a user authentication request using an authentication pattern is input (S1510), and if a user authentication request using an authentication pattern is input, generates a control signal for controlling the gaze tracking sensor (400) to operate the gaze tracking sensor (400) (S1810), calculates a user's gaze point trajectory based on the user's gaze direction information generated by the operation of the gaze tracking sensor (400) (S1820), determines whether the gaze point trajectory matches previously stored authentication pattern data (S1830), and if the gaze point trajectory matches the authentication pattern data, generates an authentication completion signal using the authentication pattern (S1840).

[0257] Specifically, when the processor (600) determines that a user authentication request using an authentication pattern has been input, the processor (600) can execute commands or program codes included in the control command generation module (730), and the processor (600) can generate a control signal for controlling the operation of the gaze tracking sensor (400) based on the user authentication request using the input authentication pattern. The processor (600) can transmit the generated control signal to the gaze tracking sensor (400), and the gaze tracking sensor (400) can start a sensing operation according to the control signal to acquire the user's gaze direction information.

[0258] The processor (600) can generate gaze point data from the user's gaze direction information input from the gaze tracking sensor (400) by executing commands or program codes included in the authentication pattern authentication module (750). Alternatively, the processor (600) can also receive gaze point data generated by the gaze tracking sensor (400). The processor (600) calculates the user's gaze point trajectory based on the generated gaze point data, and if it is determined that the calculated gaze point trajectory matches the previously stored authentication pattern data, the processor (600) can generate an authentication completion signal using the authentication pattern.

[0259] Here, the processor (600) can perform user authentication using the authentication pattern described above after the user makes a pattern input completion gesture. The processor (600) determines whether a pattern input completion gesture is identified from the user, and if a pattern input completion gesture is identified from the user, can perform user authentication using the authentication pattern. That is, the processor (600) calculates a gaze point trajectory of the user based on gaze point data up to the point at which the user makes the pattern input completion gesture, and if it is determined that the calculated gaze point trajectory matches the previously stored authentication pattern data, it can generate an authentication completion signal using the authentication pattern. At this time, the user's pattern input completion gesture may be a motion related to the user's eyes. For example, the user's pattern input completion gesture may be a state in which the user moves his gaze, inputs an authentication pattern, and then closes his eyes for several seconds, repeatedly, several times. The processor (600) can determine whether the user makes a pattern input completion gesture through the gaze tracking sensor (400). When the processor (600) determines that the user has made a pattern input completion gesture, the processor (600) can calculate the user's gaze point trajectory based on gaze point data up to the point at which the user has made the pattern input completion gesture, and determine whether the calculated gaze point trajectory matches the previously stored authentication pattern data. Meanwhile, the processor (600) of the augmented reality implementation device (10) can generate an authentication completion signal using an authentication pattern (S1840), and then transmit the generated authentication completion signal using the authentication pattern to the communication module (500), so that the communication module (500) can transmit the authentication completion signal using the authentication pattern to an external server (20) (S1850).

[0260] The user authentication method using the authentication pattern described with reference to FIGS. 15 to 18 described above may be additionally performed after the user authentication method using biometric characteristics described with reference to FIG. 13 is performed. Alternatively, it goes without saying that the user authentication method using biometric characteristics described with reference to FIG. 13 may be performed after the user authentication method using the authentication pattern described with reference to FIGS. 15 to 18 is performed.

[0261] Fig. 19 is a flowchart illustrating a user authentication method using an augmented reality implementation device according to exemplary embodiments.

[0262] Referring to FIG. 19, the processor (600) of the augmented reality implementation device (10) determines whether service content information is input (S1910), and if service content information is input, generates service content image data expressing a virtual service content image (VSI) (S1920), and causes the display module (200) to emit image light displaying the virtual service content image (VSI) toward the optical lens (100) based on the service content image data (S1930).

[0263] Specifically, the processor (600) can determine whether service content information is input by executing commands or program codes included in the service information identification module (710) to identify service information received from the communication module (500).

[0264] When the processor (600) determines that service content information has been input, the processor (600) can execute commands or program codes included in the image data generation module (720), and the processor (600) can generate service content image data representing a virtual service content image (VSI) based on the service content information.

[0265] Meanwhile, the processor (600) can generate service content image data representing that a virtual service content image (VSI) is positioned adjacent to an identified operation object (HO) based on the acquired operation object location data by executing a user request identification module (760) described later.

[0266] In exemplary embodiments, the processor (600) may generate service content image data representing that a portion of a virtual service content image (VSI) is positioned within an area of ​​an identified manipulation object (HO). Alternatively, the processor (600) may generate service content image data representing that a virtual service content image (VSI) is positioned at a predetermined distance from the identified manipulation object (HO) (e.g., at a location within reach of a user's hand).

[0267] The processor (600) can transmit the generated service content image data to the display module (200), and the display module (200) can emit image light toward the optical lens (100) to display a virtual service content image (VSI) based on the service content image data.

[0268] Fig. 20 is a flowchart illustrating a user authentication method using an augmented reality implementation device according to exemplary embodiments.

[0269] Referring to FIG. 20, the processor (600) of the augmented reality implementation device (10) determines whether service content information is input (S1910), and if service content information is input, generates a control signal for controlling the camera module (300) to operate the camera module (300) (S2010), determines whether a manipulation object (HO) is identified in the captured data generated by the operation of the camera module (300) (S2020), and if the manipulation object (HO) is identified, generates instruction image data expressing a virtual instruction image (VII) (S2030), and causes the display module (200) to emit image light displaying the virtual instruction image (VII) toward the optical lens (100) based on the instruction image data (S2040).

[0270] Specifically, the processor (600) can determine whether a manipulation object (HO), which is a real object, is identified in the photographed data received from the camera module (300) by executing commands or program codes included in the user request identification module (760). If the processor (600) identifies the manipulation object (HO) from the photographed data received, the processor (600) can generate identification data for the corresponding manipulation object (HO).

[0271] When the processor (600) identifies a manipulation object (HO), the processor (600) can generate instruction image data representing a virtual instruction image (VII) based on identification data for the manipulation object (HO) by executing commands or program codes included in the image data generation module (720).

[0272] In exemplary embodiments, the processor (600) may generate pointer image data representing that the virtual pointer image (VII) includes a pointer line image (VI-1) and a cursor image (VI-2).

[0273] The processor (600) can calculate a distance (D) between the virtual service content image (VSI) and the identified operation object (HO) based on the display position and operation object position data of the virtual service content image (VSI) described above, and generate instruction image data that expresses different virtual instruction images (VII) according to the calculated distance (D).

[0274] In exemplary embodiments, the processor (600) may generate instruction image data representing a virtual instruction image (VII) that includes only a cursor image (VI-2) when the distance (D) between the displayed virtual service content image (VSI) and the identified operation object (HO) is less than or equal to a preset distance.

[0275] In exemplary embodiments, the processor (600) may generate instruction image data representing a virtual instruction image (VII) including both an instruction line image (VI-1) and a cursor image (VI-2) when a separation distance (D) between a displayed virtual service content image (VSI) and an identified operation object (HO) exceeds a preset distance.

[0276] Meanwhile, as described above, the processor (600) may generate service content image data representing that a virtual service content image (VSI) is positioned adjacent to an identified manipulation object (HO) based on the manipulation object position data acquired by executing the user request identification module (760) described below. In this way, when the virtual service content image (VSI) is expressed as being positioned adjacent to the identified manipulation object (HO), and the distance (D) between the displayed virtual service content image (VSI) and the identified manipulation object (HO) is less than or equal to a preset distance, the processor (600) may generate instruction image data representing a virtual instruction image (VII) that includes only a cursor image (VI-2).

[0277] The processor (600) can transmit the generated instruction image data to the display module (200), and the display module (200) can emit image light toward the optical lens (100) to display a virtual instruction image (VII) based on the instruction image data.

[0278] Fig. 21 is a flowchart illustrating a user authentication method using an augmented reality implementation device according to exemplary embodiments.

[0279] Referring to FIG. 21, it is determined whether a manipulation object (HO) is identified in the shooting data generated by the operation of the camera module (300) (S2020), and if the manipulation object (HO) is identified, it is determined whether a manipulation gesture is identified in the shooting data (S2110), and if the manipulation gesture is identified, a user request signal can be generated based on the manipulation gesture (S2120).

[0280] Specifically, the processor (600) can execute commands or program codes included in the user request identification module (760) to determine whether a manipulation gesture of an operation object (HO) is identified in the photographed data received from the camera module (300). If the processor (600) identifies a manipulation gesture from the photographed data received, the processor (600) can generate identification data for the corresponding manipulation gesture.

[0281] When the processor (600) identifies a manipulation gesture from the shooting data, the processor (600) can generate a user request signal corresponding to the manipulation gesture based on the identification data for the manipulation gesture.

[0282] Meanwhile, the processor (600) of the augmented reality implementation device (10) generates a user request signal (S2120), and then transmits the generated user request signal to the communication module (500), so that the communication module (500) can transmit the service request signal to an external server (20) (S2130).

[0283] In exemplary embodiments, the manipulation gestures that the processor (600) identifies from the captured data received from the camera module (300) may include hand gestures expressed in the shape of the user's hand, hand gestures expressed in the trajectory drawn by the center point of the user's hand, and hand gestures expressed in combination with the shape of the user's hand and the trajectory drawn by the center point of the hand.

[0284] Meanwhile, the processor (600) can execute commands or program codes included in the user request identification module (760) to determine whether a manipulation gesture of the manipulation object (HO) is identified in the photographed data input from the gaze tracking sensor (400). If the processor (600) identifies a manipulation gesture of the manipulation object (HO) in the photographed data input from the gaze tracking sensor (400), it can also generate identification data for the corresponding manipulation gesture.

[0285] When the processor (600) identifies a manipulation gesture from the photographed data received from the gaze tracking sensor (400), the processor (600) can generate a user request signal corresponding to the manipulation gesture based on the identification data for the manipulation gesture.

[0286] Meanwhile, the processor (600) of the augmented reality implementation device (10) can generate a user request signal and then transmit the generated user request signal to the communication module (500), so that the communication module (500) can transmit the service request signal to an external server (20).

[0287] In exemplary embodiments, the manipulation object (HO) identified by the processor (600) from the captured data received from the gaze tracking sensor (400) may be the user's eye. Additionally, the manipulation gesture of the manipulation object (HO) identified by the processor (600) from the captured data received from the gaze tracking sensor (400) may include the user's eye blinking motion.

[0288] Although the embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the invention pertains without departing from the gist of the present invention as claimed in the claims. Furthermore, such modifications should not be understood individually from the technical idea or prospect of the present invention.

[0289] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or the components of the described electronic devices, structures, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

1. An optical lens provided to transmit image light displaying a virtual image to the user's eyes; A display module provided to emit image light displaying the virtual image toward the optical lens; A camera module that acquires shooting data; A memory that stores a program containing one or more instructions; and A processor configured to execute one or more instructions of a program stored in the memory, The above processor, Determine whether a user authentication request using biometric characteristics is entered, When a user authentication request using the above biometric characteristics is input, a control signal for controlling the camera module is generated to cause the camera module to operate, Identifying the biometric characteristics of the authentication object from the captured data generated by the operation of the above camera module, Determine whether the above biometric characteristics match the previously stored biometric authentication data, An augmented reality implementation device that generates an authentication completion signal using the biometric characteristic when the biometric characteristic matches the biometric authentication data.

2. In the first paragraph, further comprising a communication module that is communicatively connected to the server, The above processor, An augmented reality implementation device that transmits an authentication completion signal using the biometric characteristics to the communication module, and causes the communication module to transmit the authentication completion signal using the biometric characteristics to an external server.

3. In the first paragraph, the processor, Determines whether a user authentication request using an authentication pattern is entered, When a user authentication request using an authentication pattern is entered, authentication pattern image data representing a virtual authentication pattern image is generated, An augmented reality implementation device, wherein the display module emits image light displaying the virtual authentication pattern image toward the optical lens based on the authentication pattern image data.

4. In the third paragraph, further comprising a gaze tracking sensor provided to obtain information on the user's gaze direction, The above processor, When a user authentication request using the above authentication pattern is input, a control signal for controlling the eye tracking sensor is generated to cause the eye tracking sensor to operate, The user's gaze point trajectory is calculated based on the user's gaze direction information generated by the operation of the above gaze tracking sensor, Determine whether the above gaze point trajectory matches the previously stored authentication pattern data, An augmented reality implementation device that generates an authentication completion signal using an authentication pattern when the above gaze point trajectory matches the above authentication pattern data.

5. In the fourth paragraph, further comprising a communication module that is communicatively connected to the server, The above processor, An augmented reality implementation device that transmits an authentication completion signal using the above authentication pattern to the communication module, so that the communication module transmits the authentication completion signal using the above authentication pattern to an external server.

6. A user authentication method using an augmented reality implementation device equipped to transmit image light displaying a virtual image emitted by a display module to the user's eyes through an optical lens, Determine whether a user authentication request using biometric characteristics is input; When a user authentication request using the above biometric characteristics is input, a control signal for controlling the camera module is generated to cause the camera module to operate; Identifying the biometric characteristics of an authentication object from the captured data generated by the operation of the above camera module; Determine whether the above biometric characteristics match the previously stored biometric authentication data; and A user authentication method using an augmented reality implementation device, comprising generating an authentication completion signal using the biometric characteristic when the biometric characteristic matches the biometric authentication data.

7. A user authentication method using an augmented reality implementation device, further comprising transmitting an authentication completion signal using the biometric characteristic to a communication module in accordance with paragraph 6, so that the communication module transmits the authentication completion signal using the biometric characteristic to an external server.

8. In paragraph 6, Determines whether a user authentication request using an authentication pattern is input; When a user authentication request using an authentication pattern is input, authentication pattern image data representing a virtual authentication pattern image is generated; and A user authentication method using an augmented reality implementation device, further comprising: causing the display module to emit image light displaying the virtual authentication pattern image toward the optical lens based on the authentication pattern image data.

9. In paragraph 8, When a user authentication request using the above authentication pattern is input, a control signal for controlling the gaze tracking sensor is generated to cause the gaze tracking sensor to operate; The user's gaze point trajectory is calculated based on the user's gaze direction information generated by the operation of the above gaze tracking sensor; Determine whether the above gaze point trajectory matches the previously stored authentication pattern data; and A user authentication method using an augmented reality implementation device, further comprising generating an authentication completion signal using an authentication pattern when the above gaze point trajectory matches the above authentication pattern data.

10. A user authentication method using an augmented reality implementation device, further comprising transmitting an authentication completion signal using the authentication pattern to a communication module in accordance with paragraph 9, so that the communication module transmits the authentication completion signal using the authentication pattern to an external server.

Images