Augmented reality device for performing payment using gaze information of user and operating method thereof

The augmented reality device employs eye tracking to enhance payment processing by enlarging the region of interest based on the user's gaze, addressing the discomfort and recognition issues associated with traditional payment methods, thereby improving user convenience and accuracy.

WO2024248559A1PCT designated stage expired Publication Date: 2024-12-05SAMSUNG ELECTRONICS CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/007547
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-07-28
Filing Date
2024-06-03
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Augmented reality devices, such as smart glasses, face discomfort and low payment code recognition rates due to the need for users to move close to payment codes, which are small in the wide-angle camera's field of view, leading to decreased user convenience and recognition accuracy.

Method used

An augmented reality device uses eye tracking sensors to determine the user's gaze point, enlarges the corresponding region of interest in the camera's image, and recognizes payment codes, allowing for payment processing without requiring the user to physically move closer to the code.

Benefits of technology

This method improves user convenience and payment code recognition rates by enabling payments to be made from a distance, enhancing the usability of augmented reality devices in everyday life.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024007547_05122024_PF_FP_ABST
    Figure KR2024007547_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Provided are an augmented reality device for performing payment by recognizing a payment code by using gaze information of a user, and an operating method thereof. The method may comprise the steps of: acquiring a gaze point at which the gaze directions of both eyes of a user converge, and acquiring gaze information of both eyes of the user by using at least one gaze tracking sensor; determining a region of interest corresponding to the gaze point on an image acquired through a camera; enlarging the region of interest; recognizing a payment code; and performing payment on the basis of the recognized payment code.
Need to check novelty before this filing date? Find Prior Art

Description

Augmented reality device for performing payment using user's gaze information and its operation method

[0001] The present disclosure relates to an augmented reality device that performs payment using a user's gaze information, and a method for operating the same. Specifically, the present disclosure relates to an augmented reality device that uses a gaze tracking sensor to recognize a payment code viewed by a user and perform payment, and a method for operating the same.

[0002] Augmented reality (AR) is a technology that overlays virtual objects onto the physical environment or objects of the real world, presenting them alongside one another. Its advantage lies in the fusion of virtual objects and information within the real world. AR devices (e.g., smart glasses) utilizing AR technology are being used in everyday life for various purposes, such as information retrieval, route guidance, and camera photography. In particular, smart glasses are often worn as fashion accessories and are primarily used for outdoor activities.

[0003] Payment systems that utilize conventional payment codes (e.g., barcodes or QR codes) require the user to hold the mobile device close to the payment code so that the device's camera, such as a smartphone, can capture a sufficiently large image of the payment code. However, in the case of AR devices, the user typically wears the AR device on their head. Therefore, in order for the AR device's camera to recognize the payment code, the user must move very close to the payment code. This can be uncomfortable and awkward for the user when wearing the AR device on their head. Furthermore, because the cameras mounted on AR devices are typically wide-angle or ultra-wide-angle cameras, the image obtained when the camera captures the payment code only occupies a relatively small area of ​​the entire image. This leads to low recognition rates of payment codes and poor user convenience in payment systems utilizing AR devices.

[0004] One aspect of the present disclosure provides a method for an augmented reality device to perform a payment using a user's gaze information. According to one embodiment of the present disclosure, the method for operating the augmented reality device may include the steps of: obtaining a gaze point where the gaze directions of both eyes of the user converge based on the gaze information of both eyes of the user, wherein the gaze information is obtained by at least one gaze tracking sensor; determining a region of interest corresponding to the gaze point on an image obtained through a camera; enlarging the region of interest; recognizing a payment code included in the enlarged region of interest; and performing a payment using the payment code.

[0005] Another aspect of the present disclosure provides an augmented reality device that performs payment using a user's gaze information. According to one embodiment of the present disclosure, the augmented reality device may include at least one gaze tracking sensor that acquires gaze information of both eyes of the user, a camera that acquires an image, a memory that stores at least one instruction, and at least one processor that executes the at least one instruction. The at least one processor may acquire a gaze point where the gaze directions of both eyes of the user converge based on the gaze information of both eyes of the user, the gaze information is acquired using at least one gaze tracking sensor, determine a region of interest corresponding to the gaze point on an image acquired through the camera, enlarge the region of interest, recognize a payment code included in the enlarged region of interest, and perform payment using the payment code.

[0006] Another aspect of the present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an augmented reality device, such that the augmented reality device performs the following operations: obtaining a gaze point where the gaze directions of the user's two eyes converge based on gaze information of the user's two eyes, wherein the gaze information is obtained by at least one gaze tracking sensor; determining a region of interest corresponding to the gaze point on an image obtained through a camera; enlarging the region of interest; recognizing a payment code included in the enlarged region of interest; and performing a payment using the payment code.

[0007] The present disclosure can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.

[0008] FIG. 1 is a diagram illustrating an operation in which the augmented reality device of the present disclosure recognizes a payment code and performs payment using information on the user's gaze direction.

[0009] FIG. 2 is a flowchart illustrating a method for an augmented reality device to recognize a payment code using information on a user's gaze direction according to one embodiment of the present disclosure.

[0010] FIG. 3 is a flowchart illustrating a method for an augmented reality device to recognize a payment code according to one embodiment of the present disclosure.

[0011] FIG. 4 is a block diagram illustrating components of an augmented reality device according to one embodiment of the present disclosure.

[0012] FIG. 5 is a flowchart illustrating a method for an augmented reality device to select one camera among a plurality of cameras according to one embodiment of the present disclosure.

[0013] FIG. 6A is a diagram illustrating a plurality of cameras mounted on an augmented reality device according to one embodiment of the present disclosure.

[0014] FIG. 6b is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to select one camera among a plurality of cameras.

[0015] FIG. 7 is a flowchart illustrating a method for an augmented reality device to select one camera among a plurality of cameras according to one embodiment of the present disclosure.

[0016] FIG. 8 is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to select one camera among a plurality of cameras.

[0017] FIG. 9 is a flowchart illustrating a method for an augmented reality device to select one camera among a plurality of cameras according to one embodiment of the present disclosure.

[0018] FIG. 10 is a diagram illustrating a method for an augmented reality device according to one embodiment of the present disclosure to select one camera among a plurality of cameras.

[0019] FIG. 11 is a flowchart illustrating a method for an augmented reality device to recognize a payment code according to one embodiment of the present disclosure.

[0020] FIG. 12 is a flowchart illustrating a method for an augmented reality device to recognize a payment code according to one embodiment of the present disclosure.

[0021] FIG. 13 is a flowchart illustrating a method for an augmented reality device to determine an area of ​​interest according to one embodiment of the present disclosure.

[0022] FIG. 14 is a diagram illustrating a method for an augmented reality device to recognize text according to one embodiment of the present disclosure.

[0023] FIG. 15 is a flowchart illustrating a method for an augmented reality device to recognize text according to one embodiment of the present disclosure.

[0024] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0025] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0026] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0027] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory. The processor may include a central processing unit (CPU), a graphics processor (GPU), an accelerated processing unit (APU), a many integrated core (MIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator.One or more processors may control any one or any combination of other components of the computing device and / or perform operations or data processing related to communication. One or more processors may execute one or more programs stored in memory.

[0028] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0029] In this disclosure, 'Augmented Reality' means displaying a virtual image together within a physical environment space of the real world or displaying a real object and a virtual image together.

[0030] In the present disclosure, an 'augmented reality device' is a device capable of expressing augmented reality, and may be, for example, not only augmented reality glasses in the shape of glasses worn by a user on the face, but also a head-mounted display apparatus (HMD: Head Mounted Display Apparatus) worn on the head, or an augmented reality helmet.

[0031] In the present disclosure, a "payment code" is a symbolic code that includes information for payment of a product or service. In one embodiment of the present disclosure, the payment code may be a barcode including a plurality of bar shapes or a QR (Quick Response) code including a plurality of square shapes. A device (e.g., a mobile device or an augmented reality device) can capture an image of the payment code through a camera, decode the payment code included in the acquired image, thereby acquiring information for payment (e.g., product or service identification information, price information, etc.), and perform a payment process using the acquired information.

[0032] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0033] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0034] FIG. 1 is a drawing illustrating an operation in which an augmented reality device (100) of the present disclosure recognizes a payment code (10) using information on a user's gaze direction and performs payment.

[0035] Referring to FIG. 1, the augmented reality device (100) is a device that can express augmented reality by displaying a virtual image together in a physical environment space of the real world or by displaying a real object and a virtual image together. For example, the augmented reality device (100) may be an augmented reality glasses in the shape of glasses that a user wears on the face. However, the augmented reality device (100) is not limited thereto, and may be a head-mounted display apparatus (HMD) that a user wears on the head, an augmented reality helmet, or the like.

[0036] An augmented reality device (100) may include at least one gaze tracking sensor (110) and a camera (120). The augmented reality device (100) obtains gaze information of both eyes of a user using at least one gaze tracking sensor (110), and obtains a gaze point (G) where the gaze directions of both eyes of the user converge in a three-dimensional space based on the obtained gaze information. This is illustrated in operation 1.

[0037] The augmented reality device (100) determines an area of ​​interest (30) based on a gaze point (G) on an image (20) acquired through a camera (120). This is illustrated in operation 2.

[0038] The augmented reality device (100) enlarges the area of ​​interest (30). This is illustrated in operation 3.

[0039] The augmented reality device (100) recognizes a payment code (10) from an enlarged area of ​​interest (40) and performs payment using the recognized payment code (10). This is illustrated in operation 4.

[0040] Hereinafter, the functions and / or operations of the augmented reality device (100) according to the embodiment illustrated in FIG. 1 will be described in detail with reference to FIG. 2.

[0041] FIG. 2 is a flowchart illustrating a method for an augmented reality device (100) according to one embodiment of the present disclosure to recognize a payment code using information on a user's gaze direction.

[0042] In step S210, the augmented reality device (100) acquires a gaze point where the gaze directions of both eyes of the user converge based on the user's gaze information acquired through at least one gaze tracking sensor. In one embodiment of the present disclosure, the augmented reality device (100) may include a left-eye gaze tracking sensor that detects the gaze direction of the user's left eye and a right-eye gaze tracking sensor that detects the gaze direction of the user's right eye. 'Detecting the user's gaze direction' may include an operation of acquiring gaze information related to the user's gaze. In combination with operation 1 of FIG. 1, the gaze tracking sensor (110) may include a left-eye gaze tracking sensor and a right-eye gaze tracking sensor. The augmented reality device (100) may acquire information about the gaze direction of the user's left eye through the left-eye gaze tracking sensor, and may acquire information about the gaze direction of the user's right eye through the right-eye gaze tracking sensor. The augmented reality device (100) can detect a gaze point (G) where the gaze directions of the user's left eye and right eye converge, and acquire three-dimensional position coordinate information of the gaze point (G). In one embodiment of the present disclosure, the gaze point (G) determined by the gaze directions of the user's two eyes can be located at a payment code (10) in a three-dimensional space of the real world.

[0043] In the present disclosure, a 'payment code' is a symbol code that includes information for payment of a product or service, and may include, for example, a barcode or QR code.

[0044] In step S220, the augmented reality device (100) determines a region of interest corresponding to a gaze point on an image acquired through a camera. In one embodiment of the present disclosure, the augmented reality device (100) may determine a region of interest based on two-dimensional position coordinate information corresponding to the gaze point on the image. In combination with operation 2 of FIG. 1, the augmented reality device (100) may acquire an image (20) by photographing an area viewed by the user, i.e., an area corresponding to the gaze point (G), using the camera (120). In one embodiment of the present disclosure, the camera (120) is configured as an RGB camera and may acquire a two-dimensional (2D) image (20) by photographing the real world. The augmented reality device (100) may determine a region of interest based on two-dimensional position coordinate information corresponding to the gaze point on the image. In combination with operation 2 of FIG. 1, the augmented reality device (100) may acquire an image (20) by photographing an area viewed by the user, i.e., an area corresponding to the gaze point (G), using the camera (120). In one embodiment of the present disclosure, the camera (120) may be configured as an RGB camera and may acquire a two-dimensional (2D) image (20) by photographing the real world. The augmented reality device (100) may be configured to capture an image of the user's left eye (E). L ) and the direction of gaze of the right eye (E R ) is mapped onto the image (20) by mapping the gaze point (G) in the three-dimensional space where the gaze direction converges, and the point (P) corresponding to the gaze point (G) on the image (20) 2D ) and identify the point (P 2D ) can obtain two-dimensional position coordinate information. In one embodiment of the present disclosure, the augmented reality device (100) maps a gaze point (G) in a three-dimensional space onto a two-dimensional image (20) based on at least one of the position, field of view, or intrinsic parameter of the camera (120), thereby obtaining a point (P) corresponding to the gaze point (G). 2D ) can obtain two-dimensional position coordinate information.

[0045] The augmented reality device (100) acquires points (P 2D ) can be determined as a region of interest (30) having a preset size centered on the two-dimensional position coordinates of the payment code (10). In one embodiment of the present disclosure, the region of interest (30) can include an image area of ​​the payment code (10).

[0046] In step S230, the augmented reality device (100) enlarges the region of interest. In one embodiment of the present disclosure, the augmented reality device (100) can enlarge the region of interest at a preset zoom ratio. In combination with operation 3 of FIG. 1, the augmented reality device (100) can enlarge the region of interest (30) at a preset plurality (n) of zoom ratios. In one embodiment of the present disclosure, the augmented reality device (100) can enlarge the region of interest (30) by using a digital zoom or optical zoom function of the camera (120).

[0047] In step S240, the augmented reality device (100) recognizes a payment code from the region of interest. In combination with operation 4 of FIG. 1, the augmented reality device (100) can recognize a payment code (10) from the enlarged region of interest (40) of the image (20) based on the enlargement result. In one embodiment of the present disclosure, the augmented reality device (100) can recognize a payment code (10) from the enlarged region of interest (40) by performing payment code detection, for example, barcode detection or QR code detection.

[0048] The augmented reality device (100) can execute a payment application and perform a payment process using the recognized payment code (10).

[0049] Although FIGS. 1 and 2 illustrate an embodiment in which an augmented reality device (100) recognizes a payment code (10), the present disclosure is not limited thereto. In other embodiments of the present disclosure, the payment code (10) may not be recognized from the enlarged region of interest (40). The operation of the augmented reality device (100) according to the recognition result of the payment code (10) will be described in detail in FIG. 3.

[0050] FIG. 3 is a flowchart illustrating an operation method performed by an augmented reality device (100) according to one embodiment of the present disclosure depending on whether a payment code is recognized.

[0051] In step S310, the augmented reality device (100) determines whether the payment code is recognized. In one embodiment of the present disclosure, the augmented reality device (100) can determine whether the payment code is normally recognized in an area enlarged at a preset zoom ratio.

[0052] In one embodiment of the present disclosure, while step S310 is being performed, the augmented reality device (100) may output a notification message indicating that payment code recognition is in progress.

[0053] If it is determined that the payment code has been recognized (step S320), the augmented reality device (100) outputs a user interface requesting payment approval. In one embodiment of the present disclosure, the augmented reality device (100) may obtain product or service identification information, price information, etc. from the recognized payment code, and output a notification message asking the user whether to approve the payment along with the obtained information.

[0054] In step S330, the augmented reality device (100) executes a payment application and performs a payment process using a payment code.

[0055] If it is determined that the payment code was not recognized normally (step S340), the augmented reality device (100) determines whether the maximum magnification of the camera has been reached. In one embodiment of the present disclosure, the augmented reality device (100) can determine whether the zoom magnification that enlarges the area of ​​interest is the maximum zoom magnification by the digital zoom or optical zoom of the camera (120, see FIGS. 1 and 4).

[0056] If it is determined that the maximum zoom ratio of the camera has been reached (e.g., step S350), the augmented reality device (100) outputs a user interface indicating a failure in recognizing the payment code. The augmented reality device (100) may output a notification message indicating a payment failure, such as, for example, "Payment was declined because the barcode (or QR code) was not recognized."

[0057] If it is determined that the maximum zoom ratio of the camera has not been reached (e.g., step S360), the augmented reality device (100) increases the zoom ratio to enlarge the image area. In one embodiment of the present disclosure, the augmented reality device (100) may enlarge the area of ​​interest by increasing the zoom ratio compared to the existing zoom ratio. Thereafter, the augmented reality device (100) may return to step S310 and re-determine whether the payment code is recognized from the image area enlarged with the increased zoom ratio.

[0058] Generally, the augmented reality device (100) is worn on the user's head or face, so when attempting to make a payment using a payment code, the user must move the camera (120) of the augmented reality device (100) to a very close distance from the payment code. In this case, when the augmented reality device (100) is worn on the head, it may cause discomfort and awkwardness to the user. In addition, since the camera (120) mounted on the augmented reality device (100) is generally a wide-angle or ultra-wide-angle camera, in the image obtained by the camera (120) capturing the payment code, the payment code occupies only a relatively small area among the entire area of ​​the image. Due to this, there is a problem that the recognition rate of the payment code is low and user convenience is low in the payment system using the augmented reality device (100).

[0059] The present disclosure aims to provide an augmented reality device (100) and an operating method thereof that improves user convenience by automatically recognizing a payment code even from a distance based on user gaze direction information acquired using a gaze tracking sensor (110).

[0060] The augmented reality device (100) according to an embodiment of the present disclosure obtains information on the gaze direction of a user looking at a payment code (10) using an eye tracking sensor (110), determines an area of ​​interest (30) including the payment code (10) on an image (20) obtained through a camera (120) based on the gaze direction information, recognizes the payment code (10) by enlarging the area of ​​interest (30), and performs a payment process using the recognized payment code (10). The augmented reality device (100) according to an embodiment of the present disclosure can improve user convenience by enabling convenient payment code-based payment without requiring the user to bring their head close to the payment code (10). In addition, the augmented reality device (100) according to an embodiment of the present disclosure recognizes the payment code (10) by enlarging an area corresponding to a gaze point (G, see FIG. 1) on an image even when the camera (120) is a wide-angle or ultra-wide-angle camera, thereby providing a technical effect of improving the recognition rate of the payment code (10).

[0061] In addition, the method of operating the augmented reality device (100) according to one embodiment of the present disclosure can be applied not only to recognition of a payment code (10), but also to remote object recognition, remote text recognition, remote text translation, etc., thereby expanding the field of application. A specific embodiment of the augmented reality device (100) recognizing text located at a remote location will be described in detail with reference to FIGS. 14 and 15.

[0062] FIG. 4 is a block diagram illustrating components of an augmented reality device (100) according to one embodiment of the present disclosure.

[0063] Referring to FIG. 4, the augmented reality device (100) may include a gaze tracking sensor (110), a camera (120), a processor (130), a memory (140), and a display unit (150). The gaze tracking sensor (110), the camera (120), the processor (130), the memory (140), and the display unit (150) may be electrically and / or physically connected to each other, respectively. Only components for explaining the operation of the augmented reality device (100) are illustrated in FIG. 4, and the components included in the augmented reality device (100) are not limited to those illustrated in FIG. 4. In one embodiment of the present disclosure, the augmented reality device (100) may further include a communication interface for performing data communication with an external device or a payment server. In one embodiment of the present disclosure, the augmented reality device (100) is implemented as a portable device, and in this case, the augmented reality device (100) may further include a battery that supplies driving power to a gaze tracking sensor (110), a camera (120), a processor (130), and a display unit (150).

[0064] The gaze tracking sensor (110) is a device configured to detect and track the gaze direction of a user's eyes. The gaze tracking sensor (110) can detect the user's gaze direction by detecting an image of a person's pupil or pupils, or by detecting the direction or amount of reflected light, such as near-infrared light, reflected from the cornea. In one embodiment of the present disclosure, the gaze tracking sensor (110) may include an infrared light source and an infrared camera.

[0065] The gaze tracking sensor (110) includes a left-eye gaze tracking sensor and a right-eye gaze tracking sensor, and can detect the gaze direction of the user's left eye and the gaze direction of the user's right eye, respectively. Detecting the user's gaze direction may include an operation of obtaining gaze information related to the user's gaze. In one embodiment of the present disclosure, a gaze point where the gaze direction of the user's left eye detected by the gaze tracking sensor (110) and the gaze direction of the user's right eye detected by the right-eye gaze tracking sensor converge is detected, and three-dimensional position coordinate value information of the gaze point can be obtained. The gaze tracking sensor (110) can provide three-dimensional position coordinate value information of the gaze point to the processor (130).

[0066] The camera (120) is configured to capture images of the real-world space and objects within the real-world space. In one embodiment of the present disclosure, the camera (120) is implemented with a small form factor so that it can be mounted on the augmented reality device (100) and may be implemented as a lightweight RGB camera that consumes low power. However, the present disclosure is not limited thereto, and in one embodiment of the present disclosure, the camera (120) may be implemented as any type of camera known in the art, such as an RGB-depth camera including a depth estimation function, a dynamic vision sensor camera, a stereo fish-eye camera, a grayscale camera, or an infrared camera.

[0067] The camera (120) may include a lens module, an image sensor, and an image processing module. The camera (120) may acquire a still image or video of an object by means of an image sensor (e.g., CMOS or CCD). The image processing module may encode a still image composed of a single image frame acquired through the image sensor or video data composed of multiple image frames and transmit the encoded data to the processor (130). In one embodiment of the present disclosure, the camera (120) may capture an image by photographing a payment code in a real space and provide the acquired image to the processor (130).

[0068] In one embodiment of the present disclosure, the camera (120) may be configured in multiple units. When the augmented reality device (100) is implemented as augmented reality glasses, the multiple cameras (120) may be arranged, for example, on the frame, temples, or nose bridge of the augmented reality glasses. A specific exemplary embodiment regarding the arrangement positions of the multiple cameras (120) mounted on the augmented reality device (100) will be described in detail with reference to FIG. 6A.

[0069] In one embodiment of the present disclosure, each of the plurality of cameras (120) may have different optical characteristics, such as a zoom ratio or a field of view.

[0070] The processor (130) can execute one or more instructions of a program stored in the memory (140). The processor (130) may be configured with hardware components that perform arithmetic, logic, and input / output operations and image processing. Although the processor (130) is illustrated as a single element in FIG. 3 , it is not limited thereto. In one embodiment of the present disclosure, the processor (130) may be configured with one or more elements. The processor (130) may be a general-purpose processor such as a Central Processing Unit (CPU), an Application Processor (AP), a Digital Signal Processor (DSP), a graphics-only processor such as a Graphics Processing Unit (GPU), a Vision Processing Unit (VPU), or an artificial intelligence-only processor such as a Neural Processing Unit (NPU). The processor (130) may be controlled to process input data according to predefined operating rules or artificial intelligence models. In some embodiments, when the processor (130) is an artificial intelligence-only processor, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0071] The memory (140) may be configured as at least one type of storage medium, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), or an optical disk.

[0072] The memory (140) may store instructions related to functions and / or operations for the augmented reality device (100) to recognize a payment code based on the user's gaze information and perform a payment process. In one embodiment of the present disclosure, the memory (140) may store at least one of instructions, an algorithm, a data structure, a program code, and an application program that can be read by the processor (130). The instructions, algorithms, data structures, and program code stored in the memory (140) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0073] The processor (130) can be implemented by executing instructions or program codes stored in memory (140).

[0074] The processor (130) can obtain information about a gaze point where the gaze directions of the user's two eyes converge from the gaze tracking sensor (110). In one embodiment of the present disclosure, the user can look at a payment code, and in this case, the gaze point can be located on the payment code in three-dimensional space. In the present disclosure, the 'payment code' is a symbol code that includes information for payment of a product or service, and can include, for example, a barcode or QR code. The processor (130) can obtain three-dimensional position coordinate information of the gaze point located on the payment code from the gaze tracking sensor (110).

[0075] The processor (130) can obtain an image of a payment code viewed by a user from the camera (120), and obtain two-dimensional position coordinate information corresponding to the gaze point on the image. In one embodiment of the present disclosure, the processor (130) can map a gaze point having a three-dimensional position coordinate value onto a two-dimensional image obtained through the camera (120), and obtain two-dimensional position coordinate values ​​corresponding to the gaze point on the two-dimensional image. In one embodiment of the present disclosure, the processor (130) can obtain two-dimensional position coordinate values ​​of a point corresponding to the gaze point by mapping a gaze point in a three-dimensional space onto a two-dimensional image based on at least one of a position at which the camera (120) is positioned in the augmented reality device (100), a field of view, or an intrinsic parameter of the camera. The intrinsic parameter of the camera may include, for example, information regarding a focal length and a principal point of the camera (120).

[0076] The processor (130) can determine an area having a preset size centered on a two-dimensional position coordinate value corresponding to a gaze point on an image as a region of interest. In one embodiment of the present disclosure, the region of interest (30) may include an image area of ​​a payment code (10).

[0077] In one embodiment of the present disclosure, the camera (120) includes a plurality of cameras (120) having different zoom magnifications, and the processor (130) may select a camera (120) having an optimal magnification for recognizing a payment code among the plurality of cameras (120) based on a distance between a gaze point and a user in a three-dimensional space. The processor (130) may determine an area of ​​interest from an image obtained by photographing the payment code using the selected camera (120). In one embodiment of the present disclosure, the processor (130) may select a camera (120) having a higher zoom magnification than a threshold or having the highest zoom magnification among the plurality of cameras (120) in proportion to the distance between the gaze point and the user. A specific embodiment in which the processor (130) selects one camera (120) among the plurality of cameras (120) will be described in detail with reference to FIGS. 5, 6A, and 6B.

[0078] In one embodiment of the present disclosure, information regarding the physical size of the payment code may be acquired in advance and stored in the memory (140). The processor (130) may calculate the size of an area surrounding the payment code by projecting the payment code onto a plurality of images acquired through a plurality of cameras (120) based on the information regarding the size of the payment code, and may select a camera (120) configured to acquire an image having the largest size of the calculated area among the plurality of cameras (120). The processor (130) may determine a region of interest from the image acquired through the selected camera (120). An exemplary embodiment in which information regarding the physical size of the payment code is known in advance, the processor (130) determines one camera (120) from the plurality of cameras (120), and determines a region of interest from an image acquired through the determined camera (120) is described in detail in FIGS. 7 and 8 .

[0079] Information regarding the physical size of the payment code may not be acquired in advance. In an embodiment where information regarding the physical size of the payment code is not acquired, the processor (130) acquires two-dimensional position coordinate values ​​corresponding to a gaze point from a plurality of images acquired through a plurality of cameras (120), recognizes the presence of a payment code and the size of the area occupied by the payment code from an area of ​​a preset size based on the two-dimensional position coordinate values ​​of the gaze point, and selects a camera (120) configured to acquire an image having the largest size of the area occupied by the payment code among the plurality of cameras (120). The processor (130) may determine a region of interest from the image acquired through the selected camera (120). An exemplary embodiment in which the processor (130) determines one camera (120) from the plurality of cameras (120) and determines a region of interest from an image acquired through the determined camera (120) without knowing information regarding the physical size of the payment code is described in detail in FIGS. 9 and 10 .

[0080] In one embodiment of the present disclosure, when the camera (120) includes a plurality of cameras (120), the processor (130) may acquire a plurality of images by photographing an area corresponding to a gaze point using the plurality of cameras (120), acquire a two-dimensional position coordinate value corresponding to the gaze point from each of the acquired plurality of images, and determine an area of ​​a preset size as a region of interest based on the acquired two-dimensional position coordinate value. An exemplary embodiment in which the processor (130) determines a region of interest from each of the plurality of images acquired through the plurality of cameras (120) is described in detail with reference to FIGS. 11 and 12.

[0081] In one embodiment of the present disclosure, when the camera (120) includes a plurality of cameras (120), the processor (130) may acquire a plurality of images by photographing an area corresponding to a gaze point using the plurality of cameras (120), and perform image processing using the acquired plurality of images to acquire a high-quality image with improved image quality parameters including at least one of resolution, contrast ratio, and noise. The processor (130) may determine an area of ​​a preset size centered on a two-dimensional position coordinate value corresponding to the gaze point on the high-quality image as a region of interest. A specific embodiment in which the processor (130) acquires a high-quality image and determines a region of interest from the high-quality image will be described in detail with reference to FIG. 13.

[0082] The processor (130) can magnify the region of interest at a preset zoom ratio. In one embodiment of the present disclosure, the processor (130) can magnify the region of interest according to a plurality (n) of preset zoom ratios. The processor (130) can magnify the image corresponding to the region of interest using a digital zoom method, or can magnify the image of the region of interest using a digital or optical zoom function by using an optical zoom function of the camera (120).

[0083] The processor (130) can recognize a payment code from an enlarged image area. In one embodiment of the present disclosure, the processor (130) can recognize a payment code from the enlarged image area using a known barcode detection or QR code detection technology.

[0084] If the payment code is recognized normally, the processor (130) can execute the payment application and perform the payment process using the recognized payment code. In one embodiment of the present disclosure, the processor (130) can output a user interface (UI), such as a notification message requesting payment approval, through the display unit (150) before performing the payment process. The UI can be implemented as a virtual object including, for example, text or an image, and displayed to the user by being projected onto a waveguide using an optical projector of the display unit (150).

[0085] If the payment code is not recognized normally, the processor (130) can determine whether the region of interest is enlarged to the maximum zoom ratio of the camera (120). If it is determined that the region of interest is enlarged to the maximum zoom ratio of the camera (120), the processor (130) can output a user interface indicating a failure in recognition of the payment code through the display unit (150). If it is determined that the region of interest is not enlarged to the maximum zoom ratio of the camera (120), the processor (130) can increase the magnification ratio of the camera (120) to enlarge the image corresponding to the region of interest. Thereafter, the processor (130) can recognize the payment code from the image region enlarged to the increased zoom ratio.

[0086] The display unit (150) is configured to display a payment code by enlarging an area of ​​interest of an image under the control of the processor (130) or to display a virtual object such as a user interface. When the augmented reality device (100) is implemented as augmented reality glasses, the display unit (150) may be configured with a lens optical system and may include a waveguide and an optical engine. The optical engine may be configured with a projector that generates light of a three-dimensional virtual object composed of a virtual image and projects the light onto a waveguide. The optical engine may include, for example, an image panel, an illumination optical system, a projection optical system, etc. In one embodiment of the present disclosure, the optical engine may be disposed on the frame or temples of the augmented reality glasses. In one embodiment of the present disclosure, the optical engine may display a notification message by projecting light of a virtual object composed of a notification message providing information such as payment approval or payment code recognition failure onto a waveguide under the control of the processor (130).

[0087] However, it is not limited thereto, and the display unit (150) may be configured as at least one of, for example, a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a 3D display, and an electrophoretic display.

[0088] FIG. 5 is a flowchart illustrating an operation method of an augmented reality device (100) according to one embodiment of the present disclosure for obtaining an image by selecting one camera among a plurality of cameras.

[0089] Step S520 illustrated in FIG. 5 is a step that embodies step S220 of FIG. 2. Step S510 illustrated in FIG. 5 may be performed before or after the operation of step S210 of FIG. 2 is performed. For example, step S230 of FIG. 2 may be performed after the operation of step S520 is performed.

[0090] FIG. 6a is a diagram illustrating a plurality of cameras (121, 122, 123) mounted on an augmented reality device (100) according to one embodiment of the present disclosure.

[0091] FIG. 6b is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to acquire an image by selecting one of a plurality of cameras (121, 122, 123).

[0092] Hereinafter, the function and / or operation of the augmented reality device (100) will be described with reference to FIG. 5, FIG. 6a, and FIG. 6b together.

[0093] In step S510 of FIG. 5, the augmented reality device (100) selects a camera having an optimal magnification for payment code recognition among a plurality of cameras based on the distance between the gaze point and the user. Referring also to FIG. 6A, the augmented reality device (100) may include a plurality of cameras (121, 122, 123). In the embodiment illustrated in FIG. 6A, a first camera (121) among the plurality of cameras (121, 122, 123) may be disposed on a left temple of the augmented reality device (100) implemented as augmented reality glasses, a second camera (122) may be disposed on a nose bridge connecting a left lens frame and a right lens frame of the augmented reality device (100), and a third camera (123) may be disposed on a right temple of the augmented reality device (100). However, the positions and numbers of the multiple cameras (121, 122, 123) are merely examples for convenience of explanation and are not limited to those illustrated in FIG. 6A. Each of the multiple cameras (121, 122, 123) may have a different zoom ratio. In one embodiment of the present disclosure, each of the multiple cameras (121, 122, 123) may have a different field of view.

[0094] Referring to FIG. 6B, a user (60) may wear an augmented reality device (100) and look at a payment code (600). The augmented reality device (100) may obtain information about the gaze direction of both eyes of the user (60) using the gaze tracking sensor (110) and obtain three-dimensional position coordinate information of a gaze point (G) where the gaze directions of both eyes converge. The processor (130) of the augmented reality device (100) may obtain a distance (d) between the gaze point (G) and the user (60) from the information about the gaze point (G) obtained through the gaze tracking sensor (110). In one embodiment of the present disclosure, the processor (130) may obtain the distance (d) between the gaze point (G) and the user (60) based on the three-dimensional position coordinate information of the gaze point (G) and the vergence distance between the two eyes of the user (60).

[0095] The processor (130) of the augmented reality device (100) may select a camera having an optimal magnification among a plurality of cameras (121, 122, 123, see FIG. 6A) based on the distance (d) between the gaze point (G) and the user (60). In one embodiment of the present disclosure, the processor (130) may select a camera having a high magnification in proportion to the distance (d) between the gaze point (G) and the user (60). For example, the processor (130) may select a camera having a high zoom magnification among the plurality of cameras (121, 122, 123) as the distance (d) between the gaze point (G) and the user (60) increases.

[0096] Referring back to FIG. 5, in step S520, the augmented reality device (100) determines a region of interest from an image acquired by photographing a payment code using the selected camera. Referring also to FIG. 6A, for example, if the camera with the highest magnification among the plurality of cameras (121, 122, 123) is the second camera (122), the augmented reality device (100) may select the second camera (122) and acquire an image by photographing an area including the payment code where the gaze point (G) is located using the selected second camera (122). The processor (130) of the augmented reality device (100) may acquire two-dimensional position coordinate value information corresponding to the gaze point (G) from the acquired image, and determine an area of ​​a preset size centered on the two-dimensional position coordinate value as the region of interest.

[0097] Referring again to FIG. 5, in step S230, the augmented reality device (100) enlarges the area of ​​interest to a preset zoom ratio. Step S230 is the same as that described in FIG. 2, and therefore, redundant description is omitted.

[0098] If the distance (d) between the payment code (600) and the user (60) is too far, and the payment code (600) is captured too small in the image acquired through the multiple cameras (121, 122, 123), the recognition rate of the payment code may be low. In the embodiments illustrated in FIGS. 5, 6A, and 6B, the augmented reality device (100) selects a camera with the highest zoom ratio among the multiple cameras (121, 122, 123) to solve the problem of low recognition rate, and captures the payment code (600) using the selected camera to acquire an image, thereby improving the recognition rate of the payment code (600).

[0099] FIG. 7 is a flowchart illustrating an operation method for selecting one of a plurality of cameras to acquire an image when an augmented reality device (100) according to one embodiment of the present disclosure has previously acquired information regarding the size of a payment code.

[0100] Steps S710 and S720 illustrated in FIG. 7 are steps that specify the operation of step S510 of FIG. 5. Step S710 illustrated in FIG. 7 may be performed before or after the operation of step S210 of FIG. 2 is performed. For example, step S520 of FIG. 5 may be performed after the operation of step S720 illustrated in FIG. 7 is performed.

[0101] FIG. 8 is a diagram illustrating an operation of selecting one of a plurality of cameras (121, 122, 123) to acquire an image when an augmented reality device (100) according to one embodiment of the present disclosure has previously acquired information regarding the size of a payment code (800).

[0102] Hereinafter, the function and / or operation of the augmented reality device (100) will be described with reference to FIGS. 7 and 8 together.

[0103] In step S710 of FIG. 7, the augmented reality device (100) calculates the size of the area surrounding the payment code by projecting the payment code onto a plurality of images acquired through a plurality of cameras based on information about the size of the payment code. In one embodiment of the present disclosure, information about the physical size of the payment code may be acquired in advance and stored in the memory (140) of the augmented reality device (100). Referring also to FIG. 8, the physical size of the payment code (800), that is, the size of the width (W) and the size of the height (H), may be acquired in advance and stored in the memory (140) of the augmented reality device (100). For example, the width (W) of the payment code (800) may be 10 cm, and the height (H) may be 10 cm. However, this is exemplary and is not limited to the above sizes. In one embodiment of the present disclosure, the augmented reality device (100) may include a plurality of cameras (121, 122, 123). The positions and numbers of the multiple cameras (121, 122, 123) are the same as those shown in Fig. 6a, so redundant descriptions are omitted.

[0104] The processor (130) of the augmented reality device (100) can calculate the size of the area surrounding the payment code (800) by projecting the payment code (800) onto each of a plurality of images (811, 812, 813) acquired through a plurality of cameras (121, 122, 123) based on information about the size (W, H) of the payment code (800) previously stored in the memory (140). In the embodiment illustrated in FIG. 8, the processor (130) can calculate the size of the first area (821) surrounding the payment code (800) from the first image (811) acquired through the first camera (121). Similarly, the processor (130) can calculate the size of the second area (822) surrounding the payment code (800) from the second image (812) acquired through the second camera (122), and can calculate the size of the third area (823) surrounding the payment code (800) from the third image (813) acquired through the third camera (123).

[0105] Referring back to FIG. 7, in step S720, the augmented reality device (100) selects a camera that acquires an image with the largest calculated area size among the plurality of cameras. In the embodiment illustrated in FIG. 8, among the plurality of images (811, 812, 813) acquired through the plurality of cameras (121, 122, 123), the area size of the area (821, 822, 823) surrounding the payment code (800) on the plurality of images (811, 812, 813) may be the largest in the second area (822). The processor (130) of the augmented reality device (100) may select the second camera (122) that captures the second image (812) with the largest area of ​​the area (821, 822, 823) surrounding the payment code (800) among the plurality of cameras (121, 122, 123).

[0106] Referring to FIG. 7, in step S520, the augmented reality device (100) obtains a two-dimensional position coordinate value corresponding to the gaze point from a second image (812) obtained by photographing a payment code (800) using the selected second camera (122), and determines an area of ​​a preset size as an area of ​​interest based on the two-dimensional position coordinate value.

[0107] In the embodiment illustrated in FIGS. 7 and 8, if the physical size (W, H) of the payment code (800) is known in advance, the augmented reality device (100) can project the payment code (800) onto a plurality of images (812, 822, 823) acquired through a plurality of cameras (121, 122, 123) using the physical size (W, H) of the payment code (800), calculate the size of the area surrounding the payment code (800), and select a camera that captures the image with the largest calculated size. Through this, the augmented reality device (100) according to one embodiment of the present disclosure can improve the recognition rate of the payment code (800) by selecting a camera that can capture the payment code (800) at the largest size when the magnification or field of view (FOV) of each of the plurality of cameras (121, 122, 123) is different.

[0108] FIG. 9 is a flowchart illustrating an operation method of an augmented reality device (100) according to one embodiment of the present disclosure for acquiring an image by selecting one of a plurality of cameras based on the size of an area occupied by a payment code within the image.

[0109] Steps S910 to S930 illustrated in FIG. 9 are steps that specify the operation of step S510 of FIG. 5. Step S910 illustrated in FIG. 9 may be performed before or after the operation of step S210 of FIG. 2 is performed. After the operation of step S930 illustrated in FIG. 9 is performed, step S520 of FIG. 5 may be performed.

[0110] FIG. 10 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure to acquire an image by selecting one of a plurality of cameras (121, 122, 123) based on the size of an area occupied by a payment code (1000) within an image (1011, 1012, 1013).

[0111] Hereinafter, the function and / or operation of the augmented reality device (100) will be described with reference to FIGS. 9 and 10 together.

[0112] Referring to FIG. 9, in step S910, the augmented reality device (100) obtains two-dimensional position coordinate information corresponding to a gaze point from a plurality of images acquired through a plurality of cameras. Referring also to FIG. 10, the augmented reality device (100) may obtain gaze direction information of both eyes of a user looking at a payment code (1000) using a gaze tracking sensor (110), and may obtain a three-dimensional position coordinate value of a gaze point (G) where the gaze directions of both eyes converge. In one embodiment of the present disclosure, the augmented reality device (100) may include a plurality of cameras (121, 122, 123). The positions and numbers of the plurality of cameras (121, 122, 123) are the same as those illustrated in FIG. 6A, and therefore, redundant descriptions are omitted. The augmented reality device (100) can acquire multiple images (1011, 1012, 1013) by capturing the location the user is looking at, i.e., the area including the payment code (1000), through multiple cameras (121, 122, 123).

[0113] The processor (130) of the augmented reality device (100) can identify points (P1, P2, P3) corresponding to the gaze point (G) from each of the plurality of images (1011, 1012, 1013) and obtain two-dimensional position coordinate values ​​of the points (P1, P2, P3). In one embodiment of the present disclosure, the processor (130) can obtain two-dimensional position coordinate values ​​of the points (P1, P2, P3) corresponding to the gaze point (G) by mapping the gaze point (G) in the three-dimensional space onto the two-dimensional images (1011, 1012, 1013) based on at least one of the positions, fields of view, or intrinsic parameters of each of the plurality of cameras (121, 122, 123). The camera's unique parameters may include, for example, information about the focal length and principal point of each of the multiple cameras (121, 122, 123).

[0114] Referring back to FIG. 9, in step S920, the augmented reality device (100) performs payment code detection, and recognizes the presence of a payment code and the size occupied by the payment code from a set area based on the two-dimensional position coordinate information of the gaze point. Referring also to FIG. 10, the processor (130) of the augmented reality device (100) can detect a payment code (1000) from a detection area (1021, 1022, 1023) of a preset size centered on the two-dimensional position coordinate values ​​of points (P1, P2, P3) corresponding to the gaze point (G) from a plurality of images (1011, 1012, 1013). In one embodiment of the present disclosure, the processor (130) performs barcode detection or QR code detection to recognize the presence of a payment code (1000) and the size of an occupied area (1031, 1032, 1033) occupied by the payment code (1000) from a detection area (1021, 1022, 1023) determined within a plurality of images (1011, 1012, 1013).

[0115] In the embodiment illustrated in FIG. 10, the processor (130) can recognize the presence of a payment code (1000) and the size of an occupied area (1031) occupied by the payment code (1000) from a first detection area (1021) having a preset size centered on the two-dimensional position coordinate value of the first point (P1) corresponding to the gaze point (G) on the first image (1011). Similarly, the processor (130) can recognize the presence of a payment code (1000) and the size of an occupied area (1032) occupied by the payment code (1000) from a second detection area (1022) having a preset size centered on the two-dimensional position coordinate value of the second point (P2) on the second image (1012), and can recognize the presence of a payment code (1000) and the size of an occupied area (1033) occupied by the payment code (1000) from a third detection area (1023) having a preset size centered on the two-dimensional position coordinate value of the third point (P3) on the third image (1013).

[0116] In step S930 of FIG. 9, if a payment code exists, the augmented reality device (100) selects a camera among the plurality of cameras that acquires an image with the largest size of the area occupied by the payment code. Referring also to FIG. 10, if a payment code (1000) is detected as a result of payment code detection, the processor (130) of the augmented reality device (100) may select an image with the largest size of the area occupied by the payment code (1000), and select a camera among the plurality of cameras (121, 122, 123) that captures the selected image. For example, if the area size of the occupied area (1032) from the second image (1012) is the largest among the sizes of the occupied areas (1031, 1032, 1033) of the payment code (1000) within the plurality of images (1011, 1012, 1013), the processor (130) may select the second camera (122) configured to capture the second image (1012) among the plurality of cameras (121, 122, 123).

[0117] Referring again to FIG. 9, in step S520, the augmented reality device (100) obtains a two-dimensional position coordinate value corresponding to the gaze point from the second image (1012) obtained by photographing the payment code (1000) using the selected second camera (122), and determines an area of ​​a preset size as the area of ​​interest based on the two-dimensional position coordinate value.

[0118] In the embodiment illustrated in FIGS. 9 and 10, when the physical size of the payment code (1000) is not known in advance, the augmented reality device (100) may recognize the presence of the payment code (1000) among the plurality of images (1011, 1012, 1013) acquired using the plurality of cameras (121, 122, 123), identify an image (e.g., the second image (1022)) having the largest area size of the occupied area (1031, 1032, 1033) of the payment code (1000), and select the second camera (122) configured to capture the identified image among the plurality of cameras (121, 122, 123). Through this, the augmented reality device (100) according to one embodiment of the present disclosure can improve the recognition rate of the payment code (1000) by selecting the camera that can capture the payment code (1000) at the largest size when the magnification or field of view (FOV) of each of the plurality of cameras (121, 122, 123) is different.

[0119] FIG. 11 is a flowchart illustrating an operation method for recognizing a payment code from a plurality of images acquired using a plurality of cameras by an augmented reality device (100) according to one embodiment of the present disclosure.

[0120] Steps S1110 to S1130 illustrated in FIG. 11 are steps that specify the operation of step S220 of FIG. 2. Step S1110 of FIG. 11 may be performed before or after the operation of step S210 of FIG. 2 is performed. Step S1140 illustrated in FIG. 11 is a step that specifies the operation of step S230 of FIG. 2. Step S1150 illustrated in FIG. 11 is a step that specifies step S240 of FIG. 2. After the operation of step S1150 of FIG. 11 is performed, step S250 of FIG. 2 may be performed.

[0121] In step S1110, the augmented reality device (100) acquires multiple images by photographing an area corresponding to a gaze point using multiple cameras. In one embodiment of the present disclosure, the augmented reality device (100) can acquire multiple images by photographing an area including a payment code viewed by a user in real space using multiple cameras (121, 122, 123).

[0122] In step S1120, the augmented reality device (100) obtains two-dimensional position coordinate information corresponding to a gaze point from each of a plurality of images. The augmented reality device (100) can identify a point corresponding to a gaze point from each of a plurality of images and obtain a two-dimensional position coordinate value of the point. In one embodiment of the present disclosure, the augmented reality device (100) can obtain two-dimensional position coordinate values ​​of points corresponding to the gaze point by mapping a gaze point in a three-dimensional space onto a plurality of images based on at least one of a position at which a plurality of cameras are mounted, a field of view, or an intrinsic parameter of the camera.

[0123] In step S1130, the augmented reality device (100) determines a set area as a plurality of regions of interest based on two-dimensional position coordinate information among a plurality of images. In one embodiment of the present disclosure, the augmented reality device (100) may determine an area having a preset size centered on two-dimensional position coordinate values ​​of points corresponding to gaze points on a plurality of images as a region of interest.

[0124] In step S1140, the augmented reality device (100) enlarges each of the plurality of regions of interest at a preset zoom ratio. In one embodiment of the present disclosure, the augmented reality device (100) may enlarge the plurality of regions of interest using a digital zoom or an optical zoom function of multiple cameras.

[0125] In step S1150, the augmented reality device (100) recognizes a payment code from multiple areas of interest.

[0126] In the embodiment of FIG. 11, the augmented reality device (100) can simultaneously determine multiple regions of interest from multiple images acquired using multiple cameras, simultaneously enlarge the multiple regions of interest, and simultaneously recognize payment codes from the multiple regions of interest. However, the present invention is not limited thereto, and in one embodiment of the present disclosure, the augmented reality device (100) can sequentially acquire images using each of the multiple cameras, determine regions of interest from the sequentially acquired images, enlarge the regions of interest, and recognize payment codes. An embodiment in which the augmented reality device (100) sequentially recognizes payment codes from multiple images will be described in detail with reference to FIG. 12.

[0127] FIG. 12 is a flowchart illustrating an operation method of recognizing a payment code from a plurality of images acquired using a plurality of cameras and performing a payment process by an augmented reality device (100) according to one embodiment of the present disclosure.

[0128] In step S1210, the augmented reality device (100) selects a k-th camera among n cameras that capture an area corresponding to a gaze point. In one embodiment of the present disclosure, the augmented reality device (100) may include a plurality (n) of cameras. For example, n may be 3, but is not limited thereto. The augmented reality device (100) may include two or more cameras. The augmented reality device (100) may select a k-th camera among the n cameras. Here, k represents a value greater than or equal to 1 and less than or equal to n.

[0129] In step S1220, the augmented reality device (100) determines a region of interest based on a two-dimensional position coordinate corresponding to a gaze point on an image acquired through the kth camera. In one embodiment of the present disclosure, the augmented reality device (100) can identify a point mapped to a gaze point in a three-dimensional space from an image acquired through the kth camera among n cameras, and obtain a two-dimensional position coordinate value of the identified point. The augmented reality device (100) can determine an area of ​​a preset size centered on the two-dimensional position coordinate value corresponding to the gaze point as a region of interest.

[0130] In step S1230, the augmented reality device (100) enlarges the determined region of interest at a preset zoom ratio. In one embodiment of the present disclosure, the augmented reality device (100) may enlarge the region of interest using a digital zoom or an optical zoom function of the k-th camera.

[0131] In step S1240, the augmented reality device (100) can determine whether a payment code is recognized in the enlarged image area. In one embodiment of the present disclosure, the augmented reality device (100) can perform barcode detection or QR code detection to determine whether a payment code is recognized from the enlarged image area.

[0132] If it is determined that the payment code is recognized (step S1250), the augmented reality device (100) outputs a user interface requesting payment approval.

[0133] In step S1260, the augmented reality device (100) executes a payment application and performs a payment process using a payment code.

[0134] If it is determined that the payment code is not recognized normally (e.g., step S1270), the augmented reality device (100) determines whether the k value is equal to the n value. In one embodiment of the present disclosure, if it is determined that the payment code is not recognized normally, the augmented reality device (100) can re-enlarge the region of interest by increasing the zoom magnification of the image acquired by the kth camera. If the payment code is not recognized even through re-enlargement, the augmented reality device (100) can determine whether the k value is equal to the n value or whether the k value is less than the n value.

[0135] If the k value is equal to the n value (e.g., step S1280), the augmented reality device (100) outputs a user interface indicating a payment code recognition failure. The augmented reality device (100) may output a notification message indicating a payment failure, such as, for example, "Payment was declined because the barcode (or QR code) was not recognized."

[0136] If the k value is not equal to the n value (e.g., step S1290), the augmented reality device (100) may increase the index of the camera. In one embodiment of the present disclosure, the augmented reality device (100) may increase the k value by 1 to select the (k+1)th camera. The augmented reality device (100) may perform steps S1210 to S1270 again using the (k+1)th camera. For example, the augmented reality device (100) may repeatedly perform the following operations: selecting the k+1th camera (e.g., S1210), determining a region of interest on an image acquired using the k+1th camera (e.g., S1220), enlarging the determined region of interest with a preset zoom ratio (e.g., S1230), determining whether a payment code is recognized (e.g., S1240), executing a payment application to perform a payment process based on the determination result (e.g., S1260), or comparing the k+1 value with the n value (e.g., S1270). If the k+1 value is equal to the n value (e.g., S1280), the augmented reality device (100) may output a notification message indicating a payment code recognition failure. If the k+1 value is not equal to the n value (e.g., S1290), the augmented reality device (100) may increase the camera index by 1 from k+1 to k+2.

[0137] In the embodiment illustrated in FIGS. 11 and 12, the augmented reality device (100) determines a region of interest from a plurality of images acquired through a plurality of cameras, and recognizes a payment code by enlarging the region of interest, thereby improving the recognition rate and recognition accuracy of the payment code.

[0138] FIG. 13 is a flowchart illustrating an operation method of an augmented reality device (100) according to one embodiment of the present disclosure for obtaining a high-quality image and determining a region of interest from the high-quality image.

[0139] Steps S1310 to S1330 illustrated in FIG. 13 are steps that specify the operation of step S220 of FIG. 2. Step S1310 illustrated in FIG. 13 may be performed before or after the operation of step S210 of FIG. 2 is performed. After the operation of step S1330 illustrated in FIG. 13 is performed, step S230 of FIG. 2 may be performed.

[0140] In step S1310, the augmented reality device (100) acquires multiple images by photographing an area corresponding to a gaze point through multiple cameras. Step S1310 is identical to step S1110 illustrated in FIG. 11, and therefore, a duplicate description is omitted.

[0141] In step S1320, the augmented reality device (100) performs image processing using a plurality of images to obtain a high-quality image with improved image quality parameters. In one embodiment of the present disclosure, the processor (130) of the augmented reality device (100) may perform up-scaling, such as super resolution, using a plurality of images to obtain a high-quality image. In addition, in one embodiment of the present disclosure, the processor (130) may perform image processing, such as noise removal or contrast adjustment (e.g., auto contrast), using a plurality of images to correct the image quality of the image, thereby obtaining a high-quality image. Since up-scaling, noise removal, or contrast adjustment are known image quality improvement techniques, a detailed description thereof will be omitted. In one embodiment of the present disclosure, the processor (130) may obtain a high-quality image using any known image processing technique using a plurality of images in addition to the above-described techniques.

[0142] In step S1330, the augmented reality device (100) determines a region of interest based on two-dimensional position coordinate information corresponding to a gaze point on a high-quality image. In one embodiment of the present disclosure, the augmented reality device (100) may obtain two-dimensional position coordinate values ​​corresponding to a gaze point in a three-dimensional space from the high-quality image, and determine a region of a preset size as a region of interest based on the two-dimensional position coordinate values. Step S1330 is the same as step S220 illustrated in FIG. 2 except that the object of determining the region of interest is a high-quality image, and therefore, redundant descriptions are omitted.

[0143] In the embodiment illustrated in FIG. 13, the augmented reality device (100) can acquire a high-quality image using multiple images acquired through multiple cameras, determine a region of interest from the high-quality image, and recognize a payment code by enlarging the region of interest at a preset zoom ratio. The augmented reality device (100) according to one embodiment of the present disclosure can recognize a payment code from a high-quality image, thereby improving the payment code recognition rate and recognition accuracy.

[0144] FIG. 14 is a diagram illustrating an operation of an augmented reality device (100) according to one embodiment of the present disclosure recognizing text using information on a user's gaze direction and enlarging and displaying the text.

[0145] Referring to FIG. 14, the augmented reality device (100) may include at least one gaze tracking sensor (110) and a camera (120). The augmented reality device (100) obtains gaze information of both eyes of the user using the at least one gaze tracking sensor (110), and obtains a gaze point (G) where the gaze directions of both eyes of the user converge in a three-dimensional space based on the obtained gaze information. This is illustrated in operation 1 of FIG. 14.

[0146] The augmented reality device (100) determines an area of ​​interest (1430) based on a gaze point (G) on an image (1420) acquired through a camera (120). This is illustrated in operation 2 of FIG. 14.

[0147] The augmented reality device (100) enlarges the area of ​​interest (1430). This is illustrated in operation 3 of FIG. 14.

[0148] The augmented reality device (100) recognizes a menu (1400) from an enlarged area of ​​interest (1440) of an image (1420), performs OCR (Optical Character Recognition) to extract text from the menu (1400), and outputs the extracted text. This is illustrated in operation 4 of FIG. 14.

[0149] Hereinafter, the function and / or operation of a specific augmented reality device (100) will be described in detail with reference to the embodiment illustrated in FIG. 14 together with FIG. 15.

[0150] FIG. 15 is a flowchart illustrating an operation method of an augmented reality device (100) according to one embodiment of the present disclosure recognizing text using information on a user's gaze direction and displaying the text in an enlarged manner.

[0151] Referring to FIG. 15, in step S1510, the augmented reality device (100) executes a magnifying glass application. In the present disclosure, the 'magnifying glass application' is an augmented reality application that captures an area viewed by a user in a real space using a camera (120) and enlarges and displays the captured area. In one embodiment of the present disclosure, the augmented reality device (100) outputs a virtual object configured as a graphical user interface for executing the magnifying glass application, and recognizes a hand pointing input of a user for selecting the outputted virtual object to execute the magnifying glass application. However, the present disclosure is not limited thereto, and the augmented reality device (100) may receive a voice input from a user instructing execution of the magnifying glass application, and execute the magnifying glass application upon receiving the voice input.

[0152] In step S1520, the augmented reality device (100) acquires a gaze point where the gaze directions of the user's both eyes converge in a three-dimensional space based on the user's gaze information acquired through at least one gaze tracking sensor. In one embodiment of the present disclosure, the augmented reality device (100) may include a left-eye gaze tracking sensor that detects the gaze direction of the user's left eye and a right-eye gaze tracking sensor that detects the gaze direction of the user's right eye. Detecting the user's gaze direction may include an operation of acquiring gaze information related to the user's gaze. In combination with operation 1 of FIG. 14, the gaze tracking sensor (110) may include a left-eye gaze tracking sensor and a right-eye gaze tracking sensor. The augmented reality device (100) may acquire information about the gaze direction of the user's left eye through the left-eye gaze tracking sensor and may acquire information about the gaze direction of the user's right eye through the right-eye gaze tracking sensor. The augmented reality device (100) can detect a gaze point (G) where the gaze directions of the user's left eye and right eye converge, and obtain three-dimensional position coordinate information of the gaze point (G). In the embodiment illustrated in FIG. 14, the user is looking at a menu board (1400) of a restaurant or cafe in real space, and the gaze point (G) can be located on the menu board (1400).

[0153] In step S1530, the augmented reality device (100) determines an area of ​​interest based on two-dimensional position coordinate information corresponding to the gaze point on the image acquired through the camera. In combination with operation 2 of FIG. 14, the augmented reality device (100) can acquire an image (1420) by using the camera (120) to capture an area viewed by the user, i.e., an area corresponding to the gaze point (G). In step S1530, a point (P) corresponding to the gaze point (G) in three-dimensional space is acquired from the image (1420). 2D) may be the same as operation 2 of FIG. 1 and step S220 of FIG. 2, so redundant descriptions are omitted. The augmented reality device (100) acquires the acquired point (P 2D ) can be determined as a region of interest (1430) with a preset size within the image (1420) centered on the two-dimensional position coordinates. In the embodiment illustrated in FIG. 14, the region of interest (1430) can include the image area of ​​the menu board (1400).

[0154] In step S1540, the augmented reality device (100) enlarges the region of interest (1430) at a preset zoom ratio. In combination with operation 3 of FIG. 14, the augmented reality device (100) can enlarge the region of interest (1430) at a preset plurality (n) of zoom ratios. In one embodiment of the present disclosure, the augmented reality device (100) can enlarge the region of interest (1430) by using a digital zoom or optical zoom function of the camera (120).

[0155] In step S1550, the augmented reality device (100) performs OCR (Optical Character Recognition) to determine whether text can be extracted. Referring also to FIG. 14, the augmented reality device (100) can perform OCR to determine whether text can be extracted from an enlarged region of interest (1440) within an image (1420).

[0156] If it is determined that text can be extracted from the enlarged image (e.g., step S1560), the augmented reality device (100) displays the enlarged image and outputs the text. In combination with operation 4 of FIG. 14, the augmented reality device (100) can perform OCR to extract text from the enlarged region of interest (1440) of the image (1420). In the embodiment illustrated in FIG. 14, the augmented reality device (100) can extract the text 'CHICKEN KARAHI' from the enlarged region of interest (1440). The augmented reality device (100) can output the extracted text, for example, 'CHICKEN KARAHI', through the display unit (150). In one embodiment of the present disclosure, the augmented reality device (100) can output the text by generating a virtual object including the extracted text and projecting light of the generated virtual object onto a waveguide through an optical projector of the display unit (150).

[0157] If it is determined that text extraction from the enlarged image is possible (e.g., step S1570), the augmented reality device (100) determines whether the maximum magnification of the camera has been reached. Referring also to FIG. 14, the augmented reality device (100) can determine whether the zoom magnification that enlarges the area of ​​interest is the maximum zoom magnification by digital zoom or optical zoom of the camera (120).

[0158] If it is determined that the maximum zoom ratio of the camera has been reached (e.g., step S1580), the augmented reality device (100) outputs a user interface indicating that text extraction is not possible. The augmented reality device (100) may output a notification message indicating that text extraction is not possible, such as, for example, "Text was not recognized."

[0159] If it is determined that the maximum zoom ratio of the camera has not been reached (e.g., step S1590), the augmented reality device (100) increases the zoom ratio to enlarge the image area. In one embodiment of the present disclosure, the augmented reality device (100) may enlarge the area of ​​interest by increasing the zoom ratio compared to the existing zoom ratio. Thereafter, the augmented reality device (100) may return to step S1540 and re-determine whether text is extracted through OCR from the image area enlarged with the increased zoom ratio.

[0160] In the embodiments illustrated in FIGS. 14 and 15, the augmented reality device (100) determines a region of interest based on the user's gaze information, and can recognize text by enlarging the region of interest with a preset zoom ratio. However, the present invention is not limited thereto, and the augmented reality device (100) according to an embodiment of the present disclosure can expand its usability by recognizing text by enlarging the region of interest determined based on the user's gaze information, and providing the recognized text as input to an application that performs searching, translating, or TTS (Text-to-Speech) voice output. That is, the augmented reality device (100) of the present disclosure is not limited to recognizing a payment code as described with reference to FIGS. 1 to 13, and the same technical idea can be utilized in various applications such as recognizing, searching, translating, or TTS voice outputting text from objects in a real space (e.g., a restaurant menu, a product description, a bus stop sign, etc.), thereby expanding the usability of the device and improving user convenience.

[0161] One aspect of the present disclosure provides a method for an augmented reality device (100) to perform payment using a user's gaze information. An operating method of an augmented reality device (100) according to an embodiment of the present disclosure may include a step (e.g., S210) of obtaining a gaze point where the gaze directions of both eyes of a user converge, based on gaze information of both eyes of the user obtained using at least one gaze tracking sensor (110). The operating method of the augmented reality device (100) may include a step (e.g., S220) of determining a region of interest corresponding to the gaze point on an image obtained through a camera (120). The operating method of the augmented reality device (100) may include a step (e.g., S230) of enlarging the determined region of interest. The operating method of the augmented reality device (100) may include a step (e.g., S240) of recognizing a payment code included in the enlarged region of interest. The operating method of the augmented reality device (100) may include a step of performing payment using the recognized payment code.

[0162] In one embodiment of the present disclosure, the camera (120) may include a plurality of cameras having different magnifications. The method of operating the augmented reality device (100) may further include a step (e.g., S510) of selecting a camera having an optimal magnification for recognizing a payment code among the plurality of cameras based on a distance between a gaze point and a user. In the step (e.g., S220) of determining a region of interest, the augmented reality device (100) may determine a region of interest from an image obtained by photographing a payment code using the selected camera.

[0163] In one embodiment of the present disclosure, in the step of selecting the camera (120) (e.g., S510), the augmented reality device (100) may select a camera with a high magnification among a plurality of cameras in proportion to the distance between the gaze point and the user.

[0164] In one embodiment of the present disclosure, information regarding the size of the payment code may be acquired in advance and stored in the memory (140). The step of selecting the camera (120) (e.g., S510) may include a step of calculating the size of an area surrounding the payment code by projecting the payment code onto a plurality of images acquired through a plurality of cameras based on the information regarding the size of the payment code (e.g., S710), and a step of selecting a camera that acquires an image having the largest size of the calculated area among the plurality of cameras (e.g., S720).

[0165] In one embodiment of the present disclosure, the step of selecting the camera (120) (e.g., S510) may include a step of obtaining two-dimensional position coordinate information corresponding to a gaze point from a plurality of images acquired through a plurality of cameras (e.g., S910), and a step of performing payment code detection to recognize the presence of a payment code and the size of the area occupied by the payment code from a set area based on the two-dimensional position coordinate information of the gaze point (e.g., S920). The step of selecting the camera (120) (e.g., S510) may include a step of selecting a camera that acquires an image having the largest size of the area occupied by the payment code among the plurality of cameras when the recognition result shows that a payment code exists (e.g., S930).

[0166] In one embodiment of the present disclosure, the camera (120) may include a plurality of cameras. The step of determining the region of interest (e.g., S220) may include a step of obtaining a plurality of images by photographing a region corresponding to a gaze point using the plurality of cameras (e.g., S1110), a step of obtaining two-dimensional position coordinate information corresponding to the gaze point from each of the obtained plurality of images (e.g., S1120), and a step of determining a set region as a plurality of regions of interest based on the two-dimensional position coordinate information obtained from among the plurality of images (e.g., S1130). The step of enlarging the region of interest (e.g., S230) may include a step of enlarging each of the plurality of regions of interest with a preset zoom ratio (e.g., S1140). The step of recognizing the payment code (e.g., S240) may include a step of recognizing the payment code from the enlarged plurality of regions of interest (e.g., S1150).

[0167] In one embodiment of the present disclosure, if recognition of the payment code from the plurality of regions of interest fails, the augmented reality device (100) may re-perform the steps of determining a region of interest from an image captured using a camera other than the plurality of cameras (e.g., S220), enlarging the region of interest with a preset zoom ratio (e.g., S230), and recognizing the payment code from the enlarged image region (e.g., S240).

[0168] In one embodiment of the present disclosure, the camera (120) may include a plurality of cameras. The step of determining the region of interest (e.g., S220) may include a step of obtaining a plurality of images by photographing an area corresponding to a gaze point using the plurality of cameras (e.g., S1310), and a step of performing image processing using the obtained plurality of images to obtain a high-quality image with improved image quality parameters including at least one of resolution, contrast ratio, and noise (e.g., S1320). The step of determining the region of interest (e.g., S220) may include a step of determining the region of interest (e.g., S1330) based on two-dimensional position coordinate information corresponding to the gaze point on the obtained high-quality image.

[0169] In one embodiment of the present disclosure, in the step of enlarging the region of interest (e.g., S230), the augmented reality device (100) can enlarge the region of interest at a preset zoom ratio.

[0170] In one embodiment of the present disclosure, if the payment code recognition fails, the augmented reality device (100) can increase the zoom magnification for the area of ​​interest and enlarge the area of ​​interest using the increased zoom magnification.

[0171] In one embodiment of the present disclosure, the method of operating the augmented reality device (100) may further include a step of determining whether the increased zoom magnification has reached the maximum magnification of the camera (120). The method of operating the augmented reality device (100) may further include a step of outputting a user interface indicating a payment code recognition failure when the increased zoom magnification has reached the maximum magnification according to the determination result.

[0172] Another aspect of the present disclosure provides an augmented reality device (100) that performs payment using a user's gaze information. In one embodiment of the present disclosure, the augmented reality device (100) may include at least one gaze tracking sensor (110) that acquires gaze information of both eyes of the user, a camera (120) that acquires an image, a memory (140) that stores at least one instruction, and at least one processor (130) that executes the at least one instruction. The at least one processor (130) may acquire a gaze point where the gaze directions of the user's both eyes converge based on the gaze information of the user's both eyes acquired through the at least one gaze tracking sensor (110). The at least one processor (130) may determine a region of interest corresponding to the gaze point on an image acquired through the camera (120). The at least one processor (130) may enlarge the determined region of interest. The at least one processor (130) may recognize a payment code included in the enlarged region of interest. The above at least one processor (130) can perform payment using the recognized payment code.

[0173] In one embodiment of the present disclosure, the camera (120) may include multiple cameras with different magnifications. The at least one processor (130) may select a camera with an optimal magnification for recognizing the payment code from among the multiple cameras based on the distance between the gaze point and the user. The at least one processor (130) may determine a region of interest from an image obtained by photographing the payment code using the selected camera.

[0174] In one embodiment of the present disclosure, the at least one processor (130) may select a camera with a high magnification among a plurality of cameras in proportion to the distance between the gaze point and the user.

[0175] In one embodiment of the present disclosure, information regarding the size of a payment code may be acquired in advance and stored in memory (140). The at least one processor (130) may calculate the size of an area surrounding the payment code by projecting the payment code onto multiple images acquired through multiple cameras, based on the information regarding the size of the payment code. The at least one processor (130) may select a camera among the multiple cameras that acquires an image with the largest calculated area size.

[0176] In one embodiment of the present disclosure, the at least one processor (130) may obtain two-dimensional position coordinate information corresponding to a gaze point from a plurality of images acquired through a plurality of cameras. The at least one processor (130) may perform payment code detection, and recognize whether a payment code exists and the size of the area occupied by the payment code from a set area based on the two-dimensional position coordinate information of the gaze point. If the recognition result shows that a payment code exists, the at least one processor (130) may select a camera that acquires an image with the largest size of the area occupied by the payment code among the plurality of cameras.

[0177] In one embodiment of the present disclosure, the camera (120) includes a plurality of cameras, and the plurality of cameras can capture an area corresponding to a gaze point to obtain a plurality of images. The at least one processor (130) can obtain two-dimensional position coordinate information corresponding to the gaze point from each of the plurality of images obtained through the plurality of cameras, and determine a set area as a plurality of regions of interest based on the two-dimensional position coordinate information obtained from among the plurality of images. The at least one processor (130) can enlarge each of the plurality of regions of interest with a preset zoom ratio. The at least one processor (130) can recognize a payment code from the enlarged plurality of regions of interest.

[0178] In one embodiment of the present disclosure, the camera (120) includes a plurality of cameras, and the plurality of cameras can acquire a plurality of images by photographing an area corresponding to a gaze point. The at least one processor (130) can perform image processing using the plurality of images acquired through the plurality of cameras to acquire a high-quality image with improved image quality parameters including at least one of resolution, contrast ratio, and noise. The at least one processor (130) can determine a region of interest based on two-dimensional position coordinate information corresponding to the gaze point on the acquired high-quality image.

[0179] In one embodiment of the present disclosure, the at least one processor (130) can magnify the region of interest at a preset zoom ratio.

[0180] In one embodiment of the present disclosure, when payment code recognition fails, the at least one processor (130) can increase a zoom factor for a region of interest and enlarge the region of interest using the increased zoom factor.

[0181] In one embodiment of the present disclosure, the augmented reality device (100) may further include a display unit (150). The at least one processor (130) may determine whether the increased zoom magnification has reached the maximum magnification of the camera (120). If the increased zoom magnification has reached the maximum magnification according to the determination result, the at least one processor (130) may control the display unit (150) to output a user interface indicating a payment code recognition failure.

[0182] Another aspect of the present disclosure provides a computer program product including a computer-readable storage medium. The storage medium may include instructions readable by an augmented reality device (100), such that the augmented reality device (100) performs the following operations: obtaining a gaze point where the gaze directions of both eyes of the user converge based on gaze information of both eyes of the user obtained through at least one gaze tracking sensor (110); determining a region of interest corresponding to the gaze point on an image obtained through a camera (120); enlarging the determined region of interest; recognizing a payment code from the enlarged region of interest; and performing a payment using the recognized payment code.

[0183] The program executed by the augmented reality device (100) described in the present disclosure may be implemented as hardware components, software components, and / or a combination of hardware components and software components. The program may be executed by any system capable of executing computer-readable instructions.

[0184] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to do a desired thing or may independently or collectively command a processing device to do a desired thing.

[0185] Software may be implemented as a computer program containing instructions stored on a computer-readable storage medium. Examples of computer-readable storage media include magnetic storage media (e.g., read-only memory (ROM), random-access memory (RAM), floppy disks, hard disks, etc.) and optical readable media (e.g., CD-ROMs, DVDs (Digital Versatile Discs)). The computer-readable storage media may be distributed across network-connected computer systems, so that computer-readable code may be stored and executed in a distributed manner. The media may be readable by a computer, stored in a memory, and executed by a processor.

[0186] A computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium does not contain signals and is tangible, but does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0187] Additionally, programs according to the embodiments disclosed herein may be provided as part of a computer program product. The computer program product may be traded as a commodity between sellers and buyers.

[0188] A computer program product may include a software program, a computer-readable storage medium having the software program stored thereon. For example, the computer program product may be provided by a manufacturer of an augmented reality device (100) or an electronic market (e.g., Samsung Galaxy Store). TM ) may include a product in the form of a software program (e.g., a downloadable application) that is distributed electronically. For electronic distribution, at least a portion of the software program may be stored in a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server of the augmented reality device (100), a server of an electronic market, or a relay server that temporarily stores the software program.

[0189] The computer program product may include a storage medium of the server or the storage medium of the augmented reality device (100) in a system comprising an augmented reality device (100) and / or a server. In some embodiments, if there is a third device (e.g., a mobile device such as a smart phone) that is in communication with the augmented reality device (100), the computer program product may include a storage medium of the third device. In some embodiments, the computer program product may include a software program itself that is transmitted from the augmented reality device (100) to the third device or from the third device to an electronic device.

[0190] In this case, either the augmented reality device (100) or the third device may execute the computer program product to perform the method according to the disclosed embodiments. In some embodiments, at least one of the augmented reality device (100) and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0191] For example, the augmented reality device (100) may execute a computer program product stored in the memory (140, see FIG. 4) to control another electronic device (e.g., a mobile device such as a smart phone) that is in communication with the augmented reality device (100) to perform a method according to the disclosed embodiments.

[0192] As another example, a third device may execute a computer program product to control an electronic device in communication with the third device to perform a method according to the disclosed embodiment.

[0193] When a third device executes a computer program product, the third device may download the computer program product from the augmented reality device (100) and execute the downloaded computer program product. In some embodiments, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the disclosed embodiments.

[0194] Although the embodiments described above have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components such as the described computer system or modules are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

Claims

1. A method for performing payment using a user's gaze information, performed by one or more processors of an augmented reality device, A step of obtaining a gaze point where the gaze directions of the user's two eyes converge based on the gaze information of the user's two eyes, wherein the gaze information is obtained by at least one gaze tracking sensor; A step of determining a region of interest corresponding to the gaze point on an image acquired through a camera; A step of enlarging the above area of ​​interest; A step of recognizing a payment code included in the above expanded area of ​​interest; and A step for performing payment using the above payment code; A method comprising:

2. In paragraph 1, The above camera comprises multiple cameras having different magnifications, The above method, A step of selecting an optimal camera having an optimal zoom ratio for recognizing the payment code among the plurality of cameras based on the distance between the gaze point and the user; and A method for determining the region of interest corresponding to the gaze point from the image obtained by photographing the payment code based on using the optimal camera.

3. In paragraph 2, A method according to claim 1, wherein the optimal zoom scale for recognizing the payment code is the highest zoom scale in proportion to the distance between the gaze point and the user.

4. In paragraph 2, Information about the size of the above payment code is obtained in advance and stored in the memory of the augmented reality device, The steps for selecting the above camera are: A step of calculating the size of an area surrounding the payment code by projecting the payment code onto a plurality of images acquired through the plurality of cameras based on information about the size of the payment code; and A step of selecting a large region camera among the plurality of cameras that acquires an image having the largest size of the calculated region; A method comprising:

5. In paragraph 2, The steps for selecting the above camera are: A step of obtaining two-dimensional position coordinate information corresponding to the gaze point from a plurality of images obtained through the plurality of cameras; A step of recognizing the presence of the payment code from a set area based on two-dimensional position coordinate information of the gaze point in the plurality of images acquired by the plurality of cameras, and determining the size of the area occupied by the payment code; and As a result of the recognition, if the payment code exists, a step of selecting a large code camera that acquires an image having the largest size of the area occupied by the payment code among the plurality of cameras; A method comprising:

6. In paragraph 1, The above camera comprises multiple cameras, The steps for determining the above area of ​​interest are: A step of obtaining multiple images by photographing an area corresponding to the gaze point using the multiple cameras; A step of obtaining two-dimensional position coordinate information corresponding to the gaze point from each of the plurality of images; and A step of determining a region set based on the two-dimensional position coordinate information among the plurality of images as a plurality of regions of interest; Including, The step of enlarging the region of interest comprises enlarging each of the plurality of regions of interest with a preset zoom ratio, The step of recognizing the payment code is a method of recognizing the payment code from the enlarged plurality of areas of interest.

7. In paragraph 1, The above camera comprises multiple cameras, The steps for determining the above area of ​​interest are: A step of obtaining multiple images by photographing an area corresponding to the gaze point using the multiple cameras; A step of performing image processing using the acquired plurality of images to acquire a high-quality image with improved image quality parameters, wherein the image quality parameters include at least one of resolution, contrast ratio, and noise; and A step of determining the region of interest based on two-dimensional position coordinate information corresponding to the gaze point on the high-quality image; A method comprising:

8. In an augmented reality device that performs payment using the user's gaze information, At least one gaze tracking sensor for obtaining gaze information of both eyes of the user; At least one camera for acquiring images; a memory storing at least one instruction; and At least one processor executing at least one of the above instructions; Including, The at least one processor executes the at least one instruction: Based on the gaze information of the user's two eyes, a gaze point where the gaze directions of the user's two eyes converge is obtained, and the gaze information is obtained using at least one gaze tracking sensor. Determine the region of interest corresponding to the gaze point on the image acquired through the above camera, Expand the above area of ​​interest, Recognizes payment codes included in the expanded area of ​​interest; An augmented reality device that performs payment using the above payment code.

9. In paragraph 8, The above camera comprises multiple cameras having different magnifications, The at least one processor executes the at least one instruction: Based on the distance between the above gaze point and the user, the optimal camera having the optimal magnification for recognizing the payment code is selected from among the plurality of cameras, An augmented reality device that determines the area of ​​interest corresponding to the gaze point by photographing the payment code using the selected camera.

10. In paragraph 9, An augmented reality device, wherein the optimal zoom scale for recognizing the payment code is the highest zoom scale in proportion to the distance between the gaze point and the user.

11. In paragraph 9, Information about the size of the above payment code is obtained in advance and stored in the memory, The at least one processor executes the at least one instruction: Based on information about the size of the payment code, the size of the area surrounding the payment code is calculated by projecting the payment code onto each of the plurality of images acquired through the plurality of cameras, An augmented reality device, wherein a wide-area camera is selected from among the plurality of cameras to acquire an image having the largest size of the calculated area.

12. In paragraph 9, The at least one processor executes the at least one instruction: Obtain two-dimensional position coordinate information corresponding to the gaze point from multiple images acquired through the multiple cameras, Recognize whether the payment code exists in a set area based on the two-dimensional position coordinate information of the gaze point in the plurality of images acquired by the plurality of cameras, and determine the size of the area occupied by the payment code. An augmented reality device, wherein, as a result of the recognition, if the payment code exists, a large code camera is selected to acquire an image having the largest size of the area occupied by the payment code among the plurality of cameras.

13. In paragraph 8, The above camera comprises a plurality of cameras, and the plurality of cameras acquire a plurality of images by photographing an area corresponding to the gaze point, The at least one processor executes the at least one instruction: By performing image processing using multiple images acquired through the multiple cameras, a high-quality image with improved image quality parameters is acquired, and the image quality parameters include at least one of resolution, contrast ratio, and noise. An augmented reality device that determines the region of interest based on two-dimensional position coordinate information corresponding to the gaze point on the high-quality image.

14. In paragraph 8, The at least one processor executes the at least one instruction: An augmented reality device that increases the zoom magnification for the area of ​​interest and enlarges the area of ​​interest using the increased zoom magnification when the recognition of the payment code fails.

15. Including a non-transitory computer-readable recording medium, The above recording medium An operation of obtaining a gaze point where the gaze directions of the user's two eyes converge based on the gaze information of the user's two eyes, wherein the gaze information is obtained by at least one gaze tracking sensor; An action of determining a region of interest corresponding to the gaze point on an image acquired through a camera; An action to enlarge the above area of ​​interest; An action to recognize a payment code included in the expanded area of ​​interest; and An action to perform a payment using the above payment code; A recording medium comprising one or more instructions executed by an augmented reality device to perform a task.

Citation Information

Patent Citations

  • Image identification method and device, and verification device

    CN108229399A

  • Image recognition method based on eye movement fixation point guidance, MR glasses and medium

    CN112507799A

  • User terminal device and shopping system

    JP2012008746A

  • Two-dimensional code reader

    JP2021114151A

  • Pointer tracking for eye-level scanners and displays

    US20150168727A1