Electronic device for acquiring feature point from encoded image and method of operating same

Abstract

An electronic device for acquiring a feature point from an encoded image and a method of operating the electronic device are provided. An electronic device according to an embodiment of the present disclosure may receive light passing through a pattern mask through an image sensor, thereby acquiring a phase-modulated encoded image, and may acquire eye-related feature points from the encoded image. The electronic device may acquire gaze information of the user based on the feature points.

Description

Technical Field

[0001] The present invention generally relates to an electronic device for obtaining feature points from a coded image, and a method for operating the electronic device. Background Art

[0002] Recently, in various applications using an augmented reality (AR) or virtual reality (VR) device, information about the gaze direction of a user wearing the device is often needed. The gaze direction information may be used for various operations, such as an operation of building a user interface, an operation of optimizing the rendering of an image provided to the user (e.g., foveated rendering), or an operation of determining the distance to an object the user is looking at. The gaze direction information may be generated by an eye tracking (ET) sensor of the user.

[0003] However, camera systems for tracking gaze typically include bulky lens assemblies, and efforts have been made to miniaturize AR devices on which the camera systems are mounted without degrading performance.

[0004] Furthermore, in a camera system of the related art, after designing an optical system including a lens, parameters in an image processing algorithm may be adjusted to output a clearer image. Summary of the invention

[0005] Technical Solution Provided are an electronic device for extracting feature points from an image distorted when passing through a pattern mask and tracking gaze based on the feature points, and an operating method of the electronic device.

[0006] According to one aspect of the present disclosure, an electronic device may include: a light source configured to output light; a pattern mask configured to change the path of light passing through a pattern of the pattern mask; an image sensor configured to receive light output from the light source, reflected by an eye, and passing through the pattern mask; and at least one processor configured to obtain a phase-modulated coded image based on the light passing through the pattern mask, obtain feature points of the eye from the coded image, and obtain gaze information of the user based on the feature points.

[0007] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium may store instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform the following operations: obtain a phase-modulated coded image based on light reflected by the eye, transmitted through a pattern mask and received by an image sensor, obtain feature points of the eye from the coded image, and obtain gaze information of the user based on the feature points.

[0008] According to one aspect of the present disclosure, a method may include: obtaining a phase-modulated coded image based on light reflected by the eye and transmitted through a pattern mask and received by an image sensor, obtaining feature points of the eye from the coded image, and obtaining gaze information of the user based on the feature points. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which: Figure 1 is a diagram illustrating an operation of an electronic device obtaining a coded image using a pattern mask and extracting feature points from the obtained coded image according to an embodiment of the present disclosure; Figure 2 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present disclosure; Figure 3 is a diagram showing elements of an electronic device according to an embodiment of the present disclosure; Figure 4 is a diagram showing a structure of an electronic device according to an embodiment of the present disclosure; Figure 5 is a diagram showing a structure of a light detector of an electronic device according to an embodiment of the present disclosure; Figure 6 is a diagram illustrating a pattern mask of an electronic device according to an embodiment of the present disclosure; Figure 7 is a diagram illustrating a pattern mask of an electronic device according to an embodiment of the present disclosure; Figure 8 is a diagram illustrating an operation of an electronic device obtaining a coded image using a pattern mask according to an embodiment of the present disclosure; Fig. 9 is a diagram illustrating an operation of an electronic device to recognize an object through various types of feature points extracted from a code image according to an embodiment of the present disclosure; Fig.10 is a diagram illustrating an operation of an electronic device to recognize an object through various types of feature points extracted from a code image according to an embodiment of the present disclosure; Fig.11 is a diagram illustrating an operation of an electronic device obtaining a coded image using a pattern mask and reconstructing an original image from the obtained coded image according to an embodiment of the present disclosure; Fig.12 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present disclosure; Fig.13 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present disclosure; Fig.14is a diagram showing a method in which an electronic device selectively performs an operation of extracting feature points from an obtained coded image or reconstructing an original image according to an embodiment of the present disclosure; Fig.15 is a diagram illustrating a method in which an artificial intelligence (AI) model is trained to extract feature points and optimize a pattern based on the feature points according to an embodiment of the present disclosure; Fig.16 is a diagram illustrating a method in which an AI model according to an embodiment of the present disclosure is trained to reconstruct an original image and optimize a pattern based on the original image. DETAILED DESCRIPTION

[0010] Throughout this disclosure, the expression “at least one of a, b, or c” indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0011] The terms used herein are those common terms currently widely used in the art in view of the functions in the present disclosure, but these terms may vary according to the intention of a person of ordinary skill in the art, precedents, or new technologies in the art. In addition, some of the terms used herein may be arbitrarily selected by the applicant, and in this case, these terms are defined in detail below. Therefore, the specific terms used herein should be defined based on their unique meanings and the entire context of the present disclosure.

[0012] Unless an expression used in the singular form has an apparently different meaning in the context, an expression used in the singular form may encompass expressions in the plural form. Terms including technical or scientific terms used herein may have the same meanings as those generally understood by those of ordinary skill in the art described in the present disclosure.

[0013] When a component “includes” or “comprising” an element, unless there is a specific description to the contrary, the component may also include other elements without excluding other elements. In addition, terms such as “… unit” or “… module” refer to a unit that performs at least one function or operation, and the unit may be implemented as hardware or software or as a combination of hardware and software.

[0014] The expression "configured (or set) to" used in the present disclosure may be replaced by, for example, "suitable for", "having the ability to", "designed to", "suitable for", "manufactured to", or "capable of", depending on the circumstances. The term "configured (or set) to" does not always refer only to "specially designed to" by hardware. Alternatively, in some cases, the expression "the system is configured to" may refer to the system "being able to" operate together with another device or component. For example, a "processor configured (or set) to perform A, B, and C" may be a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (such as a central processing unit (CPU) or an application processor) that can perform the corresponding operations by executing at least one software program stored in a memory.

[0015] Furthermore, in the present disclosure, it will be understood that, unless otherwise specified, when elements are “connected” or “coupled” to each other, the elements may be directly connected or coupled to each other, but may optionally be connected or coupled to each other through intervening elements therebetween.

[0016] The present disclosure will now be described more fully with reference to the accompanying drawings so that those skilled in the art can carry out the present disclosure without difficulty. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments of the present disclosure set forth herein.

[0017] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0018] Figure 1 is a diagram illustrating an operation of an electronic device obtaining a coded image using a pattern mask and extracting feature points from the obtained coded image according to an embodiment of the present disclosure.

[0019] Reference Figure 1 , the electronic device may include a light source 141 , a pattern mask 1421 , and an image sensor 1422 .

[0020] The light source 141 may output light. The light source 141 may be, for example, an infrared light source that emits infrared rays. The light source 141 may be a light source that may emit infrared rays having a wavelength of 750 nm to 1 mm. However, the type of the light source 141 is merely an example, and the present disclosure is not limited thereto. For example, the light source 141 may be one of an ultraviolet light source, an X-ray light source, and a gamma-ray light source. Typically, the light source 141 may include a light emitting diode (LED), or may include a light emitting device that includes one LED or a plurality of LEDs.

[0021] The pattern mask 1421 may be a mask on which a pattern is formed. The path of the transmitted light may be changed according to the pattern of the pattern mask 1421. For example, the light transmitted through the pattern mask 1421 may be the light output from the light source 141.

[0022] The pattern mask 1421 may have a pattern formed by scratches that cause a thickness difference. That is, for example, when the pattern mask 1421 includes a plastic material, the pattern may be formed by intentionally causing scratches on the plastic surface. However, the material of the pattern mask 1421 is merely an example, and the present disclosure is not limited thereto. Figures 7 to 9 The pattern formed on the pattern mask 1421 is described in detail.

[0023] The image sensor 1422 may obtain the coded image 20 by receiving light transmitted through the pattern mask 1421. The light transmitted through the pattern mask 1421 may be light output from the light source 141 and reflected by the object. Figure 1 As shown, the light transmitted through the pattern mask 1421 may be light reflected from the eye E. The light reflected by the eye E may reach the pattern mask 1421, and the path of the light may be changed by the pattern mask 1421 to modulate the phase.

[0024] The phase-modulated light is received by the image sensor 1422. For example, the image sensor 1422 may be a two-dimensional (2D) sensor array including a plurality of pixels arranged in a matrix, and each of the plurality of pixels may include at least one photoelectric conversion element. The image sensor 1422 may detect light using the photoelectric conversion element, and may output an image signal as an electrical signal according to the detected light. The electronic device 100 may obtain the coded image 20 by converting the light received by the image sensor 1422 into an electrical signal.

[0025] Although the object from which the light output from the light source 141 is reflected is a part of the user's face including the eyes, this is merely an example, and the present disclosure is not limited thereto. For example, the object from which the output light is reflected may be a solid object (e.g., a laptop computer, a fan, or an umbrella), and / or may be another body part of the user other than the face (e.g., a hand, a foot, or a knee).

[0026] The electronic device may obtain the coded image 20 based on the light received by the image sensor 1422. The electronic device may obtain the phase-modulated coded image 20 by receiving the light transmitted through the pattern mask 1421 through the image sensor 1422.

[0027] When the light output from the light source 141 passes through the pattern mask 1421, the data obtained by the image sensor 1422 may be coded image data. The coded image 20 may vary according to at least one of the object image 10 (i.e., the object from which the light output from the light source 141 is reflected) or the pattern shape of the pattern mask 1421. Generally, the object represented in the coded image 20 phase-modulated by the pattern mask 1421 may be difficult to recognize with the naked eye.

[0028] The object image 10 may include an object. Figure 1 The object is the user's eye E, but this is merely an example, and the present disclosure is not limited thereto. For example, the object may refer to any object to be recognized by the electronic device, such as a laptop computer, a mobile phone, or a cup of coffee.

[0029] The electronic device may obtain the feature points 30 from the encoded image 20 using the processor 150. The processor 150 may include an artificial intelligence (AI) algorithm or an AI network for obtaining the feature points 30. The AI ​​algorithm included in the processor 150 may be an algorithm trained to extract the feature points 30 from the modulated image (ie, the encoded image 20).

[0030] The processor 150 may obtain the code image 20 processed by the image sensor 1422. The processor 150 may obtain the feature points 30 corresponding to the object from the code image 20.

[0031] An object may refer to an object to be detected. For example, when gaze is to be tracked, the object may include at least one of a pupil or an eye flash. In another example, when the location of a specific object is to be identified, the object may be a laptop, a fan, an umbrella, or a pupil whose location is to be identified.

[0032] The present disclosure does not limit the type of output information of the feature point 30 .

[0033] For example, the feature points 30 may be output as an image. In this case, the data amount of the feature points 30 may be determined based on the size of the image, the number of colors, and the presence or absence of glitter and pupil.

[0034] In a specific example, the data volume of the feature point 30 may be Although the feature point 30 of the image has , but this is merely an example, and the present disclosure is not limited thereto. In addition, since the feature point 30 as an image can be represented in monochrome, the data amount of the feature point 30 can be a value multiplied by 1. In addition, the feature point 30 as an image can be stored as 2 bits to indicate whether it is an area including a flash, a pupil, or both a flash and a pupil.

[0035] In another example, the feature point 30 may be output as data including a pupil image and a flash coordinate. In this case, the data amount of the feature point 30 may be determined based on the size of the pupil image, the number of colors of the pupil image, whether the pupil exists, and the number of flashes.

[0036] In a specific example, the data volume of the feature point 30 may be Bit+ The data volume of pupil image can be bits, and the data amount of the flash coordinates can be bits. Although the pupil image has , but this is merely an example and the present disclosure is not limited thereto. In addition, since the pupil image can be represented in monochrome, the data amount of the pupil image can be a value multiplied by 1. The data amount of the pupil image can be stored as 1 bit to indicate whether it is an area including the pupil. The data amount of the flash coordinates can be calculated by multiplying the number of flashes N, two coordinate values ​​(i.e., x and y coordinates in a 2D coordinate system), and 8 bits representing each coordinate.

[0037] In another example, the feature point 30 may be output as data including pupil parameters and flash coordinates. In this case, the data amount of the feature point 30 may be determined based on the number of pupil parameters and the number of flashes.

[0038] In a specific example, the data volume of the feature point 30 may be Bit+ The amount of pupil parameter data can be bits, and the data amount of the flash coordinates can be The pupil parameter can be calculated by multiplying five parameters according to the ellipse equation and 8 bits representing each parameter. A detailed description of the pupil parameter is provided below using equation (1). The data amount of the flash coordinate can be calculated by multiplying the number of flashes N, two coordinate values ​​(i.e., x and y coordinates in a 2D coordinate system), and 8 bits representing each coordinate.

[0039] The feature point 30 may include at least one of a pupil feature point 32 or a glint feature point 31 .

[0040] The flash feature point 31 may be data that can be expressed as a point. The flash feature point 31 may be data of a position coordinate value regarding a flash of an eye.

[0041] In an embodiment of the present disclosure, the flash of the eye may refer to the reflected light detected when the light output from the light source 141 is reflected by the eye and then received by the image sensor 1422. The flash feature point 31 may include data on the position coordinate value of the flash of the eye. The flash feature point 31 may be a portion having a brightness equal to or greater than a predetermined value in the detection area of ​​the eye.

[0042] The pupil feature point 32 may be data on the shape of the pupil.

[0043] The pupil may have an elliptical shape. The elliptical shape of the pupil may be described according to equation (1).

[0044]

[0045] In equation (1), a and b are parameters related to the major axis and minor axis of the ellipse. When a>b>0, the length of the major axis is 2a, and the length of the minor axis is 2b. When b>a>0, the length of the major axis is 2b, and the length of the minor axis is 2a.

[0046] In equation (1), x 0 and 0 is a parameter related to the position of the ellipse. That is, (x 0 ,y 0 ) are the coordinates of the center of the ellipse.

[0047] In order to include information about the shape of the pupil, the pupil feature point 32 may include a value (ie, a, b, x 0 and 0 corresponding pupil parameters).

[0048] In addition, in order to represent the shape of a rotated ellipse, the pupil feature point 32 may include a numerical value of a rotation angle (eg, θ). 0 ,y 0 The pupil feature point 32 can be obtained by comparing a, b, x 0 ,y 0 The corresponding values ​​of θ are used to represent data about all types of ellipses.

[0049] That is, the electronic device according to the embodiment of the present disclosure can output the flash feature point 31 and the pupil feature point 32 according to equation (1), wherein the flash feature point 31 can be represented as a position coordinate value, and the pupil feature point 32 can be represented as data about the ellipse (i.e., related to a, b, x 0 ,y 0 and θ corresponding values). Since the output feature points are represented as minimum values, the output feature points may have a small amount of data. Since the electronic device according to an embodiment of the present disclosure extracts only feature points with a small amount of data from the encoded image 20, the calculation speed may be increased.

[0050] The electronic device according to an embodiment of the present disclosure may track the user's gaze based on the obtained feature points. The electronic device may determine the user's gaze direction. The electronic device may track the user's gaze by considering the position and shape of the feature points.

[0051] Figure 2 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present disclosure.

[0052] Reference Figure 2 In operation S210, the electronic device according to an embodiment of the present disclosure may obtain a phase-modulated coded image using light that is transmitted through the pattern mask and received by the image sensor.

[0053] The electronic device may output light using a light source. The output light may be reflected from an object and may pass through a pattern mask. The output light may be modulated when passing through the pattern mask. The electronic device may receive the light passing through the pattern mask using an image sensor. The electronic device may obtain a coded image by converting information about the brightness or intensity of the received light into an electrical signal.

[0054] The light received by the image sensor may refer to light that is output from a light source, reflected by an object, and passes through a pattern mask to change a path.

[0055] In operation S220, the electronic device according to an embodiment of the present disclosure may obtain feature points of eyes from the code image.

[0056] The processor of the electronic device may include an AI algorithm or an AI network for obtaining feature points from the encoded image. In an embodiment of the present disclosure, the AI ​​algorithm or the AI ​​network may be an AI model including a feature point extraction algorithm.

[0057] The electronic device may use a processor to obtain feature points of the eye from the encoded image. To obtain the feature points, the processor of the electronic device may use an AI model including a feature point extraction algorithm. The present disclosure does not limit the feature point output information type, and the feature points may be output as information including, for example, pupil images and flash coordinates.

[0058] In operation S230, the electronic device according to an embodiment of the present disclosure may obtain the user's gaze information based on the feature points. For example, when the object to be recognized is the user's eyes, the user's gaze information may be obtained based on the position, shape, and arrangement of the feature points of the eyes.

[0059] Figure 3 is a diagram illustrating elements of an electronic device according to an embodiment of the present disclosure.

[0060] For ease of explanation, the following will be briefly provided or omitted. Figure 1 and Figure 2 The descriptions done are the same as the descriptions.

[0061] The electronic device 100 may be a device for obtaining an image by imaging a real object using a camera including a light source 141 and a light detector 142. The electronic device 100 may be any of various devices such as a mobile device, a smart phone, a laptop computer, a desktop computer, a tablet PC, a wearable device, an e-book terminal, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, an MP3 player, or a portable camera. In an embodiment of the present disclosure, the electronic device 100 may be an augmented reality (AR) device. In addition, an "AR device" may be a device capable of representing "AR" and may include AR glasses that a user usually wears on his / her face, a head-mounted display (HMD) device mounted on the head, an AR helmet, and the like.

[0062] Reference Figure 3 , the electronic device 100 may include a light source 141, a light detector 142, a processor 150, and a memory 160. The light detector 142 may include a pattern mask 1421 and an image sensor 1422. The pattern mask 1421, the image sensor 1422, the processor 150, and the memory 160 may be electrically and / or physically connected to each other.

[0063] Figure 3 The elements shown in the figure may be only examples, and the elements included in the electronic device 100 are not limited to Figure 3 The electronic device 100 may include the following components: Figure 3 For example, the electronic device 100 may further include a power unit (eg, a battery) for providing driving power to the light source 141 , the pattern mask 1421 , the image sensor 1422 , the processor 150 , and the memory 160 .

[0064] In another example, the electronic device 100 may also include a low-resolution light detection and ranging sensor or a time-of-flight (TOF) sensor for obtaining depth value information of an object to be imaged. The electronic device 100 may receive light through the TOF sensor to obtain a coded image, and may obtain feature points based on the obtained coded image. That is, the present disclosure does not limit the type of image sensor 1422 that receives light.

[0065] The light source 141 may output light. For example, the light source 141 may emit light toward an object. The light output by the light source 141 may be reflected from the object and may pass through the pattern mask 1421.

[0066] The gaze tracking sensor or TOF sensor of the present disclosure generally uses an infrared (IR) light source. In order to track gaze using the gaze tracking sensor, light in the infrared band output from the IR light source may be reflected from the eye and may pass through the pattern mask 1421. The light reflected from the eye and passing through the pattern mask 1421 may be received by the gaze tracking sensor. In order to obtain depth information using the TOF sensor, light in the infrared band output from the infrared light source may be reflected from an object and may pass through the pattern mask 1421. The light reflected from the object and passing through the pattern mask 1421 may be received by the TOF sensor. In this case, a plurality of separate IR light sources may be used as the light sources 141 corresponding to the gaze tracking sensor and the TOF sensor.

[0067] The pattern mask 1421 may be a mask on which a pattern is formed. In the pattern mask 1421, the path of the transmitted light may be changed according to the pattern. The pattern mask 1421 may have different patterns, and a plurality of regions constituting the pattern may have various thicknesses. The transmission amount and path of the light passing through the pattern mask 1421 may vary according to the shape of the pattern including the plurality of regions having various thicknesses, and the original image may be distorted according to the transmission amount and path of the light.

[0068] Will refer to Figures 7 to 9 The pattern on the pattern mask 1421 is described in detail.

[0069] The image sensor 1422 is an imaging device, wherein the imaging device is configured to receive light transmitted through the pattern mask 1421, convert the brightness or intensity of the received light into an electrical signal, and obtain a coded image by imaging the electrical signal. The image sensor 1422 may be, for example, but not limited to, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

[0070] The processor 150 may execute one or more instructions or program codes stored in the memory 160, and may perform functions and / or operations corresponding to the instructions or program codes. The processor 150 may include hardware components for performing arithmetic, logic and input / output operations and signal processing. The processor 150 may include, for example, but not limited to, at least one of a central processing unit, a microprocessor, a graphics processing unit, an application processor (AP), an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), or a field programmable gate array (FPGA).

[0071] Although the processor 150 is Figure 3 In an embodiment of the present disclosure, one or more processors 150 may be provided.

[0072] In an embodiment of the present disclosure, the processor 150 may be a dedicated hardware chip for performing AI learning.

[0073] Instructions and program codes readable by the processor 150 may be stored in the memory 160. The memory 160 may include, for example, at least one of a flash memory type, a hard disk type, a multimedia card micro, a card type memory (e.g., an SD or XD memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a mask ROM, a flash ROM, a hard disk drive (HDD), or a solid state drive (SSD).

[0074] Instructions or program codes for executing functions or operations of the electronic device 100 may be stored in the memory 160. In an embodiment of the present disclosure, at least one of instructions, algorithms, data structures, program codes, or applications readable by the processor 150 may be stored in the memory 160. The instructions, algorithms, data structures, and program codes stored in the memory 160 may be implemented in a programming or scripting language such as C, C++, Java, or assembler.

[0075] Instructions, algorithms, data structures, or program codes related to the AI ​​model for extracting feature points of the encoded image may be stored in the memory 160. A “module” included in the memory 160 may refer to a unit for processing a function or operation performed by the processor 150 and may be implemented as software such as instructions, algorithms, data structures, or program codes.

[0076] The processor 150 may be implemented by executing instructions or program codes stored in the memory 160 .

[0077] The processor 150 may obtain a phase-modulated encoded image using light that is transmitted through the pattern mask 1421 and received by the image sensor 1422 .

[0078] In an embodiment of the present disclosure, the image sensor 1422 may include a TOF sensor for obtaining depth information by receiving light. The processor 150 may obtain a coded image using light that passes through the pattern mask 1421 and is received by the TOF sensor. The coded image obtained using the light received by the TOF sensor may include depth information corresponding to the object and may be an image modulated by the pattern mask 1421.

[0079] In an embodiment of the present disclosure, the image sensor 1422 may include a light sensor that obtains an image of a target object. The processor 150 may obtain a coded image based on light that passes through the pattern mask 1421 and is received by the light sensor.

[0080] In an embodiment of the present disclosure, the electronic device 100 may include a TOF sensor and a light sensor. The processor 150 may obtain a first coded image based on the light received by the TOF sensor. The processor 150 may obtain a second coded image based on the light received by the light sensor. The processor 150 may obtain a feature point by selectively using at least one of the first coded image or the second coded image.

[0081] The processor 150 may include a plurality of processors. For example, the processor 150 may include a first processor and a second processor. In an embodiment of the present disclosure, the first processor may obtain feature points using a first coded image, and the second processor may obtain feature points using a second coded image. In this case, the light source for obtaining the first coded image and the light source for obtaining the second coded image may be light sources having different wavelength bands. In addition, as the light source for obtaining the first coded image and the light source for obtaining the second coded image, a light source including a light emitting device may be used, wherein the light emitting device includes a plurality of LEDs emitting light in a plurality of wavelength bands.

[0082] The processor 150 may obtain feature points of the eye from the coded image. The processor may obtain the feature points using an AI algorithm trained to extract feature points from the coded image phase-modulated by a pattern.

[0083] The feature point may include information about at least one of the position coordinates or the shape of the eye. Alternatively, the feature point may include at least one of a pupil feature point or a glint feature point.

[0084] In an embodiment of the present disclosure, the processor 150 may obtain only feature points of the eyes from the encoded image. That is, the processor 150 may obtain feature points directly from the encoded image without reconstructing the original image.

[0085] The processor 150 may obtain the user's gaze information based on the feature points. That is, the processor 150 may determine the user's gaze direction by considering the pupil shape and the flash coordinates.

[0086] In an embodiment of the present disclosure, the processor 150 may reconstruct an original image of the eye from the coded image. The original image may refer to an image based on light before the light is modulated by the pattern mask. The processor 150 may reconstruct the original image using an AI algorithm trained to reconstruct the original image from the coded image. The original image may be an infrared image obtained by receiving light output from a light source, reflected by the eye, and transmitted through a pattern mask.

[0087] The processor 150 may obtain authentication information of the user based on the original image. The processor 150 may perform user authentication based on the authentication information of the user. In detail, the processor 150 may recognize iris information of the user based on the original image. The processor 150 may obtain authentication information using the recognized iris information.

[0088] In an embodiment of the present disclosure, the processor 150 may obtain process determination information about the user's situation. The processor 150 may selectively obtain the user's gaze information or authentication information based on the process determination information. That is, the electronic device according to an embodiment of the present disclosure may determine whether to track the user's gaze and perform user authentication by considering the user's situation, and may perform an operation according to the determination.

[0089] The "encoded image" in the present disclosure may be an image with a distorted focus obtained using light that is phase-modulated while passing through the pattern mask 1421. The original image distorted according to the pattern, ie, the encoded image 20, may not be recognizable with the naked eye.

[0090] According to the processor of the electronic device of an embodiment of the present disclosure, an AI model trained to extract feature points of an original image may be used. In an embodiment of the present disclosure, the AI ​​model may be a deep neural network model, which is trained through supervised learning by applying a plurality of original images obtained in advance as input data, and applying a plurality of feature points corresponding to the plurality of original images respectively as the true value of the output. The term "training" may refer to a method of training a neural network to discover or learn to analyze the input data of the neural network, a method of classifying the input data, and / or a method of extracting features required to generate result data from the input data. In detail, through the learning process, the deep neural network model can optimize the weight values ​​in the neural network by training on training data (e.g., a plurality of original images and a plurality of feature points). The deep neural network model outputs the desired result by processing the input data through a neural network with optimized weight values.

[0091] The present disclosure does not limit the type of AI model, and the AI ​​model may be implemented as any one of a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q network, etc. In addition, the AI ​​model may be subdivided. For example, the convolutional neural network model may be subdivided into a deep convolutional neural network (D-CNN) or a capsule neural network (Capsnet) (not shown).

[0092] The processor 150 may obtain feature points from the encoded image using a pre-trained AI model. In an embodiment of the present disclosure, the processor 150 may input the encoded image obtained by the image sensor 1422 into the AI ​​model, and may obtain feature points corresponding to the encoded image by performing inference using the AI ​​model.

[0093] The AI ​​model may be stored in the memory 160 of the electronic device 100. However, this is merely an example, and the present disclosure is not limited thereto. For example, the AI ​​model may be stored in an external server. In this case, the electronic device 100 may also include a communication interface that can perform data communication with the external server, and may receive the AI ​​model or inference result data (e.g., feature points) of the AI ​​model from the external server through the communication interface. Typically, compared to a server, the electronic device 100 may have limited memory storage capacity, computing processing speed, and training data set collection capabilities. Therefore, operations requiring large data storage and large amounts of computing may be performed by the server, and then the required data and / or AI model may be sent to the electronic device 100 through a communication network. In this case, because the electronic device 100 can quickly and easily perform the required operations by receiving and using the AI ​​model or inference data of the AI ​​model from the server, without the need for a large-capacity memory and a high-speed processor.

[0094] Figure 4 is a diagram illustrating a structure of an electronic device according to an embodiment of the present disclosure.

[0095] Reference Figure 4 , the user can see the real object when wearing the electronic device 100. The electronic device 100 may include a gaze tracking sensor 140. However, in Figure 4 Only necessary elements for describing the operation of the electronic device 100 are shown, and the elements included in the electronic device 100 are not limited to Figure 4 Those elements shown in .

[0096] In the present disclosure, the gaze tracking sensor 140 may refer to a sensor including Figure 3 Elements of the light detector 142 (ie, the pattern mask 1421 and the image sensor 1422 ).

[0097] Furthermore, although the electronic device 100 is AR glasses that a user wears on his / her face, examples of the electronic device 100 may include an HMD device mounted on a head, an AR helmet, and the like.

[0098] In an embodiment of the present disclosure, the electronic device 100 may detect feature points by receiving reflected light reflected by the user's eyes using the gaze tracking sensor 140, and may determine the user's gaze direction based on the feature points. In an embodiment of the present disclosure, the electronic device 100 may detect feature points by receiving reflected light reflected by the user's eyes using the gaze tracking sensor 140, and may perform user authentication based on the feature points.

[0099] Figure 5 is a diagram showing the structure of a light detector of an electronic device according to an embodiment of the present disclosure. For reference, Figure 5 By magnifying Figure 4 A portion R1 of FIG. 1 is used to show an enlarged view of the interior of the gaze tracking sensor 140 .

[0100] For ease of explanation, the following will be briefly provided or omitted. Figures 1 to 4 The descriptions done are the same as the descriptions.

[0101] Reference Figure 5 , a gaze tracking sensor 140 (illustrated as 140R, but the same applies to 140L) according to an embodiment of the present disclosure may include a light detector 142 , ie, a pattern mask 1421 and an image sensor 1422 .

[0102] In an embodiment of the present disclosure, the light detector 142 may receive light L output from the light source 141 and reflected from the object. The light L reflected from the object may pass through the pattern mask. The light L reflected from the object may be phase modulated when passing through the pattern mask. The phase modulated light L may be received by the image sensor 1422.

[0103] When the image sensor 1422 receives light, the electronic device 100 may obtain a coded image. The electronic device 100 may obtain feature points from the coded image, or may reconstruct an original image.

[0104] In addition, due to the thickness of multiple lenses, the gaze tracking camera system of the prior art using various combinations of lenses may have problems in reducing the size. Because the gaze tracking sensor 140 using the electronic device 100 according to the embodiment of the present disclosure replaces the lens with a thin pattern mask 1421, the size can be reduced (a small form factor can be achieved). For example, the camera module using the electronic device 100 of the present disclosure can be inserted into the frame of AR glasses, and the protrusion of the frame that may occur when the camera module is inserted can be minimized.

[0105] Figure 6 is a diagram illustrating a pattern mask of an electronic device according to an embodiment of the present disclosure. Figure 7 is a diagram illustrating a pattern mask of an electronic device according to an embodiment of the present disclosure. Figure 8is a diagram illustrating an operation of an electronic device obtaining a coded image using a pattern mask according to an embodiment of the present disclosure.

[0106] In the embodiments of the present disclosure, Figure 8 The pattern mask 1421 and the Figure 7 The cross section of the pattern mask 1421b corresponds to.

[0107] Reference Figure 6 and Figure 7 , pattern masks 1421a and 1421b according to an embodiment of the present disclosure are shown. Figure 6 As shown, the pattern mask 1421a may have, for example, a circular pattern. The pattern mask 1421a may have a pattern of regularly repeating shapes. That is, the pattern mask 1421 may have a concentric pattern of repeating circular shapes with different radii.

[0108] However, Figure 6 and Figure 7 The shape of the pattern of the pattern mask is only an example, and the present disclosure is not limited thereto.

[0109] For example, the pattern mask 1421a may have a quadrilateral pattern. The pattern mask 1421a may have a pattern in which a quadrilateral shape is repeated. Figure 7 As shown in , the pattern mask 1421b may have an irregular pattern.

[0110] Reference Figure 8 In an embodiment of the present disclosure, the electronic device 100 may obtain a phase-modulated coded image 20 by receiving light that passes through the pattern mask 1421 and is received by the image sensor 1422 .

[0111] The light-transmitting pattern mask 1421 may include a first region A1 having a first thickness D1 and a second region A2 having a second thickness D2. The refraction degree of light passing through the pattern mask 1421 may vary according to the thickness of the region through which the light passes. Therefore, the electronic device 100 may obtain the coded image 20.

[0112] The light source 141 may output the first light L1 and the second light L2. The first light L1 and the second light L2 may be emitted to the object. Although the object is the user's eye E, this is merely an example and the present disclosure is not limited thereto. For example, the object may be a laptop computer PC (see Fig. 9 ), hand H (see Fig.10 ), a fan or a mobile phone.

[0113] First, referring to the first light L1 , the first light L1 may reach an object, that is, an eye E. The first light L1 reaching the object may be reflected from a surface of the object. The first reflected light RL1 reflected from the surface of the object may proceed toward the pattern mask 1421 .

[0114] The first reflected light RL1 may pass through the pattern mask 1421. The first reflected light RL1 may pass through the first area A1 of the pattern mask 1421. The first area A1 may be a pattern area of ​​the pattern mask 1421 having a first thickness D1. The first reflected light RL1 may be reflected by passing through the first area A1 of the first thickness D1. The first thickness D1 may be less than a second thickness D2 described below. Therefore, the first reflected light RL1 may be less refracted and may reach the image sensor 1422. The image sensor 1422 may receive the first transmitted light TL1 passing through the first area A1.

[0115] Referring to the second light L2, the second light L2 may reach the object. The second light L2 reaching the object may be reflected from a surface of the object. The second reflected light RL2 reflected from the surface of the object may proceed toward the pattern mask 1421.

[0116] The second reflected light RL2 may pass through the pattern mask 1421. The second reflected light RL2 may pass through the second area A2 of the pattern mask 1421. The second area A2 may be a pattern area of ​​the pattern mask 1421 having a second thickness D2. The second reflected light RL2 may be refracted by passing through the second area A2 of the second thickness D2. The second thickness D2 may be greater than the first thickness D1. Therefore, the second reflected light RL2 may be more refracted and may reach the image sensor 1422. The image sensor 1422 may receive the second transmitted light TL2 passing through the second area A2.

[0117] The image sensor 1422 may obtain the phase-modulated coded image 20 by receiving light output from the light source 141 and transmitted through the pattern mask 1421. The light transmitted through the pattern mask 1421 may include reflected light output from the light source 141 and reflected from the user's eyes. The image sensor 1422 may obtain the coded image 20 by receiving the first transmitted light TL1 and the second transmitted light TL2. Since the first transmitted light TL1 and the second transmitted light TL21 have different refractive degrees by respectively transmitting through the first area A1 and the second area A2 of the pattern mask 1421, the coded image 20 may include information about an object that may not be recognizable with the naked eye.

[0118] exist Figure 8 In the embodiment, although the pattern mask 1421 has the following pattern: the first region A1 having the first thickness D1 and the second region A2 having the second thickness D2 are based on Figure 7The pattern mask 1421b is irregularly arranged, but the pattern shape of the pattern mask 1421 is only an example, and the present disclosure is not limited thereto. For example, the pattern mask 1421 may have a cross-sectional pattern in which the first area A1 and the second area A2 are regularly repeated. In an embodiment of the present disclosure, the pattern mask 1421 may have a cross-sectional pattern including a first area having a first thickness, a second area having a second thickness, and a third area having a third thickness.

[0119] Fig. 9 is a diagram illustrating an operation of an electronic device recognizing an object through various types of feature points extracted from a code image according to an embodiment of the present disclosure. Fig.10 is a diagram illustrating an operation of an electronic device recognizing an object through various types of feature points extracted from a code image according to an embodiment of the present disclosure.

[0120] For ease of explanation, the following will be briefly provided or omitted. Figures 1 to 8 The descriptions done are the same as the descriptions.

[0121] Reference Fig. 9 , the electronic device 100 may obtain an original image 10b. The original image 10b may include an object, and the object may be a target object to be recognized by the electronic device 100. Fig. 9 As shown, the original image 10b may include a laptop computer PC.

[0122] Despite Fig. 9 In the example, the object is a laptop PC, but this is only an example and the present disclosure is not limited thereto. For example, the object may be any object to be recognized by the electronic device 100 (such as a laptop computer, a mobile phone, or a cup of coffee). In another example, Fig.10 As shown, the object may be a user's hand H (see Fig.10 ).

[0123] The electronic device 100 may obtain the coded image 20b. The coded image 20b may be an image that is coded while passing through the pattern mask 1421. The electronic device 100 may obtain the coded image 20b by receiving light output from the light source 141, reflected by the object (ie, the laptop PC) and passing through the pattern mask via the image sensor 1422.

[0124] The electronic device 100 may obtain a feature point 30b. The feature point 30b may correspond to an object in the original image 10b to be recognized by the electronic device 100. The feature point 30b may be as shown in FIG. Figure 1 The description may be expressed as data on point coordinates, or as data on parameters indicating shape and form.

[0125] However, the method of expressing the feature point is merely an example, and the present disclosure is not limited thereto. For example, although the feature point 30b includes a position coordinate value as data, the feature point may be expressed as a point or text.

[0126] In another example, the shape of the laptop computer may include a quadrilateral shape. Therefore, in order to represent the shape of the laptop computer, the feature point 30b may include a numerical value corresponding to a parameter related to the quadrilateral shape. In order to correspond a laptop computer having a rhombus shape to the feature point, the feature point 30b may include at least one of the coordinates of the center of the rhombus shape, the length of the long side, the length of the short side, or a small internal angle among the internal angles. In this case, the feature point 30b corresponding to the laptop computer may be a coordinate corresponding to the reference point 30b. Figure 1 The pupil feature points 32 are described in the same manner as the data stored.

[0127] In an embodiment of the present disclosure, the image sensor 1422 may include a TOF sensor. The electronic device may use the light received by the TOF sensor to obtain a coded image. The coded image obtained using the light received by the TOF sensor may include depth information corresponding to the object and may be an image modulated by the pattern mask 1421.

[0128] In an embodiment of the present disclosure, the processor 150 may obtain a depth map including depth information of an object based on a coded image obtained using a TOF sensor. The processor may obtain a depth map using an AI algorithm trained to extract a depth map from a coded image phase-modulated by a pattern.

[0129] In an embodiment of the present disclosure, the image sensor may include a TOF sensor for obtaining depth information and a light sensor for obtaining an image of a target object. The processor 150 may select one of a first coded image based on light received by the light sensor and a second coded image based on light received by the TOF sensor.

[0130] The processor 150 may obtain feature points or depth information by selectively using at least one of the first encoded image or the second encoded image. In addition, the processor 150 may obtain an original image or depth information by selectively using at least one of the first encoded image or the second encoded image.

[0131] When the processor includes a plurality of processors, the first processor may use the first encoded image to obtain the original image, and the second processor may use the second encoded image to obtain the depth information. Alternatively, the first processor may use the first encoded image to obtain the feature point, and the second processor may use the second encoded image to obtain the depth information. Fig.11 The operation of the processor 150 to obtain the original image is described.

[0132] For ease of explanation, the following will be briefly provided or omitted. Fig. 9 The same description is done as described in the description. Fig.10 , the electronic device may obtain an original image 10c. The original image 10c may include an object, and the object may be a hand H of a user.

[0133] The electronic device may obtain the coded image 20 c. The electronic device may obtain the coded image 20 c by receiving light whose path is changed when passing through the pattern mask 1421 .

[0134] The electronic device may obtain one or more feature points 30c. The one or more feature points 30c may be data corresponding to an object to be recognized by the electronic device. Fig.10 As shown, one or more feature points 30c corresponding to the object (ie, the user's hand H) may be represented as points, and thus, may be position coordinate values. The one or more feature points 30c corresponding to the user's hand may be in the form of a pixel corresponding to the reference pixel. Figure 1 The data is stored in the same manner as described for the flash feature point 31 .

[0135] In an embodiment of the present disclosure, in the case of a user's hand, one or more feature points 30c may correspond to one or more joints included in the hand. In an embodiment of the present disclosure, the electronic device may obtain a joint value based on the encoded image 20c, wherein the joint value is a three-dimensional (3D) position coordinate value, wherein the 3D position coordinate value is position information about the x-axis, y-axis, and z-axis of one or more joints included in the hand according to the recognition result of the hand. One or more feature points 30c may be data including each joint value.

[0136] For reference, the 3D gesture of the hand can be recognized by the "hand skeleton detection and tracking" technology. The hand skeleton detection and tracking technology is a technology that detects the moving joints in the image of a person's hand and divides and projects a predetermined skeleton structure. The skeleton structure of a person's hand may include but is not limited to the ends of the five fingers (i.e., the ends of the fingers ( ) joint points), a palm point and a wrist point.

[0137] In an embodiment of the present disclosure, one or more feature points 30c may be data including joint values ​​corresponding to the skeleton structure of a person's hand identified using hand skeleton detection and tracking technology. However, the method of representing feature points is merely an example, and the present disclosure is not limited thereto. For example, a feature point 30c corresponding to a user's hand may not be represented as one or more points, but may be represented as a point representing the position of a person's hand. In detail, the feature point 30c may be a position coordinate value of a point that is the center of the user's hand.

[0138] In another example, the shape of the user's hand may be represented as an elliptical shape including the outline of the hand. In this case, in order to represent the user's hand, the feature point 30c may include a numerical value corresponding to a parameter related to the elliptical shape. The feature point 30c stored as the parameter related to the elliptical shape may be a value corresponding to the reference numeral 30c. Figure 1 The data describing the pupil feature points 32 are stored in the same manner.

[0139] In an embodiment of the present disclosure, the image sensor 1422 for obtaining the feature point 30c including the joint value corresponding to the skeleton structure of the hand may include a TOF sensor. In an embodiment of the present disclosure, the processor 150 may obtain the depth information of the target object based on the encoded image obtained using the TOF sensor. This has been referred to in Fig. 9 The target object may be a hand, and the processor 150 may obtain depth information of a portion of the hand including fingers, a palm, and a back of the hand, wherein the portion includes joints corresponding to a skeleton structure of the hand. The processor 150 may recognize a 3D posture or movement of the hand by obtaining 3D position information of a feature point 30c of each joint of the finger based on the obtained depth information.

[0140] Fig.11 is a diagram illustrating an operation of an electronic device obtaining a coded image using a pattern mask and reconstructing an original image from the obtained coded image according to an embodiment of the present disclosure.

[0141] For ease of explanation, the following will be briefly provided or omitted. Figure 1 The descriptions done are the same as the descriptions.

[0142] In an embodiment of the present disclosure, the electronic device may obtain a phase-modulated coded image 20 by receiving light transmitted through the pattern mask 1421 via the image sensor 1422. The coded image 20 may be an image modulated when light reflected from an object included in the object image 10 passes through the pattern mask 1421.

[0143] The electronic device may reconstruct the original image 400 from the encoded image 20 using the processor 150. The processor 150 may include an AI algorithm or an AI network for reconstructing the original image 400. The AI ​​algorithm included in the processor 150 may be an algorithm trained to reconstruct the original image 400 from the modulated image (ie, the encoded image 20).

[0144] The processor 150 may obtain the coded image 20 processed by the image sensor 1422. The processor 150 may obtain the original image 400 reconstructed from the coded image 20. The processor 150 may obtain authentication information of the user based on the reconstructed original image 400. The processor 150 may perform user authentication.

[0145] Fig.12 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present disclosure.

[0146] In operation S1310, the electronic device may obtain a phase-modulated coded image using light transmitted through the pattern mask and received by the image sensor. A description of operation S1310 may be the same as that of operation S210, and thus will be omitted.

[0147] In operation S1320, the electronic device may reconstruct an original image of the eye from the encoded image.

[0148] The electronic device may receive light transmitted through the pattern mask using an image sensor. The received light may be light reflected from an object (eg, an eye) included in the object image. The received light may be modulated while passing through the pattern mask. The image sensor may use the received light to obtain a coded image.

[0149] The electronic device may reconstruct an original image of an object (eg, an eye) from the encoded image using a processor. The processor may reconstruct the original image from the encoded image using an AI algorithm for reconstructing the original image.

[0150] In operation S1330, the electronic device may obtain authentication information of the user based on the original image. The original image may include an image of an iris in the user's eye.

[0151] Further references Fig.13 In operation S1410, the electronic device may recognize iris information of a user based on an original image. The iris information may refer to information for distinguishing the iris of a user from the iris of another user.

[0152] In an embodiment of the present disclosure, the electronic device may extract an iris area from an original image. The electronic device may identify the user's iris information by encrypting an image of the iris area. The electronic device may store the identified user's iris information in a memory. The memory of the electronic device may store multiple pieces of iris information of various users.

[0153] In an embodiment of the present disclosure, the electronic device may remove objects other than the iris from the extracted iris region. For example, the electronic device may remove the eyelid detected in the iris region from the original image. Therefore, the electronic device may recognize a clearer iris region. The electronic device may recognize the user's iris information based on the extracted iris region.

[0154] In operation S1420, the electronic device may obtain authentication information using the identified iris information. The authentication information may include information about the result of identifying the user using the iris information. For example, although the iris information may include information about an iris pattern unique to each person, the present disclosure is not limited to the method of iris recognition. The authentication information may include information about whether the identified user is a user who can access the electronic device.

[0155] In an embodiment of the present disclosure, the electronic device may obtain multiple pieces of iris information of various users stored in a memory. The electronic device may compare the iris information of various users stored in the memory with the iris information of the obtained user. Through the comparison, the electronic device may determine whether the identified user is a user who can access the electronic device.

[0156] For example, the electronic device may store iris information of users who can access the electronic device in a memory. When determining that the iris information of users who can access the electronic device stored in the memory is the same as the iris information of the newly identified user, the electronic device may determine that the newly identified user is the user who can access the electronic device.

[0157] In another example, when it is determined that iris information of a user who can access the electronic device stored in the memory is different from iris information of a newly identified user, the electronic device may determine that the newly identified user is a user who cannot access the electronic device.

[0158] Fig.14 is a diagram illustrating a method in which an electronic device selectively performs an operation of extracting feature points from an obtained encoded image or reconstructing an original image according to an embodiment of the present disclosure.

[0159] For ease of explanation, the following will be briefly provided or omitted. Figure 1 and Fig.11 For reference, the description of the operation of the processor 150 extracting the feature points 30 using the encoded image 20 is the same as that of the description of the operation of the processor 150 using the encoded image 20. Figure 1 The description of the operations performed by the processor 150 is the same as that of the operation of reconstructing the original image 400 using the encoded image 200. Fig.12 The operations performed are the same as described above.

[0160] Reference Fig.14 , the processor 150 may extract the feature points 30 using the encoded image 20 , or may reconstruct the original image 400 .

[0161] In an embodiment of the present disclosure, the processor 150 may obtain process determination information based on the user's situation. The process determination information may be obtained according to the user's situation.

[0162] For example, in the case where the user wears the electronic device, the electronic device needs to perform user authentication. In this case, the electronic device can obtain process determination information for performing user authentication.

[0163] In another example, when the user continues to wear the electronic device, the electronic device needs to track the user's gaze. In this case, the electronic device can obtain process determination information indicating that gaze tracking should be performed.

[0164] In an embodiment of the present disclosure, the processor 150 may selectively obtain gaze information or authentication information of the user based on the process determination information.

[0165] In an embodiment of the present disclosure, when it is determined that gaze tracking should be performed according to the process information, the processor 150 may choose to obtain the user's gaze information. In this case, the processor 150 may obtain the feature point 30 using the encoded image 20. The processor 150 may track the user's gaze based on the feature point 30.

[0166] In an embodiment of the present disclosure, when user authentication should be performed according to the process determination information, the processor 150 may choose to obtain authentication information. In this case, the processor 150 may reconstruct the original image 400 using the encoded image 20. The processor 150 may perform user authentication based on the original image 400.

[0167] In an embodiment of the present disclosure, when the processor 150 includes a plurality of processors, the first processor may be configured to obtain feature points using the encoded image, and the second processor may be configured to reconstruct the original image.

[0168] Fig.15 is a diagram illustrating a method in which an AI model according to an embodiment of the present disclosure is trained to extract feature points and optimize a pattern based on the feature points.

[0169] As a reference, Figures 1 to 14 different, Fig.15 The light source 1610 , pattern mask 1620 , and image sensor 1630 in the example may be virtual elements that perform their functions to train the AI ​​model 180 .

[0170] Reference Fig.15 , a phase-modulated encoded image 20 can be obtained by receiving light passing through the pattern mask 1620 via the image sensor 1630.

[0171] The AI ​​model 180 may be a model trained by performing, through virtual simulation, a process in which light output from the light source 1610 is reflected from an object, passes through the pattern mask 1620 , and is received by the image sensor 1630 .

[0172] The virtual simulation performed by the AI ​​model 180 may use, for example, ray tracing. That is, the AI ​​model 180 may receive virtual light via the image sensor 1630 using a method of tracing the path of light to obtain the encoded image 20. However, the method of simulating the path of light is merely an example, and the present disclosure is not limited thereto.

[0173] The coded image 20 may be obtained based on the light received by the image sensor 1630 using virtual simulation. The phase-modulated coded image 20 may be obtained by receiving light transmitted through the pattern mask 1620 via the image sensor 1630. The process of obtaining the coded image 20 is similar to the process of obtaining the coded image 20. Figure 8 The process described is the same and will therefore be described briefly.

[0174] In an embodiment of the present disclosure, the AI ​​model 180 may be a model trained by executing a feature point extraction algorithm 181 and a pattern optimization algorithm 182 .

[0175] The feature point extraction algorithm 181 may be an algorithm trained to extract feature points from a modulated image. The AI ​​model 180 may extract feature points 30 from the encoded image 20 using the feature point extraction algorithm 181 .

[0176] For example, the feature point extraction algorithm 181 may be used to obtain the difference between the extracted feature point 30 and the true value based on the object image 10. The AI ​​model 180 may be trained to extract more accurate feature points from the encoded image 20 based on the difference between the feature point 30 and the true value through the feature point extraction algorithm 181. Through the feature point extraction algorithm 181, a loss function related to the difference between the feature point 30 and the true value may be obtained, and the result value of the loss function may be minimized. The AI ​​model 180 may be trained according to the minimized loss function.

[0177] The pattern optimization algorithm 182 may be an algorithm trained to extract the optimized pattern 122 of the pattern mask optimized for extracting feature points.

[0178] For example, the difference between the extracted feature points 30 and the true value based on the object image 10 can be obtained by the pattern optimization algorithm 182. The pattern of the pattern mask 1620 can be updated based on the difference between the feature points 30 and the truth by the pattern optimization algorithm 182. Therefore, the AI ​​model 180 can optimize the pattern of the pattern mask 1620 by the pattern optimization algorithm 182 to obtain accurate feature points from the encoded image 20.

[0179] The electronic device 100 may be designed to include an element of a gaze tracking camera system including a pattern mask having an optimized pattern 122. The device including the actually designed gaze tracking camera system may be, but is not limited to, an AR device or an HMD.

[0180] The feature point extraction algorithm 181 and the pattern optimization algorithm 182 executed based on the updated pattern mask or the pattern mask having the optimized pattern 122 may be paired with each other. The AI ​​model 180 may be trained by executing the feature point extraction algorithm 181 and simultaneously executing the pattern optimization algorithm 182 trained to extract an optimized pattern of a pattern mask optimized for extracting feature points.

[0181] Therefore, for the optimized pattern 122 obtained by the pattern optimization algorithm 182, the feature point extraction algorithm 181 paired with the pattern optimization algorithm 182 may be an algorithm trained to extract accurate feature points from a coded image modulated by a pattern mask having the optimized pattern 122. On the contrary, for the optimized pattern obtained by the pattern optimization algorithm 182, a feature point extraction algorithm not paired with the pattern optimization algorithm 182 may not extract accurate feature points from a coded image modulated by a pattern mask having the optimized pattern 122.

[0182] Fig.16 is a diagram illustrating a method in which an AI model according to an embodiment of the present disclosure is trained to reconstruct an original image and optimize a pattern based on the original image.

[0183] As a reference, Figures 1 to 14 different, Fig.16 The light source 1710 , pattern mask 1720 , and image sensor 1730 in the example may be virtual elements that perform their functions to train the AI ​​model 180 .

[0184] Reference Fig.16 According to an embodiment of the present disclosure, a phase-modulated coded image 20 may be obtained by receiving light transmitted through the pattern mask 1720 via the image sensor 1730 .

[0185] The AI ​​model 180 may be a model trained by performing, through virtual simulation, a process in which light output from the light source 1710 is reflected from an object, passes through the pattern mask 1720 , and is received by the image sensor 1730 .

[0186] The virtual simulation performed by the AI ​​model 180 may use, for example, ray tracing. That is, the AI ​​model 180 may receive virtual light via the image sensor 1730 using a method of tracing the path of light to obtain the encoded image 20. However, the method of simulating the path of light is merely an example, and the present disclosure is not limited thereto.

[0187] Virtual simulation can be used to obtain the coded image 20 using the light received by the image sensor 1730. The process of obtaining the coded image 20 is similar to the process of obtaining the coded image 20. Figure 8 The process described is the same and will therefore be described briefly.

[0188] In an embodiment of the present disclosure, the AI ​​model 180 may be a model trained by executing the original reconstruction algorithm 183 and the pattern optimization algorithm 184 .

[0189] The original reconstruction algorithm 183 may be an algorithm trained to reconstruct the original image from the modulated image. The original reconstruction algorithm 183 may include a program code for reconstructing the original image from the modulated image. The original reconstruction algorithm 183 may be a program code for reconstructing the original image from the coded image phase modulated by the pattern mask 1720. Therefore, the electronic device may reconstruct the original image 400 from the coded image 20 using the original reconstruction algorithm 183.

[0190] For example, the difference between the reconstructed original image 400 and the true value based on the input object image 10 can be obtained through the original reconstruction algorithm 183. The AI ​​model 180 can be trained to reconstruct a more accurate original image from the encoded image 20 based on the difference between the reconstructed original image 400 and the true value through the original reconstruction algorithm 183. Through the original reconstruction algorithm 183, a loss function related to the difference between the reconstructed original image 400 and the true value can be obtained, and the result value of the loss function can be minimized. The AI ​​model 180 can be trained according to the minimized loss function.

[0191] The pattern optimization algorithm 184 may be an algorithm trained to extract the optimized pattern 123 of the pattern mask optimized for reconstructing the original image.

[0192] For example, the difference between the reconstructed original image 400 and the true value based on the input object image 10 can be obtained through the pattern optimization algorithm 184. The pattern of the pattern mask 1720 can be updated based on the difference between the original image and the true value through the pattern optimization algorithm 184. Therefore, the AI ​​model 180 can optimize the pattern of the pattern mask 1720 through the pattern optimization algorithm 184 to obtain an accurate original image from the encoded image 20.

[0193] The electronic device 100 may actually be designed to include a gaze tracking camera system including a pattern mask having the optimized pattern 122. The actually designed device including the gaze tracking camera system may be, but is not limited to, an AR device or an HMD.

[0194] The original reconstruction algorithm 183 and the pattern optimization algorithm 184 executed based on the updated pattern mask or the pattern mask with the optimized pattern 123 may be paired with each other. The AI ​​model 180 may be trained by executing the original reconstruction algorithm 183 and simultaneously executing the pattern optimization algorithm 184 trained to extract an optimized pattern of the pattern mask optimized for reconstructing the original image.

[0195] For example, the pattern optimization algorithm 184 may optimize the pattern based on the difference between the original image 400 obtained by the original reconstruction algorithm 183 and the true value. Since the pattern optimization algorithm 184 is trained using the original image 400 obtained by the original reconstruction algorithm 183 as data, the pattern optimization algorithm 184 may be paired with the original reconstruction algorithm 183.

[0196] Therefore, for the optimized pattern 123 obtained by the pattern optimization algorithm 184, the original reconstruction algorithm 183 paired with the pattern optimization algorithm 184 may be an algorithm trained to reconstruct an accurate original image from a coded image modulated by a pattern mask having the optimized pattern 122. On the contrary, for the optimized pattern 123 obtained by the pattern optimization algorithm 184, an original reconstruction algorithm not paired with the pattern optimization algorithm 184 may not be able to reconstruct an accurate original image from a coded image modulated by a pattern mask having the optimized pattern 122.

[0197] An electronic device according to an embodiment of the present disclosure may include a light source, a pattern mask, an image sensor, and at least one processor. The light source may be configured to output light. The pattern mask may include a pattern through which light passes to change the path of the light. The image sensor may be configured to receive light output from the light source, reflected by the eye, and passed through the pattern mask. The at least one processor may be configured to obtain a phase-modulated coded image using light that passes through the pattern mask and is received by the image sensor. The at least one processor may be configured to obtain feature points of the eye from the coded image. The at least one processor may be configured to obtain gaze information of the user based on the feature points.

[0198] In an embodiment of the present disclosure, the feature point may include information on at least one of position coordinates or a shape of the eye.

[0199] In an embodiment of the present disclosure, the feature point may include at least one of a pupil feature point or a glint feature point.

[0200] In an embodiment of the present invention, the at least one processor may be further configured to reconstruct an original image of the eye from the encoded image.The at least one processor may be further configured to obtain authentication information of the user based on the original image.

[0201] In an embodiment of the present disclosure, the at least one processor may be further configured to recognize iris information of the user based on the original image. The at least one processor may be further configured to obtain authentication information using the recognized iris information.

[0202] In an embodiment of the present disclosure, the at least one processor may be further configured to obtain process determination information based on the user's situation. The at least one processor may be further configured to selectively obtain the user's gaze information or authentication information based on the process determination information.

[0203] In an embodiment of the present disclosure, the image sensor may include a TOF sensor configured to obtain depth information by receiving light. The at least one processor may also be configured to obtain a coded image using light that passes through the pattern mask and is received by the TOF sensor.

[0204] In an embodiment of the present disclosure, a coded image obtained using light received through a TOF sensor may include depth information corresponding to an object, and may be an image modulated by a pattern mask.

[0205] In an embodiment of the present disclosure, the image sensor may include a TOF sensor configured to obtain depth information and a light sensor configured to obtain an image of a target object. The at least one processor may also be configured to obtain a first coded image based on light transmitted through a pattern mask and received by the TOF sensor. The at least one processor may also be configured to obtain a second coded image based on light transmitted through a pattern mask and received by the light sensor. The at least one processor may also be configured to obtain feature points using at least one of the first coded image or the second coded image.

[0206] In an embodiment of the present disclosure, the at least one processor may also be configured to obtain only feature points of the eye from the encoded image. That is, the at least one processor may directly obtain feature points from the encoded image without reconstructing the original image.

[0207] The electronic device according to an embodiment of the present disclosure may further include: an AI model trained to extract feature points from an image phase-modulated by a pattern. The at least one processor of the electronic device may also be configured to obtain feature points from the encoded image using the trained AI model.

[0208] The method according to an embodiment of the present disclosure may include obtaining a phase-modulated coded image using light transmitted through a pattern mask and received by an image sensor. The method may include obtaining feature points of an eye from the coded image. The method may include obtaining gaze information of a user based on the feature points.

[0209] In an embodiment of the present disclosure, the feature point may include information on at least one of position coordinates or a shape of the eye.

[0210] In an embodiment of the present disclosure, the feature point may include at least one of a pupil feature point or a glint feature point.

[0211] In an embodiment of the present disclosure, the method may further include: reconstructing an original image of the eye from the encoded image. The method may further include obtaining authentication information of the user based on the original image.

[0212] In an embodiment of the present disclosure, obtaining the authentication information of the user may include: recognizing the iris information of the user based on the original image. Obtaining the authentication information of the user may include: obtaining the authentication information using the recognized iris information.

[0213] In an embodiment of the present disclosure, obtaining the user's gaze information and obtaining the user's authentication information may include: obtaining process determination information based on the user's situation. Obtaining the user's gaze information and obtaining the user's authentication information may include: selectively obtaining the user's gaze information or authentication information based on the process determination information.

[0214] In an embodiment of the present disclosure, the image sensor may include a TOF sensor configured to obtain depth information by receiving light. Obtaining the coded image may include: obtaining the coded image using light that is transmitted through the pattern mask and received by the TOF sensor.

[0215] In an embodiment of the present disclosure, a coded image obtained using light received through a TOF sensor may include depth information corresponding to an object, and may be an image modulated by a pattern mask.

[0216] The machine-readable storage medium may be provided as a non-transitory storage medium. "Non-transitory" may indicate that the storage medium does not include a signal and is tangible, but does not distinguish whether data is stored semi-permanently or temporarily in the storage medium. For example, a "non-transitory storage medium" may include a buffer that temporarily stores data.

[0217] According to an embodiment of the present disclosure, the method according to various embodiments of the present disclosure may be provided in a computer program product. A computer program product is a product that can be bought and sold between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed through an application store (e.g., downloaded or uploaded), or distributed directly or online between two user devices (e.g., smart phones). When distributed online, at least a portion of the computer program product (e.g., a downloadable application) may be temporarily generated or at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, a memory of an application store's server, or a memory of a forwarding server).

[0218] The embodiments of the present disclosure disclosed in the specification and the drawings are only provided as examples to easily describe the technical content according to the embodiments of the present disclosure and to help understand the embodiments of the present disclosure, and are not intended to limit the scope of the embodiments of the present disclosure. Therefore, in addition to the embodiments disclosed herein, the scope of the various embodiments of the present disclosure should be interpreted as covering all modifications or changes derived from the technical spirit of the various embodiments of the present disclosure.

Claims

1. An electronic device, comprising: a light source (141) configured to output light; A pattern mask (1421) configured to change a path of light passing through a pattern of the pattern mask; An image sensor (1422) configured to receive light output from the light source (141), reflected by the eye, and transmitted through the pattern mask (1421); as well as At least one processor (150) is configured to: Based on the light transmitted through the pattern mask (1421) and received by the image sensor (1422), a phase-modulated coded image (20) is obtained, obtaining feature points of the eye from the encoded image (30), and The user's gaze information is obtained based on the feature points (30).

2. The electronic device (100) according to claim 1, wherein: The feature point (30) includes information on at least one of the position coordinates of the eye and the shape of the eye.

3. The electronic device according to claim 1, wherein: The feature points (30) include at least one of a pupil feature point (32) and a flash feature point (31).

4. The electronic device according to any one of claims 1 to 3, wherein: The at least one processor (150) is further configured to: reconstructing an original image (400) of the eye from the encoded image (20), and Authentication information of the user is obtained based on the original image (400).

5. The electronic device according to claim 4, wherein: The at least one processor (150) is further configured to: Recognizing the user's iris information based on the original image (400) (S1410); Authentication information is obtained based on the recognized iris information (S1420).

6. The electronic device according to any one of claims 4 and 5, wherein: The at least one processor (150) is further configured to: obtaining process determination information based on the user's situation, and Based on the process determination information, the gaze information of the user or the authentication information of the user is selectively obtained.

7. The electronic device according to any one of claims 1 to 6, wherein: The image sensor (1422) includes a time-of-flight (TOF) sensor configured to obtain depth information based on received light, and Wherein, the at least one processor (150) is further configured to obtain the coded image (20) based on the light transmitted through the pattern mask (1421) and received by the TOF sensor.

8. The electronic device according to claim 7, wherein: The coded image (20) obtained based on the light received by the TOF sensor includes depth information corresponding to the eye, and The encoded image includes an image modulated by the pattern mask (1421).

9. The electronic device according to any one of claims 1 to 6, wherein: The image sensor (1422) includes a time-of-flight (TOF) sensor configured to obtain depth information and a light sensor configured to obtain an image of a target object, and Wherein, the at least one processor (150) is further configured to: Based on the light transmitted through the pattern mask (1421) and received by the TOF sensor, a first coded image is obtained, obtaining a second coded image based on the light transmitted through the pattern mask (1421) and received by the light sensor, and The feature points are obtained based on at least one of the first encoded image and the second encoded image.

10. The electronic device according to any one of claims 1 to 9, further comprising: An artificial intelligence AI model (180) is configured to extract feature points from an image phase-modulated by the pattern through the pattern mask, and The at least one processor (150) is further configured to obtain the feature points (30) from the encoded image (20) using the AI ​​model (180).

11. A method comprising: Obtaining a phase-modulated coded image (20) based on light reflected by the eye and transmitted through the pattern mask (1421) and received by the image sensor (1422); Obtaining feature points of the eye from the encoded image (30); and The user's gaze information is obtained based on the feature points (30).

12. The method according to claim 11, wherein: The feature points (30) include at least one of a pupil feature point (32) and a flash feature point (31).

13. The method according to any one of claims 11 to 12, further comprising: reconstructing an original image (400) of the eye from the encoded image (20); as well as Authentication information of the user is obtained based on the original image (400).

14. The method according to any one of claims 11 to 13, wherein: The image sensor (1422) includes a time-of-flight (TOF) sensor configured to obtain depth information by receiving light, and Wherein, obtaining the coded image (20) comprises: obtaining the coded image (20) based on light that passes through the pattern mask (1421) and is received by the TOF sensor.

15. A non-transitory computer-readable recording medium, wherein: The non-transitory computer-readable recording medium has recorded thereon a program for implementing the method according to claim 11 .

Images