Electronic device and method for gaze tracking

Abstract

Provided are an electronic device and a method for gaze tracking. The electronic device according to an embodiment disclosed herein generates a plurality of beams of light by means of a polarizing filter and a polarization changing element or through a plurality of light sources, and obtains an encoded image by means of a metalens that refracts the plurality of generated beams of light into different paths. The electronic device can acquire gaze information about a user on the basis of the encoded image and track the gaze of the user.

Description

Electronic devices and methods for eye tracking

[0001] The present disclosure relates to an electronic device and method for eye tracking. Specifically, the present disclosure relates to an electronic device and its operating method for tracking eye gaze based on a distorted eye image generated by passing incident light through a multiplexed metalens, which refracts light into different paths depending on its optical properties.

[0002] Augmented Reality is a technology that overlays virtual images onto the physical environment or real world objects of the real world. Augmented reality devices (e.g., smart glasses) that utilize AR technology are being used in everyday life for purposes such as information retrieval, route guidance, and camera shooting.

[0003] In particular, with the advancement of image sensors and AI-based image processing technology, the uses of cameras have expanded beyond simple photography to include object recognition, biometrics, and information acquisition. While camera pixel resolution continues to increase for various applications, physical limitations exist in reducing the size and cost of camera modules.

[0004] Recently, lensless imaging, a technique that replaces lenses with thin phase masks, has been attracting attention. A phase mask is an extremely thin optical element that can modulate the phase of incident light. Lensless imaging uses a thin phase mask, which modulates light, instead of a lens to capture images of objects. This technique allows for reduced lens module thickness and focal length, enabling the creation of ultra-thin cameras that overcome the physical limitations of conventional cameras.

[0005] In lensless imaging, the most crucial factor determining camera performance is the structure of the phase mask. Each phase mask can be designed with a different, finely undulated structure. Metalenses, a type of phase mask, can achieve this finely undulated structure by arranging relatively small nanostructures in a two-dimensional manner. In particular, multiplexed metalenses can be utilized to provide a variety of focal lengths or process light of various wavelengths within a single lens, contributing to the miniaturization and durability of camera modules.

[0006] In order to solve the above-described technical problem, an electronic device according to an embodiment of the present disclosure may include a polarization filter, a polarization conversion element, a metalens, an image sensor, at least one processor, and a memory. The polarization filter may be configured to receive incident light and change the polarization of the incident light. The polarization conversion element may be configured to receive light from the polarization filter. The metalens may be configured to refract the light received from the polarization conversion element into different paths depending on the polarization direction of the light received from the polarization conversion element. The image sensor may be configured to receive the light refracted by the metalens. The electronic device may be operated by having at least one processor execute at least one instruction stored in the memory. The electronic device may cause the polarization filter to change the incident light into a first polarization in a first direction and transmit the first polarization. The electronic device may obtain, through the image sensor, a first encoded image based on the first polarization refracted into the first path by the metalens. The electronic device can obtain the user's gaze information from the first encoded image, and based on whether the electronic device can obtain the user's gaze information from the first encoded image, cause the polarization change element to change the first polarization into the second polarization in the second direction and transmit the second polarization, obtain the second encoded image based on the second polarization refracted in the second path by the metalens through the image sensor, and obtain the user's gaze information from the second encoded image.

[0007] In order to solve the above-described technical problem, an electronic device according to an embodiment of the present disclosure may include a first light source, a second light source, a metalens, an image sensor, at least one processor, and a memory. The first light source may output a first light. The second light source may output a second light. The metalens may be configured to refract the incident light into different paths depending on the wavelength of the incident light. The image sensor may be configured to receive the light refracted by the metalens. The electronic device may operate by having at least one processor execute at least one instruction stored in the memory. The electronic device may obtain a first encoded image based on the first light refracted into a first path by the metalens through the image sensor. The electronic device may obtain the user's gaze information from the second encoded image obtained through the image sensor based on the second light refracted into a second path by the metalens, based on whether or not the user's gaze information can be obtained from the first encoded image.

[0008] A method according to one embodiment of the present disclosure may include a step of obtaining a first coded image by receiving light of a first polarization refracted into a first path by a metalens configured to refract incident light into different paths according to a polarization direction. The method may include a step of determining whether or not information about a user's gaze can be obtained from the first coded image. The method may include a step of obtaining a second coded image by receiving light of a second polarization refracted into a second path by the metalens based on whether or not information about the user's gaze can be obtained from the first coded image, and a step of obtaining information about the user's gaze from the second coded image.

[0009] The present disclosure can be readily understood by the following detailed description and its accompanying drawings, wherein reference numerals refer to structural elements.

[0010] FIG. 1 is a conceptual diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to track a gaze based on an encoded image obtained by receiving light transmitted through a multiplexed metalens.

[0011] FIG. 2 is a conceptual diagram illustrating a process of tracking gaze based on an encoded image by an electronic device according to an embodiment of the present disclosure.

[0012] FIG. 3 is a flowchart illustrating an operation method of an electronic device according to an embodiment of the present disclosure.

[0013] FIG. 4 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present disclosure.

[0014] FIG. 5 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present disclosure.

[0015] FIG. 6 is a conceptual diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to track a gaze based on an encoded image obtained by receiving light generated using a light source and a multiplexed metalens.

[0016] FIG. 7 is a conceptual diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to track a gaze based on an encoded image obtained by receiving at least one of a plurality of lights generated using a plurality of light sources and a multiplexed metalens.

[0017] FIG. 8 is a flowchart illustrating a method for an electronic device according to an embodiment of the present disclosure to track a gaze based on multiple lights generated using multiple light sources and multiplexed metalens.

[0018] FIG. 9 is a conceptual diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to track gaze based on an encoded image obtained by receiving light output by a plurality of light sources.

[0019] Figure 10a is a conceptual diagram for explaining an operation in which a first light of a first wavelength output by a light source forms an image on an image sensor.

[0020] Figure 10b is a conceptual diagram for explaining an operation in which a second light of a second wavelength output by a light source forms an image on an image sensor.

[0021] FIG. 11 is a flowchart illustrating a method for tracking gaze by extracting feature points based on an encoded image by an electronic device according to an embodiment of the present disclosure.

[0022] FIG. 12 is a flowchart illustrating a method for tracking gaze by restoring an image based on an encoded image by an electronic device according to an embodiment of the present disclosure.

[0023] FIG. 13 is a block diagram illustrating components of an electronic device according to an embodiment of the present disclosure.

[0024] The terms used in the embodiments of this specification have been selected from widely used, current terms, taking into account the functions of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant embodiments. Therefore, the terms used in this specification should not be defined simply as names of terms, but rather based on their meanings and the overall content of the present disclosure.

[0025] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein.

[0026] Throughout this disclosure, when a part is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used herein refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0027] As used herein, the expression "configured to" can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a system configured to" can mean that the system is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured to perform A, B, and C" can mean a dedicated processor for performing the operations (e.g., an embedded processor), or a general-purpose processor (e.g., a CPU or an application processor) that can perform the operations by executing one or more software programs stored in memory.

[0028] Additionally, when a component is referred to as being "connected" or "connected" to another component in the present disclosure, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless otherwise specifically stated.

[0029] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0030] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0031] FIG. 1 is a conceptual diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to track a gaze based on an encoded image obtained by receiving light transmitted through a multiplexed metalens.

[0032] Referring to FIG. 1, the electronic device may include a polarizing filter (110), a polarization change element (120), a multiplexed metalens (130), an image sensor (140), and a processor (150).

[0033] In one embodiment, the polarizing filter (110) can transmit only the first polarized light (PL1; Polarized Light 1) having a specific polarization among external incident light (IL). The polarizing filter (110) can include a polarizing axis (X1) for polarizing the incident light (IL) in a specific direction. The polarizing filter (110) can transmit a light component vibrating in a direction along the polarization axis (X1) among external incident light (IL) and block a light component vibrating in a direction different from the direction along the polarization axis (X1).

[0034] For example, the polarizing filter (110) can transmit only vertically polarized light among external incident light (IL) or horizontally polarized light. The vertically polarized light and the horizontally polarized light may each be light polarized in a direction perpendicular to the propagation direction of the external incident light (IL).

[0035] In one embodiment, the external incident light (IL) may be light that vibrates in all directions perpendicular to the propagation direction, and may be, for example, natural light. The incident light (IL) is passed through a polarizing filter (110), and the polarizing filter (110) may generate a first polarized light (PL1) in the horizontal direction by polarizing the incident light (IL) in the horizontal direction. Although FIG. 1 illustrates that only the first polarized light (PL1) having a horizontal polarization is transmitted through the polarizing filter (110), the present disclosure is not limited thereto.

[0036] In one embodiment, the external incident light (IL) may be light reflected from an object (10). For example, the polarizing filter (110) may transmit only polarized light having vertical polarization among the incident light (IL) reflected from the object (10). For example, the object (10) may be an eye as illustrated in FIG. 1, and the processor (150) may track the user's gaze based on the light reflected from the eye.

[0037] In one embodiment, the polarization change element (120) can be configured to transmit the first polarization (PL1) in the first direction transmitted by the polarization filter (110) or change it to the second polarization (PL2) in the second direction. The polarization change element (120) can change the phase of a traveling wave when linearly polarized light is incident perpendicularly to a surface. For example, the polarization change element (120) can generate the second polarization (PL2) having the vertical polarization from the first polarization (PL1) having the horizontal polarization. The polarization change element (120) can change the polarization form of the incident light. In addition, the polarization change element (120) can transmit the first polarization (PL1) having the horizontal polarization as it is from the first polarization (PL1) having the horizontal polarization.

[0038] The polarization change element (120) can selectively generate first polarized light (PL1) or second polarized light (PL2) that is linearly polarized in a first direction or a second direction when light linearly polarized in one direction is incident perpendicularly to the front surface of the polarization change element (120). In one embodiment, the first direction and the second direction can be orthogonal to each other.

[0039] However, the number of polarization directions selectable by the polarization change element (120) is merely an example, and the present disclosure is not limited thereto. For example, the processor (150) can control the polarization change element (120) to selectively generate linearly polarized light in one of the first to third directions.

[0040] In one embodiment, the polarization change element (120) may be an element that changes the direction of polarization of incident light according to an external influence such as a mechanical force or an electromagnetic field. In one embodiment, the polarization change element (120) may include a polarization cell and a polarization direction control unit. For example, the polarization cell may be a configuration through which light incident on the polarization change element (120) is transmitted, and a configuration in which the polarization direction of the incident light is changed. The polarization cell may be a single-layer film or a rectangular box. For example, the polarization direction control unit may control the polarization direction that the polarization cell changes for incident light by applying pressure to the polarization cell. The external pressure may mean a mechanical force or an electromagnetic field, etc.

[0041] In one embodiment, the second polarization (PL2) generated by the polarization change element (120) may be incident toward the multiplexed metalens (130). The incident light (IL) transformed by the polarization filter (110) and the polarization change element (120) and the second polarization (PL2) generated by the polarization filter (110) and the polarization change element (120) may include information about the object (10).

[0042] In one embodiment, a multiplexed metalens (130) may be a lens in which nanostructures are arranged in a two-dimensional manner. The multiplexed metalens (130) may be configured to modulate the phase of transmitted light. The multiplexed metalens (130) may change the path of light by modulating the phase of transmitted light. The detailed arrangement shape of the nanostructures included in the multiplexed metalens (130) may vary depending on the optical performance required for the multiplexed metalens (130).

[0043] In one embodiment, a shape of a nanostructure included in a multiplexed metalens (130) may have a relatively strong reactivity to incident light having specific optical properties. The nanostructure included in the multiplexed metalens (130) may change the path or phase of the transmitted light by affecting the vibrational form of the transmitted light. However, the principle of changing the path or phase of the light is not limited to the present disclosure.

[0044] For example, one shape of the nanostructure may have a strong reactivity to incident light having a wavelength of 940 nm, and may change the incident light having a wavelength of 940 nm to propagate in a specific direction when passing through the multiplexed metalens (130). Alternatively, the incident light having a wavelength of 940 nm may be modulated to propagate in a specific phase when passing through the multiplexed metalens (130). At the same time, one shape of the nanostructure may have a weak reactivity to incident light having a wavelength of 850 nm, and may change the incident light having a wavelength of 850 nm to propagate in another direction when passing through the multiplexed metalens (130).

[0045] In one embodiment, the multiplexed metalens (130) may be a lens that changes the transmitted light to travel along different paths depending on its optical properties. As the multiplexed metalens (130) changes the transmitted light to travel along different paths, the electronic device according to the present disclosure may provide a camera module that provides various focal lengths depending on the transmitted light by using the multiplexed metalens (130).

[0046] For example, light having a first wavelength may be incident on the multiplexed metalens (130), and the multiplexed metalens (130) may guide the light having the first wavelength to form an image at a first point on the imaging surface of the image sensor (140). In addition, light having a second wavelength different from the first wavelength may be incident on the multiplexed metalens (130), and the multiplexed metalens (130) may guide the light having the second wavelength to form an image at a second point on the imaging surface of the image sensor (140). The first point may be different from the second point. That is, the multiplexed metalens (130) may change the paths of the incident light to different paths depending on the wavelength of the incident light, and may provide different focal lengths depending on the incident light.

[0047] As another example, polarized light (PL1) having vertical polarization may be incident on the multiplexed metalens (130), and the multiplexed metalens (130) may guide the polarized light (PL1) having vertical polarization to form an image at a first point on the imaging plane of the image sensor (140). In addition, polarized light having horizontal polarization may be incident on the multiplexed metalens (130), and the multiplexed metalens (130) may guide the light having horizontal polarization to form an image at a second point on the imaging plane of the image sensor (140). The first point may be different from the second point. That is, the multiplexed metalens (130) may change the paths of incident light to different paths depending on the polarization direction of the incident light, and may provide different focal lengths depending on the incident light.

[0048] However, the multiplexed metalens (130) can change the incident light to proceed in different paths depending on the optical properties of the incident light, and the wavelength and polarization direction described as examples of the optical properties are only examples and do not limit the present disclosure. In one embodiment, the multiplexed metalens (130) may include a surface made of nanostructures arranged on a substrate, and the path of the transmitted light may be changed depending on the arrangement and shape of the nanostructures of the multiplexed metalens (130). For example, the characteristics of the multiplexed metalens (130) may vary depending on the height, shape, arrangement, properties, etc. of the nanostructures formed on the multiplexed metalens (130).

[0049] The nanostructures included in the multiplexed metalens (130) can guide polarized light incident on the multiplexed metalens (130) to an arbitrary focus according to their arrangement shape. Here, the “array shape” may mean at least one of the size, shape, and arrangement spacing of each nanostructure, the size distribution by position, the shape distribution by position, and the spacing distribution by position of the nanostructures with respect to the area where the multiplexed metalens (130) is located. The detailed arrangement shape of the nanostructures included in the multiplexed metalens (130) may vary depending on the optical performance required for the multiplexed metalens (130). For example, the arrangement shape of the nanostructures may vary depending on the wavelength band, focal length, etc. of light to be focused through the multiplexed metalens (130).

[0050] In one embodiment, the electronic device can obtain an encoded image by using an image sensor (140) to receive second polarization (PL2) transmitted through a multiplexed metalens (130). The electronic device can obtain an encoded image (20) corresponding to an object (10) based on the distribution of the second polarization (PL2) transmitted through the multiplexed metalens (130).

[0051] Light transmitted through the multiplexed metalens (130) is received by the image sensor (140). For example, the image sensor (140) may be a two-dimensional sensor assembled in the form of an array including a plurality of pixels arranged in a matrix, and each of the plurality of pixels may include at least one photoelectric conversion element. The image sensor (140) may detect light using the photoelectric conversion element and output an image signal, which is an electrical signal according to the detected light. An electronic device may obtain an encoded image (20) by converting light received through the image sensor (140) into an electrical signal.

[0052] In one embodiment, the multiplexed metalens (130) is configured to refract incident light into different paths depending on its optical properties. For example, light having vertical polarization may be guided to form an image at a first point on the imaging plane of the image sensor (140) by passing through the multiplexed metalens (130). As another example, light having horizontal polarization may be guided to form an image at a second point on the imaging plane of the image sensor (140) by passing through the multiplexed metalens (130). An electronic device may obtain an encoded image (20) by converting light received at each point on the imaging plane of the image sensor (140) into an electrical signal. The function of the multiplexed metalens (130) can be easily understood through FIG. 2.

[0053] The encoded image (20) may vary depending on at least one of the object (10) from which light is reflected, the arrangement shape of the nanostructures included in the multiplexed metalens (130), and the optical properties of the light refracted by the multiplexed metalens (130). In general, an object expressed in the encoded image (20) phase-modulated by the multiplexed metalens (130) may be difficult to identify with the naked eye.

[0054] In one embodiment, the electronic device can obtain the user's gaze information from the encoded image (20) using the processor (150). The electronic device can track the user's gaze direction based on the user's gaze information.

[0055] In one embodiment, the electronic device may obtain feature points from an encoded image (20) by means of a processor (150). The processor (150) may include an artificial intelligence algorithm or an artificial intelligence network for obtaining feature points. The artificial intelligence algorithm included in the processor (150) may be an algorithm trained to extract feature points from a modified image, i.e., an encoded image (20). The electronic device may track the user's gaze direction based on the feature points extracted using the artificial intelligence algorithm.

[0056] For example, the feature point may include at least one of a pupil feature point and a glint feature point. The electronic device may extract at least one of the pupil feature point and the glint feature point from the encoded image (20). The electronic device may track the user's gaze direction based on at least one of the pupil feature point and the glint feature point extracted using an artificial intelligence algorithm.

[0057] In one embodiment, the electronic device can restore an image from an encoded image (20) using a processor (150). The processor (150) may include an artificial intelligence algorithm or an artificial intelligence network for restoring the image. The artificial intelligence algorithm included in the processor (150) may be an algorithm trained to restore an image from a modified image, i.e., an encoded image (20). The electronic device can track the user's gaze direction based on the restored image using the artificial intelligence algorithm.

[0058] FIG. 2 is a conceptual diagram illustrating a process of tracking gaze based on an encoded image by an electronic device according to an embodiment of the present disclosure.

[0059] For convenience of explanation, details that overlap with those described using Fig. 1 are simplified or omitted. The first polarization (PL1) illustrated in Fig. 2 may be light reflected by an object, for example, an eye, and polarized by a polarizing filter (110 in Fig. 1).

[0060] For reference, the metalens (131, 132) illustrated in FIG. 2 may have the same configuration as the metalens (130) described using FIG. 1. In addition, the image sensors (141, 142) illustrated in FIG. 2 may have the same configuration as the image sensors (140) described using FIG. 1.

[0061] In one embodiment, the electronic device can generate second polarization (PL2) having horizontal polarization from first polarization (PL1) having vertical polarization using a polarization change element (121). The generated second polarization (PL2) can be incident toward a multiplexed metalens (131). The electronic device can obtain a first encoded image (21) by receiving light transmitted by the multiplexed metalens (131) using an image sensor (141). In one embodiment, depending on the area of ​​the image sensor (141), if the light transmitted by the multiplexed metalens (131) is formed outside the imaging surface (145) of the image sensor (141), the obtained first encoded image (21) may not include information about an object, for example, an eye. Therefore, it may be impossible to obtain gaze information based on the obtained first encoded image (21).

[0062] In one embodiment, the electronic device can determine whether or not it can obtain the user's gaze information from the acquired first encoded image (21). In step S10, if the electronic device cannot obtain the user's gaze information from the acquired first encoded image (21), it can control the polarization change element (122) to transmit the first polarization (PL1) having vertical polarization as it is.

[0063] In one embodiment, the electronic device can control the polarization change element (122) so that the first polarized light (PL1) having vertical polarization is transmitted as is. For example, the polarization change elements (121, 122) can generate polarized light (PL1, PL2) having at least two optical properties from the incident first polarized light (PL1) through ON / OFF operation.

[0064] The transmitted first polarization (PL1) may be incident toward the multiplexed metalens (132). The electronic device may obtain a second encoded image (22) by receiving the light transmitted by the multiplexed metalens (132) using the image sensor (142). In one embodiment, depending on the area of ​​the image sensor (142), when the light transmitted by the multiplexed metalens (132) is imaged within the imaging surface (146) of the image sensor (142), the obtained second encoded image (22) may include information about an object, for example, an eye. Therefore, the electronic device may obtain gaze information based on the obtained second encoded image (22).

[0065] The electronic device can obtain the user's gaze information from the second encoded image (22). The electronic device can track the user's gaze based on the second encoded image (22).

[0066] FIG. 3 is a flowchart illustrating an operation method of an electronic device according to an embodiment of the present disclosure.

[0067] In step S310, the electronic device can obtain a first encoded image by receiving the first polarization refracted into the first path by the multiplexed metalens.

[0068] In one embodiment, an electronic device can generate a first polarized light that vibrates in one direction from external incident light. The external incident light can be light reflected from an object, such as an eye. The external incident light can be light that vibrates in any direction perpendicular to the direction of propagation of the light. The electronic device can generate the first polarized light in one direction from the external incident light by transmitting the external incident light through a polarizing filter. The polarizing filter can be configured to transmit the first polarized light in one direction from the incident light.

[0069] In one embodiment, the first polarization in one direction may be light vibrating in a direction perpendicular to the propagation direction. For example, the first polarization in one direction may be a polarization component having horizontal polarization or a polarization component having vertical polarization.

[0070] In one embodiment, a first polarization generated by a polarizing filter can be transmitted by a polarization change element. An electronic device can change the polarization form of the first polarization of the one direction using the polarization change element. The polarization change element can be configured to change the polarization form of the first polarization transmitted by the polarizing filter.

[0071] For example, a polarization converter can modulate the phase of a traveling wave when linearly polarized light is incident perpendicularly to a surface. The polarization converter can generate a second polarization having a perpendicular polarization from a first polarization in one direction. Alternatively, the polarization converter can transmit the first polarization in one direction as is.

[0072] In one embodiment, the multiplexed metalens can be configured to refract incident light into different paths depending on its optical properties. For example, the multiplexed metalens can be configured to refract incident light into different paths depending on the polarization direction of the incident light. The multiplexed metalens can refract light of a first polarization into a first path. The multiplexed metalens can refract light of a second polarization into a second path.

[0073] A multiplexed metalens can be a lens in which nanostructures are arranged in a two-dimensional manner. A multiplexed metalens can be configured to modulate the phase of transmitted light. By modulating the phase of transmitted light, a multiplexed metalens can change the path of light.

[0074] In particular, the shape of the nanostructures included in a multiplexed metalens can exhibit relatively strong reactivity to incident light with specific optical properties. The nanostructures included in a multiplexed metalens can influence the vibrational pattern of transmitted light, thereby altering the path or phase of the transmitted light differently depending on the optical properties.

[0075] In one embodiment, an electronic device can receive a first polarized light transmitted through a multiplexed metalens and transmitted through a first path using an image sensor. The electronic device can obtain a first encoded image by receiving the first polarized light. The electronic device can obtain a first encoded image corresponding to an object based on the distribution of the first polarized light transmitted through the multiplexed metalens.

[0076] The first encoded image may vary depending on at least one of the object from which the light is reflected, the arrangement shape of the nanostructures included in the multiplexed metalens, and the optical properties of the light refracted by the multiplexed metalens. Depending on the irregular arrangement shape of the nanostructures of the multiplexed metalens, the first encoded image obtained by receiving the first polarized light transmitted by the multiplexed metalens may be difficult to discern with the naked eye.

[0077] In step S320, the electronic device can determine whether it can obtain the user's gaze information from the first encoded image.

[0078] For example, light of the first polarization transmitted by a multiplexed metalens may form part of an image of the eye within the image plane of the image sensor. An electronic device may obtain a first encoded image by receiving light of the first polarization, and the electronic device may determine that information about the user's gaze can be obtained based on the first encoded image in which the image of the eye is normally formed.

[0079] In another example, the first polarization transmitted by the multiplexed metalens may form part of an image for the eye outside of the image sensor's imaging plane. The electronic device may not be able to receive the first polarization using the image sensor, and the first encoded image may not contain information about the eye contained in the first polarization. The electronic device may determine that it is possible to obtain information about the user's gaze based on the first encoded image in which the image for the eye is not normally formed.

[0080] In step S330, if the electronic device cannot obtain the user's gaze information based on a determination as to whether the user's gaze information can be obtained, the electronic device can change the first polarization transmitted by the polarization filter to the second polarization using the polarization change element.

[0081] A polarization converter can modulate the phase of a traveling wave when linearly polarized light is incident perpendicularly to a surface. The polarization converter can generate a second polarization having a vertical polarization from a first polarization in one direction. Alternatively, the polarization converter can transmit the first polarization in one direction as is. When an electronic device cannot obtain information about a user's line of sight, the electronic device can generate a second polarization from the first polarization using the polarization converter to utilize the second polarization.

[0082] However, although it has been described that the electronic device uses the first polarization in step S310 and the electronic device generates the second polarization in step S330, the order in which the first polarization and the second polarization are utilized is not limited to the present disclosure. For example, in step S310, the electronic device may utilize the second polarization, and if it is not possible to obtain the user's gaze information using the second polarization, the electronic device may utilize the first polarization as a secondary priority.

[0083] In step S340, the electronic device can obtain a second encoded image by receiving the second polarization refracted into the second path by the multiplexed metalens. Anything overlapping with what was described using step S310 is simplified or omitted.

[0084] In one embodiment, an electronic device can use an image sensor to receive a second polarized light transmitted by a multiplexed metalens. The electronic device can obtain a second encoded image by receiving the second polarized light. The electronic device can obtain a second encoded image corresponding to an object based on the distribution of the second polarized light transmitted by the multiplexed metalens.

[0085] In step S350, the electronic device can obtain the user's gaze information from the second encoded image.

[0086] In one embodiment, the first polarization and the second polarization transmitted by the multiplexed metalens can travel along different paths. The first polarization and the second polarization can be imaged at different locations on the imaging plane of the image sensor. The electronic device can obtain a first encoded image by receiving the first polarization using the image sensor, and can obtain a second encoded image by receiving the second polarization.

[0087] In one embodiment, the first polarization and the second polarization may be imaged at different locations within and outside the imaging plane of the image sensor. For example, the first polarization may be imaged at a point outside the imaging plane of the image sensor. The second polarization may be imaged at a point within the imaging plane of the image sensor. The electronic device may obtain a first encoded image by receiving the first polarization, but may not be able to obtain the user's gaze information based on the obtained first encoded image. The electronic device may obtain a second encoded image by receiving the second polarization, and may obtain the user's gaze information based on the obtained second encoded image.

[0088] FIG. 4 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present disclosure.

[0089] For convenience of explanation, parts that overlap with those described using Figure 3 are simplified or omitted.

[0090] Referring to FIG. 4, step S410 may be performed after step S320 of FIG. 3. In one embodiment, in step S320, the electronic device may determine whether or not it can obtain the user's gaze information. In step S410, if the electronic device can obtain the user's gaze information based on a determination that the user's gaze information can be obtained, the electronic device may obtain the user's gaze information from the first encoded image.

[0091] If the user's gaze information can be obtained, the electronic device can obtain the user's gaze information from the first encoded image according to step S410.

[0092] If the user's gaze information cannot be obtained, the electronic device can generate a second polarization according to step S330. The electronic device can additionally perform steps S340 to S350 to obtain the user's gaze information from the second encoded image obtained by receiving the second polarization.

[0093] FIG. 5 is a flowchart illustrating an operating method of an electronic device according to an embodiment of the present disclosure.

[0094] For convenience of explanation, parts that overlap with those described using Figure 3 are simplified or omitted.

[0095] Referring to FIG. 5, step S510 may be performed after step S340 of FIG. 3. In step 510, the electronic device may re-determine whether it can acquire the user's gaze information from the second encoded image. Step S510 is omitted as it overlaps with what was described using step S320.

[0096] In step S520, if the electronic device cannot obtain the user's gaze information based on a re-determination of whether or not the electronic device can obtain the user's gaze information, the electronic device can change the first polarization transmitted by the polarization filter to a third polarization using a polarization change element. The electronic device can generate the third polarization. Step S520 overlaps with what was described using step S330, and is therefore simplified or omitted.

[0097] In one embodiment, an electronic device can generate a first polarization from external incident light by transmitting the external incident light through a polarization filter. The electronic device can change the polarization form of the first polarization using a polarization change element. The electronic device can transmit the first polarization in a first direction as is, or generate a second polarization in a second direction or a third polarization in a third direction from the first polarization using the polarization change element.

[0098] The third polarization may be different from the first polarization and the second polarization. The vibration direction of the third polarization may be different from the vibration directions of the first polarization and the second polarization, respectively.

[0099] In step S530, the electronic device can obtain a third encoded image by receiving the third polarization refracted into the third path by the multiplexed metalens. Step S530 overlaps with that described using steps S310 and S340, and is therefore simplified or omitted.

[0100] In one embodiment, the electronic device can use an image sensor to receive a third polarized light transmitted by a multiplexed metalens. The electronic device can obtain a third encoded image by receiving the third polarized light. The electronic device can obtain a third encoded image corresponding to an object based on the distribution of the third polarized light transmitted by the multiplexed metalens.

[0101] In step S540, the electronic device can obtain the user's gaze information from the third encoded image. Step S540 overlaps with what was described using step S350, and is therefore briefly described or omitted.

[0102] In one embodiment, the first to third polarizations transmitted by the multiplexed metalens can travel along different paths. The first to third polarizations can form images at different locations on the imaging plane of the image sensor. The electronic device can obtain a first encoded image by receiving the first polarization using the image sensor, a second encoded image by receiving the second polarization, and a third encoded image by receiving the third polarization.

[0103] In one embodiment, the first to third polarizations may be imaged at different locations within or outside the imaging plane of the image sensor. For example, the first and second polarizations may be imaged at locations outside the imaging plane of the image sensor. The third polarization may be imaged at locations within the imaging plane of the image sensor.

[0104] The electronic device may obtain a first encoded image by receiving light of a first polarization, but may not be able to obtain information about the user's gaze based on the obtained first encoded image. The electronic device may obtain a second encoded image by receiving light of a second polarization, but may not be able to obtain information about the user's gaze based on the obtained second encoded image. The electronic device may obtain a third encoded image by receiving light of a third polarization, and may obtain information about the user's gaze based on the obtained third encoded image.

[0105] FIG. 6 is a conceptual diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to track a gaze based on an encoded image obtained by receiving light generated using a light source and a multiplexed metalens.

[0106] For convenience of explanation, any part that overlaps with what was explained using Figure 1 is simplified or omitted.

[0107] Referring to FIG. 6, in one embodiment, the electronic device may further include a light source (102) that outputs light.

[0108] In one embodiment, a light source (102) can output light. The light source (102) can output light toward an object (10). The light output from the light source (102) can be reflected from the object (10) located in front of a light modulation module including a polarizing filter (110), a polarization change element (120), and a multiplexed metalens (130). The light reflected from the object (10) can be incident toward the polarizing filter (110), and the polarization form of the incident light can be modified by the polarizing filter (110). The incident light can be polarized by the polarizing filter (110), the polarization form can be modified by the polarization change element (120), and the path of the light can be changed according to optical properties by the multiplexed metalens (130). For example, the path of the light can be changed according to the wavelength or polarization direction of the light.

[0109] In one embodiment, the light source (102) can output light according to a predetermined wavelength range. For example, the light source (102) can be an infrared light source, and the light source (102) can output light in the infrared band. In another example, the light source (102) can be an ultraviolet light source, and the light source (102) can output light in the ultraviolet band.

[0110] In one embodiment, the light source (102) can output light according to a wavelength range including a wavelength of 850 nm or 940 nm.

[0111] In one embodiment, light output from a light source (102) may be reflected from an object (10). Incident light (IL) incident toward a polarizing filter (110) may be light output from a light source (102) and reflected from an object (10). The incident light (IL) passes through the polarizing filter (110), and the polarizing filter (110) may generate a first polarized light (PL1) in a horizontal direction by polarizing the incident light (IL) in a horizontal direction. Although FIG. 1 illustrates that only the first polarized light (PL1) having a horizontal polarization is transmitted by the polarizing filter (110), the polarization direction is not limited to the present disclosure.

[0112] In one embodiment, a first polarized light (PL1) having horizontal polarization can be transmitted by a polarization change element (120). An electronic device can generate a second polarized light (PL2) having vertical polarization from the first polarized light (PL1) having horizontal polarization by using the polarization change element (120). The second polarized light (PL2) generated by the polarization change element (120) can be incident toward a multiplexed metalens (130).

[0113] In one embodiment, the second polarization (PL2) can be transmitted by the multiplexed metalens (130). The electronic device can refract the transmitted second polarization (PL2) into different paths according to optical properties by using the multiplexed metalens (130). The electronic device can change the path of the second polarization (PL2) by using the multiplexed metalens (130).

[0114] In one embodiment, the electronic device can obtain an encoded image by using an image sensor (140) to receive second polarization (PL2) transmitted through a multiplexed metalens (130). The electronic device can obtain an encoded image (20) corresponding to an object (10) based on the distribution of the second polarization (PL2) transmitted through the multiplexed metalens (130).

[0115] In one embodiment, the electronic device can obtain the user's gaze information from the encoded image (20) using the processor (150). The electronic device can track the user's gaze direction based on the user's gaze information.

[0116] FIG. 7 is a conceptual diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to track a gaze based on an encoded image obtained by receiving at least one of a plurality of lights generated using a plurality of light sources and a multiplexed metalens.

[0117] For convenience of explanation, any part that overlaps with what was explained using Figure 6 is simplified or omitted.

[0118] Referring to FIG. 7, the electronic device may include a plurality of light sources. The electronic device may include a first light source (102) and a second light source (104). The first light source (102) may output a first light. The second light source (104) may output a second light. The wavelength range of the first light and the wavelength range of the second light may be different from each other.

[0119] In one embodiment, the first light source (102) can output the first light. The first light source (102) can output the first light toward the object (10). The first light output from the first light source (102) can be reflected from the object (10) located in front of the light modulation module including a polarizing filter (110), a polarization change element (120), and a multiplexed metalens (130).

[0120] In one embodiment, the first light reflected from the object (10) may be incident toward the polarizing filter (110). In FIG. 7, the first light incident toward the polarizing filter (110) may be the first incident light (IL1). The first light passes through the polarizing filter (110), and the electronic device may generate horizontal polarization by horizontally polarizing the first light using the polarizing filter (110). The first light polarized in the horizontal direction may be the first polarization (PL1).

[0121] In one embodiment, the first polarization (PL1) can be transmitted by the polarization change element (120). The electronic device can generate the second polarization (PL2) having a vertical polarization from the first polarization (PL1) by using the polarization change element (120). The second polarization (PL2) generated by the polarization change element (120) can be incident toward the multiplexed metalens (130).

[0122] The polarization change element (120) can transmit the first polarization (PL1) as it is or change it to the second polarization (PL2). In Fig. 7, the description is based on the changed second polarization (PL2), but the encoded image can also be obtained based on the first polarization (PL1) transmitted as it is.

[0123] The polarization change element (120) can selectively generate polarized light that is linearly polarized in a first direction or a second direction when light that is linearly polarized in one direction is vertically incident on the front surface of the polarization change element (120). In Fig. 7, the second polarized light (PL2) having vertical polarization is shown to be generated, but the second polarized light (PL2) may also have horizontal polarization, and the electronic device can directly transmit the first polarized light (PL1) having vertical polarization or horizontal polarization by controlling the polarization change element (120).

[0124] In one embodiment, the second polarization (PL2) can be transmitted by the multiplexed metalens (130). The electronic device can refract the transmitted second polarization (PL2) into different paths according to optical properties by using the multiplexed metalens (130). The electronic device can change the path of the second polarization (PL2) by using the multiplexed metalens (130).

[0125] In one embodiment, the electronic device can obtain a first encoded image by using an image sensor (140) to receive second polarization (PL2) transmitted through a multiplexed metalens (130). The electronic device can obtain a first encoded image (21) corresponding to an object (10) based on the distribution of the second polarization (PL2) transmitted through the multiplexed metalens (130).

[0126] In one embodiment, the electronic device can obtain the user's gaze information from the first encoded image (21) using the processor (150). The electronic device can track the user's gaze direction based on the user's gaze information.

[0127] In one embodiment, the second light source (104) can output the second light. The second light source (104) can output the second light toward the object (10). The electronic device can obtain the second encoded image (22) by receiving the second light transmitted by the polarizing filter (110), the polarization change element (120), and the multiplexed metalens (130). The method by which the electronic device obtains the second encoded image (22) using the second light is the same as the method by which the electronic device obtains the first encoded image (21) using the first light, and therefore, is omitted.

[0128] In one embodiment, the electronic device can obtain the user's gaze information from the second encoded image (22) using the processor (150). The electronic device can track the user's gaze direction based on the user's gaze information.

[0129] In one embodiment, the electronic device may obtain the user's gaze information from the first encoded image (21) or the second encoded image (22) using the processor (150). The electronic device may track the user's gaze direction based on the user's gaze information.

[0130] For example, the electronic device can determine whether or not it can obtain the user's gaze information from the first encoded image (21). If the electronic device can obtain the user's gaze information from the first encoded image (21), the electronic device can track the user's gaze direction using the first encoded image (21). If the electronic device cannot obtain the user's gaze information from the first encoded image (21), the electronic device can output a second light using the second light source (104) and obtain a second encoded image (22) based on the output second light. The electronic device can track the user's gaze direction from the second encoded image (22).

[0131] Of course, the electronic device can re-determine whether or not it can obtain the user's gaze information from the second encoded image (22). If the electronic device can obtain the user's gaze information from the second encoded image (22), it can track the user's gaze direction using the second encoded image (22). If the electronic device cannot obtain the user's gaze information from the second encoded image (22), it can output a third light using another light source and obtain a third encoded image based on the output third light. The electronic device can track the user's gaze direction from the third encoded image.

[0132] The electronic device may include three or more light sources, and may repeat an operation of generating multiple encoded images based on the multiple lights so that encoded images capable of obtaining information about the user's gaze can be obtained. The number of light sources and the degree of light delay distinguished by the polarization change element are not limited to the present disclosure.

[0133] FIG. 8 is a flowchart illustrating a method for an electronic device according to an embodiment of the present disclosure to track a gaze based on multiple lights generated using multiple light sources and multiplexed metalens.

[0134] For convenience of explanation, parts that overlap with those described using Figure 3 are simplified or omitted.

[0135] In step S810, first to fourth polarizations can be generated using a polarization change element. The electronic device can transmit the first polarization by transmitting the first light through a polarization filter. The electronic device can transmit the first polarization as is from the transmitted first polarization, or change it to the second polarization. The electronic device can transmit the third polarization by transmitting the second light through the polarization filter. The electronic device can transmit the third polarization as is from the transmitted third polarization, or change it to the fourth polarization.

[0136] In one embodiment, the electronic device can change the polarization form of a first polarization or a third polarization using a polarization change element. The electronic device can transmit the first polarization having a first direction polarization as is or generate the second polarization having a second direction polarization from the first polarization. The electronic device can selectively generate the first polarization or the second polarization by controlling the polarization change element.

[0137] In one embodiment, the electronic device may include a first light source and a second light source. The first light source may output a first light. The second light source may output a second light.

[0138] The first light and the second light may be different from each other. The wavelength range of the first light and the wavelength range of the second light may be different from each other. For example, an electronic device may output a first light having a wavelength of 850 nm using a first light source, and may output a second light having a wavelength of 940 nm using a second light source.

[0139] In one embodiment, the electronic device can generate a third polarized light polarized in one direction by transmitting the second light through a polarizing filter.

[0140] In one embodiment, the electronic device can transmit the third polarization light having a third direction polarization as is, or generate the fourth polarization light having a fourth direction polarization from the third polarization light. The electronic device can selectively generate the third polarization light or the fourth polarization light by controlling the polarization changing element. For example, the third direction can be the same as the first direction, and the fourth direction can be the same as the second direction.

[0141] In step S820, the electronic device can obtain a third encoded image by receiving one of the first to fourth polarizations using an image sensor. In step 830, the electronic device can obtain the user's gaze information from the third encoded image.

[0142] In one embodiment, an electronic device can change the path of one of the first to fourth polarizations using a multiplexed metalens. The electronic device can obtain an encoded image by receiving one of the first to fourth polarizations whose path has been changed. The electronic device can obtain information about the user's gaze from the obtained encoded image. The electronic device can track the user's gaze based on the encoded image.

[0143] In one embodiment, the electronic device can acquire an encoded image based on one of the first to fourth polarizations. The electronic device can determine whether or not it can acquire user gaze information from the acquired encoded image. If the electronic device can acquire the user gaze information, the electronic device can track the user's gaze from the acquired encoded image. However, if the electronic device cannot acquire the user's gaze information, the electronic device can reacquire the encoded image based on another one of the first to fourth polarizations. The electronic device can acquire the user's gaze information based on the reacquired encoded image.

[0144] The electronic device can repeat an operation of generating a plurality of encoded images based on the first polarization to the fourth polarization so that an encoded image capable of obtaining the user's gaze information can be obtained.

[0145] In one embodiment, the encoded image may have different shapes depending on the optical properties of the first to fourth polarizations. The optical properties of the first to fourth polarizations may be different from each other, and the encoded images obtained based on the first to fourth polarizations may be different from each other.

[0146] For example, the first polarization and the second polarization may be light having a wavelength of 940 nm, which is light transformed based on light having a wavelength of 940 nm output by the first light source. The third polarization and the fourth polarization may be light having a wavelength of 850 nm, which is light transformed based on light having a wavelength of 850 nm output by the second light source.

[0147] In addition, the first polarization and the second polarization may be light of the same wavelength since they are generated based on the first light output by the same light source, but may be light whose polarization directions are modified to be different from each other by the polarization change element. The third polarization and the fourth polarization may be light of the same wavelength since they are generated based on the second light output by the same light source, but may be light whose polarization directions are modified to be different from each other by the polarization change element.

[0148] FIG. 9 is a conceptual diagram illustrating an operation of an electronic device according to an embodiment of the present disclosure to track gaze based on an encoded image obtained by receiving light output from multiple light sources. For convenience of explanation, details that overlap with those described using FIG. 1 are simplified or omitted.

[0149] Referring to FIG. 9, the electronic device may include a first light source (102), a second light source (104), a multiplexed metalens (230), an image sensor (240), and a processor (250).

[0150] In one embodiment, the first light source (102) can output the first light (L1). The first light source (102) can output the first light (L1) toward the object (10). The first light (L1) output from the first light source (102) can be reflected from the object (10) located in front of the multiplexed metalens (230).

[0151] In one embodiment, the first light (L1) reflected from the object (10) can be transmitted by the multiplexed metalens (230). The electronic device can refract the transmitted first light (L1) into different paths according to optical properties by using the multiplexed metalens (230). The electronic device can change the path of the first light (L1) by using the multiplexed metalens (230).

[0152] In one embodiment, the electronic device can obtain a first encoded image (21) by using an image sensor (240) to receive the first light (L11) transmitted through the multiplexed metalens (230). The electronic device can obtain the first encoded image (21) corresponding to the object (10) based on the distribution of the first light (L11) transmitted through the multiplexed metalens (230).

[0153] In one embodiment, the electronic device can obtain the user's gaze information from the first encoded image (21) using the processor (250). The electronic device can track the user's gaze direction based on the user's gaze information.

[0154] In one embodiment, the second light source (104) can output the second light (L2). The second light source (104) can output the second light (L2) toward the object (10). The second light (L2) output from the second light source (104) can be reflected from the object (10) located in front of the multiplexed metalens (230).

[0155] The method of obtaining the second encoded image (22) using the second light (L2) is the same as the method of obtaining the first encoded image (21) using the first light (L1), so it is omitted.

[0156] In one embodiment, the electronic device can obtain the user's gaze information from the second encoded image (22) using the processor (250). The electronic device can track the user's gaze direction based on the user's gaze information.

[0157] In one embodiment, the electronic device may obtain the user's gaze information from the first encoded image (21) or the second encoded image (22) using the processor (250). The electronic device may track the user's gaze direction based on the user's gaze information.

[0158] For example, the electronic device can determine whether or not it can obtain the user's gaze information from the first encoded image (21). If the electronic device can obtain the user's gaze information from the first encoded image (21), it can track the user's gaze direction using the first encoded image (21). If the electronic device cannot obtain the user's gaze information from the first encoded image (21), it can output the second light (L2) using the second light source (104) and obtain the second encoded image (22) based on the output second light (L2). The electronic device can track the user's gaze direction from the second encoded image (22).

[0159] Of course, the electronic device can re-determine whether or not it can obtain the user's gaze information from the second encoded image (22). If the electronic device can obtain the user's gaze information from the second encoded image (22), it can track the user's gaze direction using the second encoded image (22). If the electronic device cannot obtain the user's gaze information from the second encoded image (22), it can output a third light using another light source and obtain a third encoded image based on the output third light. The electronic device can track the user's gaze direction from the third encoded image.

[0160] The electronic device may include three or more light sources and may repeat an operation of generating multiple encoded images based on the multiple lights so that an encoded image capable of obtaining information about the user's gaze may be obtained. The number of light sources is not limited to the present disclosure.

[0161] Figure 10a is a conceptual diagram for explaining an operation in which a first light of a first wavelength output by a light source forms an image on an image sensor.

[0162] In one embodiment, the electronic device controls a first light source to emit a first wavelength ( ) can output the first light (61). The first light (61) can be reflected by an object and transmitted by the multiplexed metalens (230). The electronic device can change the path of the first light (61) by transmitting the first light (61) through the multiplexed metalens (230). After the first light (61) is transmitted by the multiplexed metalens (230), the electronic device can change the path of the first light (61) so that an image is formed at the first point (P1) on the imaging surface (241) of the image sensor (240). The first light (61) whose path has been changed can reach the first point (P1) in the imaging surface (241) of the image sensor (240) to form an image.

[0163] For example, the nanostructures of the multiplexed metalens (230) are irradiated with the first wavelength of the first light (61). ) can change the path of light so that the first light (61) is focused on the first point (P1) by reacting with the area in a biased manner.

[0164] Figure 10b is a conceptual diagram for explaining an operation in which a second light of a second wavelength output by a light source forms an image on an image sensor.

[0165] In one embodiment, the electronic device controls the second light source to emit a second wavelength ( ) can output the second light (62). The second light (62) can be reflected by the object and transmitted by the multiplexed metalens (230). The electronic device can change the path of the second light (62) by transmitting the second light (62) through the multiplexed metalens (230). The electronic device can change the path of the second light (62) so that an image is formed at the second point (P2) on the imaging surface (241) of the image sensor (240) after the second light (62) is transmitted by the multiplexed metalens (230). The second light (62) whose path has been changed can form an image by reaching the second point (P2) in the imaging surface (241) of the image sensor (240).

[0166] For example, the nanostructures of the multiplexed metalens (230) are capable of transmitting the second wavelength of the second light (62). ) can change the path of light so that the second light (62) is focused on the second point (P2) by reacting with the area in a biased manner.

[0167] In one embodiment, the multiplexed metalens (230) can change the path of transmitted light into different paths depending on the wavelength of the light. Accordingly, the electronic device can change the path of the light so that the first light of FIG. 10A is imaged at the first point (P1), and the second light of FIG. 10B is imaged at the second point (P2) by using the multiplexed metalens (230).

[0168] In one embodiment, unlike as illustrated in FIGS. 10A and 10B, the electronic device can change the path of light so that an image is formed outside the imaging surface (241) of the image sensor (240) by transmitting the light through the multiplexed metalens (230). If the image is formed outside the imaging surface (241), the electronic device can obtain an encoded image based on the light received through the image sensor (240), but can determine that the user's gaze information cannot be obtained from the obtained encoded image. Based on the determination, the electronic device can obtain an encoded image based on another light, and obtain the user's gaze information from the obtained encoded image.

[0169] FIG. 11 is a flowchart illustrating a method for tracking gaze by extracting feature points based on an encoded image by an electronic device according to an embodiment of the present disclosure. Step S350 of FIG. 3 may include steps S1110 and S1120.

[0170] In step S1110, the electronic device can obtain feature points about the eye from the encoded image.

[0171] In the present disclosure, a feature point may refer to a point representing a key feature or point of interest in an image. For example, a feature point may be a corner where two or more edges intersect, detected based on changes in pixel values, or a point with a maximum or minimum pixel value in an image.

[0172] A feature point may include information about at least one of the position coordinates and shape of an object. For example, a feature point may include at least one of a pupil feature point and a glint feature point.

[0173] In one embodiment, an electronic device can acquire feature points from an encoded image. The electronic device can input the acquired encoded image into an artificial intelligence (AI) model to acquire feature points. The AI ​​model may be a model trained to extract feature points. The AI ​​model may be a model trained to extract feature points from an encoded image acquired based on light transmitted through a multiplexed metalens.

[0174] In one embodiment, the electronic device may input a first encoded image or a second encoded image into an artificial intelligence model to obtain feature points. The encoded image input to obtain feature points is not limited to the present disclosure.

[0175] In step S1120, the electronic device can obtain the user's gaze information based on the feature points.

[0176] In one embodiment, an electronic device can obtain user gaze information based on feature points. The electronic device can track the user's gaze direction based on the feature points. For example, the user's gaze direction can be determined by considering factors such as the shape of the pupil and the location of glint coordinates.

[0177] FIG. 12 is a flowchart illustrating a method for tracking gaze by restoring an image based on an encoded image by an electronic device according to an embodiment of the present disclosure. Step S350 of FIG. 3 may include steps S1210 and S1220.

[0178] In step S1210, the electronic device can obtain a restored image of the eye from the encoded image.

[0179] The encoded image may vary depending on at least one of the object reflecting the light and the pattern shape of the multiplexed metalens. In general, the object represented in the encoded image phase-modulated by the multiplexed metalens may be difficult to discern with the naked eye.

[0180] In one embodiment, an electronic device can obtain a restored image from an encoded image. The electronic device can input the obtained encoded image into an artificial intelligence model to obtain a restored image. The artificial intelligence model may be a model trained to restore images. The artificial intelligence model may be a model trained to obtain a restored image by restoring an encoded image obtained based on light transmitted through a multiplexed metalens.

[0181] In step S1220, the electronic device can obtain the user's gaze information based on the restored image.

[0182] In one embodiment, the electronic device can obtain information about the user's gaze based on the restored image. The electronic device can track the user's gaze direction based on the restored image. For example, the user's gaze direction can be determined by considering the shape of the pupil, the gaze direction, etc. in the restored image.

[0183] FIG. 13 is a block diagram illustrating components of an electronic device (1300) according to one embodiment of the present disclosure.

[0184] For convenience of explanation, parts that overlap with those described using Figures 1 and 2 are simplified or omitted.

[0185] The electronic device (1300) may be a device that acquires an image by receiving light transmitted through a multiplexed metalens (1320) via an image sensor (1330). The electronic device (1300) may be implemented as various devices, such as, for example, a mobile device, a smart phone, a laptop computer, a desktop, a tablet PC, a wearable device, an e-book reader, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation device, an MP3 player, a camcorder, etc. In one embodiment of the present disclosure, the electronic device (1300) may be an augmented reality device. An augmented reality device is a device that can express 'augmented reality', and can include not only augmented reality glasses that are generally worn on the user's face, but also a head-mounted display apparatus (HMD) or an augmented reality helmet that is worn on the head.

[0186] Referring to FIG. 13, the electronic device (1300) may include a light control unit (1310), a multiplexed metalens (1320), an image sensor (1330), a processor (1340), and a memory (1350). The light control unit (1310), the processor (1340), and the memory (1350) may each be electrically and / or physically connected to each other.

[0187] The components illustrated in FIG. 13 are merely in accordance with one embodiment of the present disclosure, and the components included in the electronic device (1300) are not limited to those illustrated in FIG. 13. The electronic device (1300) may not include some of the components illustrated in FIG. 13, and may further include components not illustrated in FIG. 13. For example, the electronic device (1300) may further include a power supply (e.g., a battery) that supplies driving power to the light control unit (1310), the processor (1340), and the memory (1350).

[0188] In one embodiment, the image sensor (1330) of the present disclosure may be used to track gaze. In this case, the image sensor (1330) typically uses an infrared (IR) light source. To track gaze using the image sensor (1330), light in the infrared band output from the infrared light source may be reflected from the eye and transmitted through a multiplexed metalens (1320). The light reflected from the eye and transmitted through the multiplexed metalens (1320) may be received by the image sensor (1330).

[0189] The light control unit (1310) can be configured to selectively generate a plurality of lights having different optical properties.

[0190] In one embodiment, the light control unit (1310) may include a polarizing filter and a polarization change element.

[0191] A polarizing filter can transmit only polarized light (PL) with a specific polarization among incident light. A polarization change element can change the phase of a traveling wave when linearly polarized light is incident perpendicular to a surface.

[0192] The electronic device (1300) can selectively generate one of a plurality of lights having distinct optical properties based on externally incident light using a light control unit (1310). The externally incident light can be polarized in one direction by a polarizing filter.

[0193] Polarized light can have its phase modulated by a polarization change element. The electronic device (1300) can modulate the phase of the polarized light by controlling the polarization change element, and can selectively generate one of a plurality of lights having different phases depending on the degree to which the phase of the polarized light is modulated.

[0194] In one embodiment, the light control unit (1310) may include a first light source and a second light source.

[0195] A first light source can output light having a first wavelength. A second light source can output light having a second wavelength. The first wavelength and the second wavelength may be different. For example, the first wavelength may be 850 nm and the second wavelength may be 940 nm.

[0196] The electronic device (1300) can selectively generate one of two lights having different wavelengths by controlling the first light source and the second light source.

[0197] In one embodiment, the light control unit (1310) may include a first light source, a second light source, a polarizing filter, and a polarization change element.

[0198] The electronic device (1300) can selectively generate one of the first light having a first wavelength and the second light having a second wavelength by controlling the first light source and the second light source.

[0199] When the first light is incident, the electronic device (1300) can modulate the phase of the polarized first light by controlling the polarization filter and the polarization change element, and can selectively generate one of a plurality of first lights having different phases depending on the degree to which the phase of the polarized first light is modulated.

[0200] When the second light is incident, the electronic device (1300) can modulate the phase of the polarized second light by controlling the polarization filter and the polarization change element, and can selectively generate one of a plurality of second lights having different phases depending on the degree to which the phase of the polarized second light is modulated.

[0201] A multiplexed metalens (1320) may be a lens in which relatively small-scale nanostructures are arranged in a two-dimensional manner. The multiplexed metalens (1320) may include a surface formed of nanostructures arranged on a substrate, and the phase of light transmitted through the multiplexed metalens (1320) may be modulated depending on the arrangement and shape of the nanostructures.

[0202] A multiplexed metalens (1320) can be configured to refract incident light into different paths depending on its optical properties. The electronic device (1300) can use the multiplexed metalens (1320) to refract incident light into different paths, thereby controlling the position at which an image is formed.

[0203] For example, the electronic device (1300) can obtain an encoded image of an object using the second light refracted by the multiplexed metalens (1320), even if the first light refracted by the multiplexed metalens (1320) does not form an image within the imaging plane of the image sensor (1330). The electronic device (1300) can normally obtain an encoded image of an object using a plurality of lights if a portion of the path of at least one light refracted by the multiplexed metalens (1320) forms an image within the imaging plane of the image sensor (1330).

[0204] Accordingly, the electronic device (1300) can normally acquire an encoded image of an object if a portion of the path of at least one light refracted by the multiplexed metalens (1320) is formed within the imaging plane of the image sensor (1330) by using at least one light among the plurality of lights, and can track the user's gaze based on the encoded image. Since the electronic device can utilize a plurality of lights, the area of ​​the image sensor (1330) can be minimized, and if an image of at least one light is formed within the imaging plane of the miniaturized image sensor (1330), the user's gaze can be tracked.

[0205] For example, the image sensor (1330) can be reduced in length and width, and its width can also be reduced as fewer internal components are required. The electronic device (1300) including the image sensor (1330), i.e., the eye tracking camera system, can be similarly miniaturized. Specifically, for example, the volume of the existing eye tracking camera is 3.2 * 3.2 * 3.9 (mm). 3 ), the volume of the eye tracking camera to which the electronic device (1300) according to one embodiment of the present disclosure is applied is 1.6 * 1.6 * 1.5 (mm) 3 ) may be.

[0206] Additionally, as the image sensor (1330) becomes smaller, the power consumption of the electronic device (1300) can be reduced. For example, if the power consumption of an existing gaze tracking camera was 85 mW, the power consumption of the gaze tracking camera to which the electronic device (1300) according to an embodiment of the present disclosure is applied can be 42.5 mW. According to an embodiment of the present disclosure, the power consumption of the gaze tracking camera can be reduced by 50%.

[0207] However, this is only an example, and the degree of miniaturization of the image sensor (1330) and the power consumption of the electronic device (1300) depending on the required performance required by the situation do not limit the present disclosure.

[0208] The image sensor (1330) is a photographing element configured to receive light transmitted by the multiplexed metalens (1320), convert the brightness or intensity of the received light into an electrical signal, and image the electrical signal, thereby obtaining a coded image. The image sensor (1330) may be implemented as, for example, a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS), but is not limited thereto.

[0209] The processor (1340) may execute one or more instructions or program codes stored in the memory (1350), and perform functions and / or operations corresponding to the instructions or program codes. The processor (1340) may be configured with hardware components that perform arithmetic, logic, and input / output operations and signal processing. The processor (1340) may be configured with at least one of, for example, a central processing unit (CPU), a microprocessor, a graphic processing unit (GRAP), an application processor (AP), an application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs), but is not limited thereto.

[0210] Although the processor (1340) is illustrated as a single element in FIG. 13, it is not limited thereto. In one embodiment, the processor (1340) may be configured as one or more than one processor.

[0211] In one embodiment of the present disclosure, the processor (1340) may be configured as a dedicated hardware chip that performs artificial intelligence (AI) learning.

[0212] The memory (1350) may store instructions and program codes that can be read by the processor (1340). The memory (1350) may include, for example, at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a mask ROM, a flash ROM, a flash ROM, etc.), a hard disk drive (HDD), or a solid state drive (SSD).

[0213] The memory (1350) may store instructions or program codes for performing functions or operations of the electronic device (1300). In one embodiment of the present disclosure, the memory (1350) may store at least one of instructions, algorithms, data structures, program codes, and application programs that can be read by the processor (1340). The instructions, algorithms, data structures, and program codes stored in the memory (1350) may be implemented in a programming or scripting language such as, for example, C, C++, Java, or an assembler.

[0214] The memory (1350) may store commands, algorithms, data structures, or program codes related to an artificial intelligence model for obtaining user gaze information from an encoded image. For example, the memory (1350) may store program codes for obtaining feature points or restored images from an encoded image and tracking the user's gaze using the feature points or restored images. A 'module' included in the memory (1350) refers to a unit that processes a function or operation performed by the processor (1340), and this may be implemented as software such as commands, algorithms, data structures, or program codes.

[0215] The processor (1340) can track the user's gaze from the encoded image. The processor (1340) can track the user's gaze by using an artificial intelligence algorithm trained to obtain the user's gaze information from the encoded image phase-modulated by a predetermined pattern of the multiplexed metalens (1320).

[0216] In the following embodiments, the processor (1340) may be implemented by executing instructions or program codes stored in the memory (1350).

[0217] In one embodiment, the processor (1340) can obtain a first coded image by receiving a first polarization light refracted into a first path by the multiplexed metalens (1320). The processor (1340) can determine whether or not the user's gaze information can be obtained from the first coded image. If the processor (1340) cannot obtain the user's gaze information based on the determination, the processor (1340) can change the first polarization light transmitted by the polarization filter into a second polarization light using a polarization change element. The processor (1340) can obtain a second coded image by receiving the second polarization light refracted into a second path by the multiplexed metalens (1320). The processor (1340) can obtain the user's gaze information from the second coded image.

[0218] In one embodiment, the processor (1340) may obtain the user's gaze information from the first encoded image if the processor (1340) can obtain the user's gaze information based on the decision.

[0219] In one embodiment, the processor (1340) can re-determine whether the user's gaze information can be obtained from the second encoded image. If the processor (1340) cannot obtain the user's gaze information based on the re-determination, the processor (1340) can change the first polarization transmitted by the polarization filter to a third polarization using a polarization change element. The processor (1340) can obtain a third encoded image by receiving the third polarization refracted to the third path by the multiplexed metalens (1320). The processor (1340) can obtain the user's gaze information from the third encoded image.

[0220] In one embodiment, by transmitting a first light through a polarizing filter, the first polarization is transmitted, and the first polarization can be transmitted without change or changed to a second polarization from the first polarization transmitted by the polarization change element. By transmitting a second light through the polarizing filter, the third polarization is transmitted, and the third polarization can be transmitted without change or changed to a fourth polarization from the third polarization transmitted by the polarization change element. The processor (1340) can obtain a third encoded image by receiving the first polarization, the second polarization, the third polarization, or the fourth polarization using the image sensor (1330). The processor (1340) can obtain information about the user's gaze from the third encoded image.

[0221] In one embodiment, the processor (1340) can obtain feature points related to the eye from the second encoded image. The processor (1340) can obtain information about the user's gaze based on the feature points.

[0222] In one embodiment, the processor (1340) can obtain a restored image of the eye from the second encoded image. The processor (1340) can obtain information about the user's gaze based on the restored image.

[0223] In one embodiment, the processor (1340) may obtain a first encoded image by receiving first light refracted by the multiplexed metalens (1320) along a first path using the image sensor (1330). The processor (1340) may determine whether or not the user's gaze information can be obtained from the first encoded image. If the processor (1340) cannot obtain the user's gaze information based on the determination, the processor (1340) may obtain a second encoded image by receiving second light refracted by the multiplexed metalens (1320) along a second path using the image sensor (1330). The processor (1340) may obtain the user's gaze information from the second encoded image.

[0224] In one embodiment, the processor (1340) may obtain the user's gaze information from the first encoded image if the processor (1340) can obtain the user's gaze information based on the decision.

[0225] In one embodiment, the processor (1340) may re-determine whether the user's gaze information can be obtained from the second encoded image. If the processor (1340) cannot obtain the user's gaze information based on the re-determination, the processor (1340) may obtain a third encoded image by receiving the third light refracted by the multiplexed metalens (1320) along a third path using the image sensor (1330). The processor (1340) may obtain the user's gaze information from the third encoded image.

[0226] In one embodiment, the processor (1340) can obtain feature points related to the eye from the second encoded image. The processor (1340) can obtain information about the user's gaze based on the feature points.

[0227] In one embodiment, the processor (1340) can obtain a restored image of the eye from the second encoded image. The processor (1340) can obtain information about the user's gaze based on the restored image.

[0228] In one embodiment, a processor of an electronic device (1300) may utilize an artificial intelligence model trained to obtain user gaze information from an encoded image. In one embodiment of the present disclosure, the artificial intelligence model may be a deep neural network model trained by a supervised learning method by applying an encoded image obtained according to a simulated light distribution based on a multiplexed metalens (1320) as input data and applying the user's gaze corresponding to the encoded image as an output ground truth. 'Training' may mean training a neural network to be able to discover or acquire on its own a method of analyzing input data for the neural network, a method of classifying input data, and / or a method of extracting features necessary for generating result data from input data. Specifically, through a learning process, the deep neural network model may learn learning data (e.g., multiple original images and multiple feature points) and optimize weight values ​​within the neural network. A deep neural network model processes input data through a neural network with optimized weight values ​​to output the desired result.

[0229] The type of artificial intelligence model is not limited to the present disclosure, and the artificial intelligence model may be implemented as any one of a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network. In addition, the artificial intelligence model may be subdivided. For example, the convolutional neural network model may be subdivided into a deep convolutional neural network (D-CNN) or a Capsnet neural network.

[0230] The processor (1340) can obtain the user's gaze information from the encoded image using a pre-trained artificial intelligence model. In one embodiment of the present disclosure, the processor (1340) inputs the encoded image obtained through the image sensor (1330) into the artificial intelligence model and performs inference using the artificial intelligence model, thereby obtaining the user's gaze information corresponding to the encoded image. The pre-trained artificial intelligence model can obtain, for example, feature points or restored images corresponding to the encoded image, and obtain the user's gaze information based on the feature points or restored images.

[0231] The artificial intelligence model may be stored in the memory (1350) of the electronic device (1300). However, this is merely an example and does not limit the present disclosure. For example, the artificial intelligence model may be stored in an external server. In this case, the electronic device (1300) further includes a communication interface capable of performing data communication with the external server, and may receive the artificial intelligence model or the inference result data (e.g., feature points) by the artificial intelligence model from the external server through the communication interface. In general, the electronic device (1300) may be limited compared to the server in terms of memory storage capacity, computational processing speed, and training data set collection capability. Therefore, operations requiring large-scale data storage and large-scale computational amounts may be performed on the server, and then the necessary data and / or the artificial intelligence model may be transmitted to the electronic device (1300) via a communication network. In this case, the electronic device (1300) may receive and utilize the artificial intelligence model or the inference data by the artificial intelligence model through the server, without requiring a processor with large-scale memory and fast computational capabilities, thereby performing the necessary operations quickly and easily.

[0232] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0233] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

Claims

1. A polarizing filter configured to receive incident light and change the polarization of the incident light; A polarization changing element configured to receive light from the polarizing filter; A metalens configured to refract light received from the polarization changing element into different paths depending on the polarization direction of the light received from the polarization changing element; An image sensor configured to receive the refracted light from the metalens; at least one processor; and Contains memory, By causing said at least one processor to execute at least one instruction stored in said memory, the electronic device, The polarizing filter changes the incident light into a first polarization in a first direction and transmits the first polarization (cause), Through the image sensor, a first encoded image based on the first polarization refracted into the first path by the metalens is obtained, Based on whether the user's gaze information can be obtained from the first encoded image: The polarization change element changes the first polarization into the second polarization in the second direction and transmits the second polarization (cause), Through the image sensor, a second encoded image is obtained based on the second polarization refracted into the second path by the metalens, An electronic device that obtains the user's gaze information from the second encoded image.

2. In paragraph 1, By causing said at least one processor to execute at least one instruction stored in said memory, the electronic device, An electronic device that obtains the user's gaze information from the first encoded image based on whether the user's gaze information can be obtained from the first encoded image.

3. In paragraph 1, An electronic device wherein the first direction is perpendicular to the second direction.

4. In paragraph 1, By causing said at least one processor to execute at least one instruction stored in said memory, the electronic device, Based on whether the user's gaze information can be obtained from the second encoded image: The polarization change element changes the first polarization into a third polarization in a third direction and transmits the third polarization (cause), Through the image sensor, a third encoded image is obtained based on the third polarization refracted to the third path by the metalens, An electronic device that obtains the user's gaze information from the third encoded image.

5. In paragraph 1, The first encoded image is formed by the first polarization reaching a first point in the imaging plane of the image sensor along the first path, An electronic device, wherein the second encoded image is formed by the second polarization reaching a second point within the imaging plane of the image sensor along the second path.

6. In paragraph 1, The above first path does not intersect the imaging plane of the image sensor, An electronic device wherein the second encoded image is formed by the second polarization reaching the imaging plane of the image sensor along the second path.

7. In paragraph 1, Further comprising a first light source outputting a first light, By causing said at least one processor to execute at least one instruction stored in said memory, the electronic device, An electronic device that allows the first polarized light to be transmitted by transmitting the first light through the polarizing filter.

8. In paragraph 4, a first light source outputting a first light; and Further comprising a second light source outputting a second light, By causing said at least one processor to execute at least one instruction stored in said memory, the electronic device, An electronic device that transmits the first polarized light by transmitting the first light through the polarizing filter, or transmits the third polarized light by transmitting the second light through the polarizing filter.

9. In paragraph 8, An electronic device wherein the wavelength range of the first light and the wavelength range of the second light are different from each other.

10. In paragraph 1, a first light source outputting a first light; and Further comprising a second light source outputting a second light, By causing said at least one processor to execute at least one instruction stored in said memory, the electronic device, By transmitting the first light through the polarizing filter, the first polarized light is transmitted, The polarization change element causes the first polarization to be transmitted, or changes the first polarization to the second polarization and transmits the second polarization. By transmitting the second light through the polarizing filter, the third polarized light in the third direction is transmitted, The polarization change element causes the third polarization to be transmitted, or changes the third polarization into the fourth polarization in the fourth direction and causes the fourth polarization to be transmitted. An electronic device that obtains the user's gaze information from a third encoded image based on the first polarization, the second polarization, the third polarization, or the fourth polarization through an image sensor.

11. In paragraph 1, By causing said at least one processor to execute at least one instruction stored in said memory, the electronic device, Obtaining feature points about the eye from the second encoded image, An electronic device that obtains the user's gaze information based on the above-mentioned feature points.

12. In paragraph 1, By causing said at least one processor to execute at least one instruction stored in said memory, the electronic device obtains a restored image of the eye from said second encoded image, An electronic device that obtains gaze information of the user based on the restored image.

13. In paragraph 1, The above metalens comprises a surface formed of a plurality of nanostructures aligned on a substrate, An electronic device wherein the metalens is configured to refract light received from the polarization changing element into different paths by modulating the phase of the light received from the polarization changing element based on one of the polarization and wavelength of the light received from the polarization changing element.

14. In paragraph 1, An electronic device wherein the metalens is configured to refract light received from the polarization change element into different paths based on at least one of a size of each of the plurality of nanostructures, a shape of each of the plurality of nanostructures, and an arrangement of the plurality of nanostructures on the substrate.

15. A step of obtaining a first coded image by receiving first polarized light refracted along a first path by a metalens configured to refract incident light along different paths according to a polarization direction; A step of determining whether the user's gaze information can be obtained from the first encoded image; Based on whether the user's gaze information can be obtained from the first encoded image: A step of obtaining a second encoded image by receiving a second polarized light refracted to a second path by the metalens; and A step of obtaining the user's gaze information from the second encoded image; A method comprising:

Images