Wearable device and method for identifying eye movement patterns, and non-transitory computer-readable recording medium

Abstract

A wearable device is disclosed. The wearable device may comprise: a display; a sensor that acquires data for identifying a user gaze; at least one processor including a processing circuit; and a memory, which stores instructions and includes at least one storage medium. The wearable device may identify movements of the user gaze through the sensor while displaying content through the display. The movements can include a combination of gaze saccade and gaze fixation. The wearable device may: identify, on the basis of the characteristics of the saccade and / or the fixation of the movement, at least one read pattern corresponding to the movements from among a plurality of read patterns having different saccade characteristics and / or fixation characteristics; and provide a read assistance function corresponding to the at least one read pattern through the display.

Description

Wearable device, method, and non-transitory computer-readable recording medium for identifying eye movement patterns

[0001] The following descriptions relate to a wearable device, method, and non-transitory computer-readable recording medium for identifying eye movement patterns.

[0002] Eye tracking refers to a technology that tracks eye movements. An electronic device can acquire images of the user's eyes through a sensor and identify the user's gaze through these images. The electronic device can then identify eye movements based on changes in gaze.

[0003] A wearable device is disclosed. The wearable device may include a display. The wearable device may include a sensor that acquires data for identifying a user's gaze. The wearable device may include at least one processor including a processing circuit. The wearable device may include a memory that stores instructions and includes one or more storage media. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify a movement of the user's gaze through the sensor while displaying content through the display. The movement may include a combination of a sprint of the gaze and a fixation of the gaze. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, based on the characteristics of the leap and / or the fixation of the movement, at least one reading pattern corresponding to the movement among a plurality of reading patterns having different characteristics of leap and / or fixation. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to provide, through the display, a reading assistance function corresponding to the at least one reading pattern.

[0004] A method is disclosed. The method may be performed on a wearable device including a display and a sensor for acquiring data for identifying a user's gaze. The method may include an operation of identifying a movement of the user's gaze through the sensor while displaying content through the display. The movement may include a combination of a leap of the gaze and a fixation of the gaze. The method may include an operation of identifying at least one reading pattern corresponding to the movement among a plurality of reading patterns having different leap characteristics and / or fixation characteristics based on a characteristic of the leap of the movement and / or the fixation. The method may include an operation of providing a reading assistance function corresponding to the at least one reading pattern through the display.

[0005] A non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium may store a program including instructions. The instructions, when individually or collectively executed by at least one processor of a wearable device including a display and a sensor for obtaining data for identifying a user's gaze, may cause the wearable device to identify a movement of the user's gaze through the sensor while displaying content via the display. The movement may include a combination of a hop of the gaze and a fixation of the gaze. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, based on characteristics of the hop of the movement and / or the fixation, at least one reading pattern corresponding to the movement among a plurality of reading patterns having different characteristics of the hop and / or the fixation. The above instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to provide, through the display, a reading assistance function corresponding to the at least one reading pattern.

[0006] FIG. 1 illustrates an example of a block diagram of a wearable device according to one embodiment.

[0007] FIG. 2 is a diagram illustrating a situation in which a wearable device identifies a user's gaze according to one embodiment.

[0008] FIG. 3 is a diagram illustrating the movement of a user's gaze identified by a wearable device according to one embodiment.

[0009] FIG. 4A is a diagram illustrating eye movement corresponding to a reading pattern according to one embodiment.

[0010] FIG. 4b is a diagram illustrating gaze movements corresponding to a scanning pattern according to one embodiment.

[0011] FIG. 4c is a diagram illustrating gaze movements corresponding to a cross-reading pattern according to one embodiment.

[0012] FIG. 5 is a diagram illustrating changes between states of a reading pattern according to one embodiment.

[0013] FIG. 6 is a diagram illustrating changes in the similarity between gaze movement and reading patterns according to one embodiment.

[0014] FIG. 7 is a diagram illustrating a change in state over time of each of a plurality of reading patterns according to one embodiment.

[0015] FIG. 8 is a diagram illustrating a change in state over time of each of a plurality of reading patterns according to one embodiment.

[0016] FIG. 9 is a diagram illustrating a change in state over time of each of a plurality of reading patterns according to one embodiment.

[0017] FIG. 10 is a diagram illustrating a change in state over time of each of a plurality of reading patterns according to one embodiment.

[0018] FIG. 11 is a diagram illustrating an operation of a wearable device providing a reading assistance function corresponding to a reading pattern according to one embodiment.

[0019] FIG. 12 is a diagram illustrating an operation of a wearable device providing a reading assistance function corresponding to a scanning pattern according to one embodiment.

[0020] FIG. 13 is a diagram illustrating an operation of a wearable device providing a reading assistance function corresponding to a cross-reading pattern according to one embodiment.

[0021] FIG. 14 is a diagram illustrating an operation of a wearable device providing a reading assistance function corresponding to a free view pattern according to one embodiment.

[0022] FIG. 15 is a diagram illustrating an operation of a wearable device generating a lifelog according to gaze reading patterns according to one embodiment.

[0023] FIG. 16 is a flowchart illustrating the operation of a wearable device according to one embodiment.

[0024] FIG. 17 is a diagram illustrating a situation in which an electronic device identifies a user's gaze according to one embodiment.

[0025] FIG. 18 is a block diagram of an electronic device within a network environment according to various embodiments.

[0026] FIG. 1 is a block diagram of a wearable device (101) within a network environment (100) according to various embodiments.

[0027] FIG. 1 illustrates an example of a block diagram of a wearable device according to one embodiment.

[0028] Referring to FIG. 1, a wearable device (101) according to one embodiment may include at least one of a processor (110), a memory (115), a display (120), a camera (125), a sensor (130), or a communication circuit (135). The processor (110), the memory (115), the display (120), the camera (125), the sensor (130), and the communication circuit (135) may be electrically and / or operatively connected to each other by an electronic device (or electrical device) such as a communication bus (102). The type and / or number of hardware components included in the wearable device (101) is not limited to those illustrated in FIG. 1. For example, the wearable device (101) may include only some of the hardware components illustrated in FIG. 1. The wearable device (101) according to one embodiment may correspond to the electronic device (1801) of FIG. 18.

[0029] The elements (e.g., layers and / or modules) within the memory described below may be logically separated. The elements within the memory (115) may be included within a hardware component that is separate from the memory (115). The operation performed by the processor (110) using each element within the memory (115) is one embodiment, and the processor (110) may perform a different operation from the above operation through at least one of the elements within the memory (115).

[0030] A processor (110) of a wearable device (101) according to an embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), and / or a central processing unit (CPU). The number of processors (110) may be one or more. For example, the processor (110) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core. The processor (110) of a wearable device (101) according to an embodiment may correspond to the processor (1810) of FIG. 18.

[0031] According to an embodiment, a memory (115) of a wearable device (101) may include a hardware component for storing data and / or instructions input and / or output to a processor (110). The memory (115) may include, for example, a volatile memory such as a random-access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multi media card (eMMC).

[0032] The memory (115) of the wearable device (101) according to one embodiment may correspond to the memory (1830) of FIG. 18.

[0033] In one embodiment, the display (120) of the wearable device (101) can output visualized information to a user of the wearable device (101). For example, the display (120) can be controlled by a processor (110) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (120) can include a flat panel display (FPD) and / or electronic paper. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs).

[0034] The display (120) of the wearable device (101) according to one embodiment may correspond to the display module (1860) of FIG. 18.

[0035] In one embodiment, the camera (125) of the wearable device (101) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (125) may be arranged in the form of a two-dimensional array. The camera (125) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (125) may mean one (a) two-dimensional frame data acquired from the camera (125). For example, video data captured using the camera (125) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (125) according to a frame rate. The camera (125) may further include a flash light that is positioned toward the direction in which the camera (125) receives light and outputs light toward the direction.

[0036] According to one embodiment, the wearable device (101) may include a plurality of cameras, for example, cameras (125), arranged in different directions. A first camera among the plurality of cameras may be referred to as a motion recognition camera, and a second camera may be referred to as a gaze tracking camera. The wearable device (101) may identify a position, shape, and / or gesture of a hand using an image acquired using the first camera. The wearable device (101) may identify a direction of a gaze of a user wearing the wearable device (101) using an image acquired using the second camera. For example, the direction in which the first camera faces may be opposite to the direction in which the second camera faces.

[0037] The camera (125) of the wearable device (101) according to one embodiment may correspond to the camera module (1880) of FIG. 18.

[0038] According to one embodiment, a sensor (130) of a wearable device (101) may generate electrical information that may be processed by a processor (110) and / or a memory (115) of the wearable device (101) from non-electronic information related to the wearable device (101). The information may be referred to as sensor data. The sensor (130) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (101), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (101).

[0039] The sensor (130) of the wearable device (101) according to one embodiment may correspond to the sensor module (1876) of FIG. 18.

[0040] In one embodiment, the communication circuit (135) of the wearable device (101) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (101) and an external electronic device. The communication circuit (135) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (135) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.

[0041] The communication circuit (135) of the wearable device (101) according to one embodiment may correspond to the communication module (1890) of FIG. 18.

[0042] According to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (110) of the wearable device (101) may be stored in the memory (115) of the wearable device (101). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, and / or an application. For example, the wearable device (101) and / or the processor (110) may perform at least one of the operations of FIG. 16 when a set of multiple instructions distributed in the form of an operating system, firmware, driver, and / or application is executed. Hereinafter, the fact that an application is installed in a wearable device (101) may mean that one or more instructions provided in the form of an application are stored in a memory (115), and that the one or more applications are stored in a format executable by the processor (110) (e.g., a file having an extension specified by the operating system of the wearable device (101)). For example, an application may include a program and / or a library related to a service provided to a user.

[0043] Referring to FIG. 1, programs installed in the wearable device (101) may be classified into any one of different layers, including an application layer (140), a framework layer (150), and / or a hardware abstraction layer (HAL) (180), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (120), the camera (120), and / or the sensor (130)) of the wearable device (101) may be classified within the hardware abstraction layer (180). The framework layer (150) may be referred to as an XR framework layer in that it includes one or more programs for providing an XR (extended reality) service. For example, FIG. 1 illustrates layers being divided within the memory (115), the layers may be logically divided. However, the present invention is not limited thereto. Depending on the embodiment, the layers may be stored in a designated area within the memory (115).

[0044] For example, within the framework layer (150), programs designed to target at least one of the hardware abstraction layer (180) and / or the application layer (140) (e.g., a position tracker (171), a space recognizer (172), a gesture tracker (173), and / or an eye tracker (174), a face tracker (175)) may be classified. Programs classified within the framework layer (150) may provide an executable API (application programming interface) based on other programs.

[0045] For example, within the application layer (140), programs designed to target users controlling wearable devices (101) may be classified. As examples of programs classified into the application layer (140), an extended reality (XR) system user interface (UI) and / or an XR application (142) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) classified into the application layer (140) may call an application programming interface (API) to cause execution of functions supported by programs classified into the framework layer (150).

[0046] For example, the wearable device (101) may display one or more visual objects on the display (120) for performing interaction with a user for using a virtual space based on the execution of the XR system UI (141). A visual object may refer to an object that can be placed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide a service for controlling functions available within a virtual space to the user based on the execution of the XR system UI (141).

[0047] Referring to FIG. 1, a lightweight renderer (143) and / or an XR plug-in (144) are illustrated as being included within the XR system UI (141), but are not limited thereto. For example, the XR system UI (141) may cause execution of functions supported by the lightweight renderer (143) and / or the XR plug-in (144) included within the application layer (140).

[0048] For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (143). The lightweight renderer (143) may be referred to as a lightweight renderer pipeline from the perspective of defining a rendering pipeline that allows partial changes. The lightweight renderer (143) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (144). The XR plug-in (144) may be referred to as an open XR native client from the perspective of defining (or configuring) an entire rendering pipeline.

[0049] For example, the wearable device (101) may display a screen representing at least a portion of a virtual space on the display (120) based on the execution of the XR application (142). The XR plug-in (144-1) included in the XR application (142) may be referenced by the XR plug-in (144) of the XR system UI (141). Descriptions of the XR plug-in (144-1) that overlap with the description of the XR plug-in (144) may be omitted. The wearable device (101) may cause the execution of the screen composition manager (151) based on the execution of the XR application (142).

[0050] According to one embodiment, the wearable device (101) may provide a virtual space service based on the execution of the screen composition manager (151). For example, the screen composition manager (151) may include a platform (e.g., an Android platform) for supporting the virtual space service. The wearable device (101) may display the posture of a virtual object representing the user's posture rendered using data acquired through the sensor (130) on the display based on the execution of the screen composition manager (151). The screen composition manager (151) may be referred to as a composition presentation manager (CPM).

[0051] For example, the screen composition manager (151) may include a runtime service (152). As an example, the runtime service (152) may be referred to as an OpenXR runtime module. The wearable device (101) may be used to provide at least one of a pose prediction function, a frame timing function, and / or a spatial input function to a user through the wearable device (101) based on the execution of the runtime service (152). As an example, the wearable device (101) may be used to perform rendering for a virtual space service to a user based on the execution of the runtime service (152). For example, an application (e.g., unity or an OpenXR native application) may be implemented based on the execution of the runtime service (152).

[0052] For example, the screen configuration manager (151) may include a pass-through library (153). Based on the execution of the pass-through library (153), the wearable device (101) may display a screen representing a virtual space on the display (120), while another screen representing a real space acquired through the camera (125) may be superimposed on at least a portion of the screen.

[0053] For example, the screen composition manager (151) may include a renderer (e.g., the renderer (540-1) of FIG. 5). The wearable device (101) may render a screen to be displayed on the display by compositing virtual layers (or virtual nodes) rendered based on sensor data (e.g., sensing data acquired through the camera (125) or sensor (130)) and pass-through layers (or pass-through nodes) acquired through the pass-through library (153) using the screen composition manager (151). The virtual layers may be referred to as virtual nodes and / or virtual surfaces. The wearable device (101) may render each of the virtual layers or all of the virtual layers through the screen composition manager (151).

[0054] For example, the screen configuration manager (151) may include an input manager (154). The wearable device (101) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (170) based on the execution of the input manager (154). The wearable device (101) may initiate execution of at least one of the functions of the wearable device (101) using the acquired data.

[0055] For example, the perception abstract layer (160) can be used for data exchange between the screen configuration manager (151) and the perception service layer (170). From the perspective of being used for data exchange between the screen configuration manager (151) and the perception service layer (170), the perception abstract layer (160) can be referred to as an interface. For example, the perception abstract layer (160) can be referred to as OpenPX and / or PPAL (perception platform abstract layer). The perception abstract layer (160) can be used for a perception client and a perception service.

[0056] According to one embodiment, the recognition service layer (170) may include one or more programs for processing data acquired from a sensor (130) (or a camera (125)). The one or more programs may include at least one of a position tracker (171), a space recognizer (172), a gesture tracker (173), an eye tracker (174), and / or a face tracker (175). The type and / or number of the one or more programs included in the recognition service layer (170) are not limited to those illustrated in FIG. 1.

[0057] For example, the wearable device (101) can identify the pose of the wearable device (101) using the sensor (130) based on the execution of the position tracker (171). The wearable device (101) can identify the 6 degrees of freedom pose (6 DOF pose) of the wearable device (101) using data acquired using the camera (125) and the IMU based on the execution of the position tracker (171). The position tracker (171) may be referred to as a head tracking (HeT) module.

[0058] For example, the wearable device (101) may be used to construct a three-dimensional virtual space surrounding the wearable device (101) (or a user of the wearable device (101)) based on the execution of the space recognizer (172). The wearable device (101) may reconstruct the three-dimensional surroundings of the wearable device (101) using data acquired using the camera (125) based on the execution of the space recognizer (172). The wearable device (101) may identify at least one of a plane, a slope, or stairs based on the three-dimensionally reconstructed surroundings of the wearable device (101) based on the execution of the space recognizer (172). The space recognizer (172) may be referred to as a scene understanding (SU) module.

[0059] For example, the wearable device (101) may be used to identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (101) based on the execution of the gesture tracker (173). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user using data acquired from a sensor (130) based on the execution of the gesture tracker (173). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using a camera (125) based on the execution of the gesture tracker (173). The gesture tracker (173) may be referred to as a hand tracking (HaT) module and / or a gesture tracking module.

[0060] For example, the wearable device (101) may identify (or track) eye movements of a user of the wearable device (101) based on the execution of the gaze tracker (174). As an example, the wearable device (101) may identify eye movements of the user using data acquired from at least one sensor based on the execution of the gaze tracker (174). As an example, the wearable device (101) may identify eye movements of the user based on data acquired using a camera (125) and / or an infrared light emitting diode (IR LED) based on the execution of the gaze tracker (174). The gaze tracker (174) may be referred to as an eye tracking (ET) module and / or a gaze tracking module.

[0061] For example, the recognition service layer (170) of the wearable device (101) may further include a face tracker (175) for tracking the user's face. For example, the wearable device (101) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (175). The wearable device (101) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (175). As an example, the wearable device (101) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a camera based on the execution of the face tracker (175).

[0062] FIG. 2 is a diagram illustrating a situation in which an electronic device identifies a user's gaze according to one embodiment.

[0063] The wearable device (101) of FIG. 2 may correspond to the wearable device (101) of FIG. 1. FIG. 2 may be described with reference to FIG. 1. In the description of FIG. 2, the operations exemplified as being performed by the wearable device (101) may be performed by instructions stored in the memory (115) being individually or collectively executed by at least one processor (110110). However, the present invention is not limited thereto.

[0064] Referring to FIG. 2, the wearable device (101) may include a display (120) that displays content (230) and a sensor (130). In one embodiment, the display (120) may correspond to the display module (1860) of FIG. 18. In one embodiment, the sensor (130) may correspond to the camera module (1880) and / or the sensor module (1876) of FIG. 18. For example, the sensor (130) may be arranged to face the face of the user (200) when the user (200) looks at the display (120) of the wearable device (101) so as to capture at least a portion of the face of the user (200) (e.g., eyes (221, 215)).

[0065] In one embodiment, a user (200) can look at a wearable device (101) (or a display (120) of the wearable device (101)) through the eyes (211, 215). The user (200) can view content (230) displayed through the display (120). In one embodiment, the user (200) can view content (230) within a field of view (FOV) (260) of the display (120) of the wearable device (101) through the eyes (211, 215). For example, the content (230) can be displayed as a stereoscopic image on the display (120). The stereoscopic image can be an image that takes into account the binocular disparity of the user (200). A stereoscopic image may be an image for providing a three-dimensional (3D) sense of space to a user (200). For example, the content (230) may include visual elements (or visual objects) (e.g., text, images, videos, or a combination thereof). In one embodiment, the wearable device (101) may provide a convenience function when the user (200) views the content (230). For example, the wearable device (101) may provide a convenience function corresponding to the reading pattern of the user (200) (or the eye movement pattern of the user (200)) when the user (200) views the content (230).

[0066] In one embodiment, the wearable device (101) may perform an operation of identifying the gaze of the user (200) (hereinafter, the operation of identifying the gaze), an operation of identifying a reading pattern corresponding to the gaze (or the movement of the gaze) (hereinafter, the operation of identifying the reading pattern), and an operation of providing an additional function (or a reading assistance function) corresponding to the reading pattern (hereinafter, the operation of providing the additional function). In one embodiment, the operations performed by the wearable device (101) may be performed based on the processor (110) executing instructions stored in the memory (130). In one embodiment, the operation of identifying the gaze and the operation of identifying the reading pattern may be defined by instructions included in a program (e.g., an eye tracker (174)) included in the framework layer (150). In one embodiment, the operation of providing the additional function may be defined by instructions included in a program (e.g., an XR application (142)) included in the application layer (140). In one embodiment, the providing operation of the add-on may be defined by the same application as the application providing the content (230), but is not limited thereto.

[0067] Hereinafter, with reference to FIG. 2, the identification operation of the gaze of the wearable device (101), the identification operation of the reading pattern, and the provision operation of the additional function are described.

[0068] Below, the gaze identification operation of the wearable device (101) is described.

[0069] In one embodiment, the wearable device (101) may obtain a gaze identification request. For example, the wearable device (101) may obtain a gaze identification request from an XR application (142) displaying content (230). For example, a gaze tracker (174) running through a processor (110) may obtain a gaze identification request from an XR application (142) running through the processor (110) displaying content (230). However, the present invention is not limited thereto.

[0070] In one embodiment, the wearable device (101) can obtain spatial information (e.g., a depth map) about a user facing (or looking at) (or viewing) (or facing) the display (120) of the wearable device (101) through a sensor (130) (e.g., a camera module (1880) of FIG. 18 and / or a sensor module (1876) of FIG. 18). In one embodiment, the wearable device (101) can obtain spatial information (e.g., a depth map) about a user based on a time-of-flight (ToF) between the wearable device (101) and the user (200) measured (or identified) through the sensor (130). In one embodiment, the wearable device (101) can identify the distance between the wearable device (101) and the user (200) (or the eyes (211, 215) of the user (200)) based on data (e.g., depth map, and / or ToF) acquired through the sensor (130).

[0071] In one embodiment, the wearable device (101) can obtain an image including the eyes (211, 215) of the user (200) through the sensor (130). In one embodiment, the wearable device (101) can identify the eyes (211, 215) (or the positions of the pupils of the eyes (211, 215)) included in the image obtained through the sensor (130). In one embodiment, the wearable device (101) can identify the gaze of the user (200) based on the identified eyes (211, 215) (or the positions of the pupils of the eyes (211, 215)).

[0072] In one embodiment, the wearable device (101) can identify the direction of the user's (200) gaze (or the area (220, 225) on the display (120) toward which the gaze is directed) based on the position of the pupils of the eyes (211, 215) and the distance from the user's (200) eyes (211, 215).

[0073] In one embodiment, the wearable device (101) can identify a region (220, 225) on the display (120) toward which the user's (200) gaze is directed (or where the gaze is fixed) (or a change in the region of interest (ROI) (or viewing area)) based on the eyes (211, 215) (or the positions of the pupils of the eyes (211, 215)) identified in successive images.

[0074] Below, the identification operation of the reading pattern of the wearable device (101) is described.

[0075] In one embodiment, the wearable device (101) can identify characteristics of the gaze (or gaze movement) of the user (200) through successive images. In one embodiment, the characteristics of the gaze (or gaze movement) can include fixation and saccade of the gaze.

[0076] In one embodiment, the wearable device (101) can identify that the gaze of the user (200) is fixed (or stopped) through images continuously acquired for a specified length of time (or time window). For example, the wearable device (101) can identify (or measure) the time (or gaze fixation time) (or dwell time) for which the gaze of the user (200) is fixed (or stopped) through consecutive images. For example, the gaze fixation can refer to a situation in which the eyes (211, 215) are focused on a specific location for a predetermined period of time (e.g., 200 milliseconds) or longer.

[0077] In one embodiment, the wearable device (101) can identify an area (220, 225) on the content (230) on which the user's (200) gaze is fixed (or directed) (or located). In one embodiment, the wearable device (101) can identify a type of object (e.g., text, image, or video) or content of an object (e.g., meaning or content of a word (or sentence), content of an image, or content of a video) displayed in an area (220, 225) on the content (230) on which the user's (200) gaze is fixed (or directed) (or located). However, the present invention is not limited thereto.

[0078] In one embodiment, the wearable device (101) can identify a gaze leap (or gaze movement) of the user (200) through images acquired continuously over a specified period of time (or time window). For example, the wearable device (101) can identify a distance, a direction, and / or a time of leap of the user's (200) gaze through continuous images. For example, a gaze leap can refer to a situation in which the eyes (211, 215) move rapidly between two adjacent gaze fixations.

[0079] In one embodiment, the wearable device (101) can identify when the gaze of the user (200) jumps (or moves) between two regions (220, 225) on the content (230) where the gaze is fixed (or directed) (or positioned). For example, the wearable device (101) can identify when the gaze of the user (200) jumps (or moves) from region (220) to region (225). Before the gaze jumps from region (220) to region (225), the gaze may remain fixed on region (220) for some length of time. After the gaze jumps from region (220) to region (225), the gaze may remain fixed on region (225) for at least some length of time. In one embodiment, at least some of the time duration may be, but is not limited to, a predetermined time (e.g., 200 milliseconds) for which fixation of the gaze is determined.

[0080] In one embodiment, the wearable device (101) can identify a pattern (or, reading pattern) (or, movement patterns) in which the user (200) views content (230) through the movement of the user's (200) gaze. In one embodiment, the wearable device (101) can identify at least one pattern in which the user (200) looks at content (230) among a plurality of patterns (or, reading patterns) (or, movement patterns). In one embodiment, the wearable device (101) can identify at least one pattern in which the user (200) looks at content (230) through a comparison between each of the plurality of patterns (or, reading patterns) (or, movement patterns) and the movement of the user's (200) gaze. At least one pattern in which the user (200) looks at content (230) can be identified. For example, the multiple patterns may include a reading pattern, a scanning pattern, a cross-reading pattern, or a free-viewing pattern. For example, the multiple patterns may be defined differently in terms of the characteristics of fixation (e.g., fixation duration, type of object fixed, and / or content of the object) and / or the characteristics of gaze hops (e.g., distance of gaze hop, direction of hop, and / or hop time).

[0081] In one embodiment, the wearable device (101) can identify at least one pattern corresponding to the movement of the user's (200) gaze based on the characteristics of the user's (200) gaze leap and / or the characteristics of the gaze fixation.

[0082] In one embodiment, the wearable device (101) can determine whether a specified number of consecutive fixations (e.g., two) and gaze hops between the specified number of consecutive fixations (e.g., two) correspond to a reading pattern. In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a reading pattern if the direction of the gaze hop is in the writing direction of the content (230) (e.g., horizontal direction (or rightward direction)) and / or the line break direction (e.g., vertical direction (or downward direction)). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a reading pattern if the gaze fixation times of the gaze fixations are greater than or equal to a reference fixation time. However, the present invention is not limited thereto. In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a reading pattern if the gaze spurt is continuous in the writing direction of the content (230) (e.g., horizontal direction). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a reading pattern if the gaze spurt is not repeated in the line break direction of the content (230) (e.g., vertical direction). The number of gaze fixations for determining the reading pattern exemplified above is only an example. Depending on the embodiment, the reading pattern can be determined based on two or more different numbers of gaze fixations and the resulting gaze spurts.

[0083] In one embodiment, the wearable device (101) can determine whether a specified number of consecutive fixations (e.g., two) and gaze hops between the specified number of consecutive fixations (e.g., two) correspond to a scanning pattern. In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a scanning pattern if the distance of the gaze hops is greater than or equal to a reference hop distance (e.g., a distance required for a line break, or a distance in which a specific number of words are written). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a scanning pattern if the direction of the gaze hops is repeated in the direction of a line break (e.g., a vertical direction). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a glance pattern if the gaze fixation time is less than a reference fixation time. The number of gaze fixations used to determine the glance pattern exemplified above is merely an example. In some embodiments, the glance pattern can be determined based on a number of gaze fixations other than three and the resulting gaze hops.

[0084] In one embodiment, the wearable device (101) can determine whether a specified number of consecutive fixations (e.g., four) and at least two gaze hops between the specified number of consecutive fixations (e.g., four) correspond to a perusal pattern. In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a cross-reading pattern if two or more of the at least three gaze hops are hops between multiple regions of the content (230). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a cross-reading pattern if the gaze repeatedly hops between one region of the content (230) and multiple other regions. If the gaze repeatedly jumps in a first direction (e.g., a line break direction (e.g., vertical direction (or downward direction)) or a writing direction (e.g., horizontal direction (or rightward direction))) and a second direction opposite to the first direction (e.g., a direction opposite to the line break direction (e.g., upward direction) or a direction opposite to the writing direction (e.g., leftward direction)), the gaze movement can be identified (or determined) as corresponding to a cross-reading pattern. In one embodiment, the wearable device (101) can identify (or determine) the gaze movement as corresponding to a cross-reading pattern if the gaze jumps are repeated between multiple regions of the content (230) and the gaze fixation time of the gaze fixation is greater than or equal to a reference fixation time. The number of gaze fixations for determining the cross-reading pattern exemplified above is merely an example. Depending on the embodiment, the cross-reading pattern can be determined based on four or more different numbers of gaze fixations and the resulting gaze jumps.

[0085] In one embodiment, the wearable device (101) can identify a degree of association (or similarity) between the gaze movement and each of the plurality of patterns based on characteristics of a plurality of gaze fixations representing gaze movement over a specified length of time (or time window) and / or characteristics of a plurality of gaze spurts. For example, the wearable device (101) can determine whether a combination of a minimum number of gaze fixations that allow interpretation of a pattern and a plurality of gaze spurts between the plurality of gaze fixations corresponds to each of the plurality of patterns.

[0086] For example, if a reading pattern is determined based on two gaze fixations and one gaze spurt, the wearable device (101) can determine whether each of nine gaze movements combined based on ten gaze fixations and nine gaze spurts corresponds to the reading pattern. For example, the wearable device (101) can identify the ratio of movements corresponding to the reading pattern among the nine gaze movements as a combination stage (or similarity) for the reading pattern.

[0087] For example, if a glance pattern is determined based on three gaze fixations and two gaze hops, the wearable device (101) can determine whether each of eight gaze movements combined based on ten gaze fixations and nine gaze hops corresponds to the glance pattern. For example, the wearable device (101) can identify the proportion of movements corresponding to the glance pattern among the eight gaze movements as a combination step (or similarity) for the glance pattern.

[0088] For example, if a cross-reading pattern is determined based on four gaze fixations and three gaze hops, the wearable device (101) can determine whether each of the seven gaze movements combined based on ten gaze fixations and nine gaze hops corresponds to the cross-reading pattern. For example, the wearable device (101) can identify the proportion of movements corresponding to the cross-reading pattern among the seven gaze movements as a combination step (or similarity) for the cross-reading pattern.

[0089] Identifying a combination phase (or similarity) based on the rate of movement is only an example. For example, the wearable device (101) can identify the similarity as the distance (or cosine similarity) between the values ​​(or vector values) represented by the characteristics of a plurality of gaze fixations representing the movement of the gaze during a specified time period (or time window) and / or the characteristics of a plurality of gaze spurts and the values ​​(or vector values) represented by the characteristics of the gaze fixations and gaze spurts set for each of the plurality of patterns. However, the present invention is not limited thereto. In one embodiment, the wearable device (101) can identify the probability for each of the plurality of patterns as the similarity by inputting the values ​​(or vector values) represented by the characteristics of a plurality of gaze fixations representing the movement of the gaze during a specified time period (or time window) and / or the characteristics of a plurality of gaze spurts into an AI (artificial intelligence) model (e.g., a model employing a convolutional neural network (CNN)) for classifying classes (or patterns).

[0090] In one embodiment, the wearable device (101) can identify the state of each of the plurality of patterns based on the similarity of each of the plurality of patterns. In one embodiment, the state of each of the plurality of patterns can include idle, ready, active, or transition. In one embodiment, the wearable device (101) can identify a pattern among the plurality of patterns whose state is other than idle (i.e., ready, active, or transition) as a pattern corresponding to the movement of the gaze.

[0091] In one embodiment, after the state of the pattern changes from transition to idle, during a time period during which the similarity between the pattern and the gaze movement remains below a first criterion similarity, the pattern may be determined to be in an idle state. In one embodiment, from a time period during which the similarity between the pattern and the gaze movement rises above a first criterion similarity while the state of the pattern is idle, during a time period during a preparation time, the pattern may be determined to be in a ready state. In one embodiment, from a time period after the preparation time when the state of the pattern is ready, during a time period during which the similarity is above a second criterion similarity, the pattern may be determined to be in an active state. In one embodiment, during a time period during a transition time, from a time period during which the similarity between the pattern and the gaze movement falls below a second criterion similarity while the state of the pattern is active, the pattern may be determined to be in a transition state. In one embodiment, if the similarity of the pattern does not rise above the first criterion similarity during the transition time, the pattern may be determined to be in an idle state after the transition time.

[0092] Below, the operation of providing additional functions of the wearable device (101) is described.

[0093] In one embodiment, the wearable device (101) may obtain information about the characteristics of eye movement and / or at least one pattern corresponding to eye movement. For example, the wearable device (101) may obtain data for identifying the characteristics of eye movement and / or at least one pattern corresponding to eye movement from the middleware (144). In one embodiment, the application (146) that generated the gaze identification request may provide additional functions (or determine whether to provide additional functions) based on the data from the middleware (144). However, the present invention is not limited thereto.

[0094] In one embodiment, the wearable device (101) can identify an additional function (or a reading assistance function) corresponding to a pattern corresponding to eye movement. For example, if there is only one pattern corresponding to eye movement, the wearable device (101) can identify one additional function.

[0095] In one embodiment, the wearable device (101) may identify auto-scroll as an additional function (or a reading assistance function) if the pattern corresponding to the gaze movement is a perusing pattern. In one embodiment, the wearable device (101) may identify a summary of the content (230) as an additional function if the pattern corresponding to the gaze movement is a skimming pattern. In one embodiment, the wearable device (101) may identify provision of a visual cue and / or provision of an auxiliary screen (e.g., a screen representing one of the two areas (220, 225) to which the gaze leaps) as an additional function if the pattern corresponding to the gaze movement is a cross-reading pattern. In one embodiment, the wearable device (101) may identify position adjustment of additional content (e.g., subtitles) as an additional function if the pattern corresponding to the gaze movement is a free-viewing pattern.

[0096] In one embodiment, if there are two or more patterns corresponding to eye movement, the wearable device (101) can identify two or more additional functions. However, this is not limited thereto. For example, if there are two or more patterns corresponding to eye movement, the wearable device (101) can identify an additional function corresponding to a combination of the two or more patterns corresponding to eye movement. For example, if there are two or more patterns corresponding to eye movement, the wearable device (101) can identify an additional function selected from among the two or more additional functions corresponding to the two or more patterns corresponding to eye movement.

[0097] For example, if the pattern corresponding to the movement of the gaze is a perusal pattern and a cross-reading pattern, the wearable device (101) may identify automatic scrolling and provision of visual cues (or, auxiliary screen) as additional functions. For example, while scrolling content (230) through automatic scrolling, the wearable device (101) may display an object displayed through one of the two regions (220, 225) to which the gaze jumps (e.g., region (220)) around the other region (e.g., region (225)) (e.g., region (220)) to which the gaze jumps (e.g., region (220)). In one embodiment, one region (e.g., region (220)) may be a region in which a perusal pattern is identified. In one embodiment, the other region (e.g., region (225)) may be a region in which a perusal pattern is not identified. However, the present invention is not limited thereto. One region (e.g., region (220)) may be a region containing text (or a region not containing an image or video). Another region (e.g., region (225)) may be a region containing an image or video.

[0098] In one embodiment, the wearable device (101) may provide (or perform) at least one identified additional function.

[0099] As described above, the wearable device (101) can provide various user-friendly functions through classification of reading patterns. Furthermore, when multiple reading patterns are activated simultaneously (or sequentially), the wearable device (101) can enhance user convenience by providing (or performing) at least one function corresponding to multiple reading patterns.

[0100] FIG. 3 is a diagram illustrating the movement of a user's gaze identified by an electronic device according to one embodiment.

[0101] Figure 3 can be explained with reference to Figures 1 and 2.

[0102] Referring to FIG. 3, the wearable device (101) can identify the movement of the gaze of a user (200) looking at content (300) including text and images. In one embodiment, the wearable device (101) can identify an area (220, 225) on the display (120) to which the gaze of the user (200) is directed (or where the gaze is fixed) (or a change in an area of ​​interest (or an area of ​​viewing)) based on the positions of the pupils of the eyes (211, 215) identified in successive images.

[0103] In one embodiment, the wearable device (101) can identify characteristics of the gaze (or gaze movement) of the user (200) through successive images. In one embodiment, the characteristics of the gaze (or gaze movement) can include fixation and saccade of the gaze.

[0104] For example, the wearable device (101) can identify gaze fixations (1, 2, 3, 4) on some words (e.g., automobile, motor, vehicle, and wheels) among a plurality of words (e.g., an, automobile, is, a, motor, vehicle, with, and wheels) included in a text. For example, the wearable device (101) can identify gaze fixations (5, 6) on an image. For example, the wearable device (101) can identify (or measure) characteristics (e.g., gaze fixation time, type of object on which gaze is fixed, and / or content of the object) of each of the gaze fixations (1, 2, 3, 4, 5, 6).

[0105] For example, the wearable device (101) can identify gaze hops (311, 313, 315, 317, 319) between gaze fixations (1, 2, 3, 4, 5, 6). For example, the wearable device (101) can identify gaze hops (311, 313, 315) from one text to another. For example, the wearable device (101) can identify gaze hops (317) from a text to an image. For example, the wearable device (101) can identify gaze hops (319) to change the object being viewed within an image. For example, the wearable device (101) can identify (or measure) characteristics of each of the gaze hops (311, 313, 315, 317, 319) (e.g., distance the gaze hopped, direction of the hop, and / or time of the hop).

[0106] In one embodiment, the wearable device (101) can identify a pattern in which a user (200) views content (300) based on eye movement (or, eye fixations (1, 2, 3, 4, 5, 6) and eye sprints (311, 313, 315, 317, 319)). In one embodiment, the wearable device (101) can identify a pattern in which a user (200) views content (300) based on characteristics of eye movement (or, characteristics of eye fixations (1, 2, 3, 4, 5, 6) and characteristics of eye sprints (311, 313, 315, 317, 319)). Hereinafter, with reference to FIGS. 4A to 4C, the operation of the wearable device (101) to identify patterns corresponding to eye movement is described.

[0107] FIG. 4A is a diagram illustrating eye movement corresponding to a reading pattern according to one embodiment.

[0108] Figure 4a can be explained with reference to Figures 1, 2, and 3.

[0109] Referring to FIG. 4A, the wearable device (101) can identify the movement of the user's (200) gaze while looking at content (410) (e.g., a document) that is aligned according to a writing direction. In one embodiment, the wearable device (101) can identify an area on the display (120) toward which the user's (200) gaze is directed (or a change in an area of ​​interest (or an area of ​​viewing)) based on the eyes (211, 215) (or the positions of the pupils of the eyes (211, 215)) identified in successive images. In one embodiment, the writing direction can be a direction in which lines are aligned vertically (or downwardly) and characters within a line are aligned horizontally (or toward the right).

[0110] In one embodiment, the wearable device (101) can identify gaze fixations (1 to 21) based on the gaze (or gaze movement) of the user (200) through successive images. For example, the wearable device (101) can identify (or measure) characteristics of each of the gaze fixations (1 to 21) (e.g., gaze fixation time, type of object on which the gaze is fixed, and / or content of the object).

[0111] For example, the wearable device (101) can identify gaze hops between gaze fixations (1 to 21). For example, the wearable device (101) can identify (or measure) characteristics of each gaze hop (e.g., distance the gaze hopped, direction of the hop, and / or time of the hop). In one embodiment, the wearable device (101) can identify a pattern corresponding to gaze movement based on characteristics of two or more gaze fixations and characteristics of at least one gaze hop between two or more gaze fixations.

[0112] In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a reading pattern if the direction of the gaze hop corresponds to a writing direction (e.g., horizontal direction (or rightward direction)) and / or a line break direction (e.g., vertical direction (or downward direction)) of the content (410). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a reading pattern if the direction of the gaze hop is continuous in a direction (e.g., horizontal direction) in which characters of the content (230) are listed (or aligned). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a reading pattern if the direction of the gaze hop is not repeated in a line break direction (e.g., vertical direction) of the content (230). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a reading pattern when the gaze fixation time of the gaze fixation is longer than the reference fixation time.

[0113] For example, based on the fact that the gaze fixation time of the gaze fixations (1 to 4) is longer than the reference fixation time and the gaze leap between two neighboring gaze fixations (e.g., gaze fixations (1, 2), gaze fixations (2, 3), gaze fixations (3, 4)) is in the writing direction, the wearable device (101) can identify (or determine) that each of the gaze movements for the gaze fixations (1 to 4) corresponds to a reading pattern.

[0114] For example, based on the non-consecutiveness of gaze hops in the direction of line breaks (e.g., hops between fixations (4, 5), hops between fixations (9, 10), hops between fixations (13, 14), or hops between fixations (17, 18)), each of the gaze movements can be identified (or determined) as corresponding to a reading pattern. In one embodiment, the non-consecutiveness of gaze hops in the direction of line breaks can indicate that two or more consecutive gaze hops do not hop in the direction of line breaks. For example, the non-consecutiveness of gaze hops in the direction of line breaks can indicate that gaze hops in the direction of writing are included between gaze hops in the direction of line breaks. In some embodiments, the non-consecutiveness of gaze hops in the direction of line breaks can be described as a periodic repetition of gaze hops in the direction of line breaks.

[0115] In one embodiment, the wearable device (101) can identify a combination phase (or similarity) of a gaze movement and a reading pattern based on a plurality of gaze fixations and / or a plurality of gaze hops representing gaze movements during a specified length of time (or time window). For example, if a reading pattern is determined based on two consecutive gaze fixations and one gaze hop, the wearable device (101) can identify 20 gaze movements based on 21 gaze fixations and 20 gaze hops. For example, the wearable device (101) can identify a proportion of movements corresponding to the reading pattern among the 20 gaze movements as a combination phase (or similarity) to the reading pattern.

[0116] FIG. 4b is a diagram illustrating gaze movements corresponding to a scanning pattern according to one embodiment.

[0117] Figure 4b can be explained with reference to Figures 1, 2, and 3.

[0118] Referring to FIG. 4B, the wearable device (101) can identify the movement of the user's (200) gaze while looking at content (420) (e.g., a document) that is aligned according to a writing direction. In one embodiment, the wearable device (101) can identify an area on the display (120) toward which the user's (200) gaze is directed (or a change in an area of ​​interest (or an area of ​​viewing)) based on the eyes (211, 215) (or the positions of the pupils of the eyes (211, 215)) identified in successive images. In one embodiment, the writing direction can be a direction in which lines are aligned vertically (or downwardly) and characters within a line are aligned horizontally (or toward the right).

[0119] In one embodiment, the wearable device (101) can identify gaze fixations (1 to 9) based on the gaze (or gaze movement) of the user (200) through successive images. For example, the wearable device (101) can identify (or measure) characteristics of each of the gaze fixations (1 to 9) (e.g., gaze fixation time, type of object on which the gaze is fixed, and / or content of the object).

[0120] For example, the wearable device (101) can identify gaze hops between gaze fixations (1 to 9). For example, the wearable device (101) can identify (or measure) characteristics of each gaze hop (e.g., distance the gaze hopped, direction of the hop, and / or time of the hop). In one embodiment, the wearable device (101) can identify a pattern corresponding to gaze movement based on characteristics of two or more gaze fixations and characteristics of at least one gaze hop between two or more gaze fixations.

[0121] In one embodiment, the wearable device (101) can determine whether a specified number of consecutive fixations (e.g., two) and gaze hops between the specified number of consecutive fixations (e.g., two) correspond to a scanning pattern. In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a scanning pattern if the distance of the gaze hops is greater than or equal to a reference hop distance (e.g., a distance required for a line break, or a distance in which a specific number of words are written). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a scanning pattern if the direction of the gaze hops is repeated in the direction of a line break (e.g., a vertical direction). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a scanning pattern when the gaze fixation time of the gaze fixation is less than a reference fixation time.

[0122] For example, based on the distance of the gaze hop between the fixations (1 and 2) being greater than or equal to a reference hop distance (e.g., a distance over which a certain number of words are written), the wearable device (101) can identify (or determine) that the gaze movement for the fixations (1 and 2) corresponds to a scanning pattern. For example, based on the distance of the gaze hop between the fixations (3 and 4) being greater than or equal to a reference hop distance (e.g., a distance required for a line break), the wearable device (101) can identify (or determine) that the gaze movement for the fixations (3 and 4) corresponds to a scanning pattern.

[0123] For example, based on the fact that consecutive gaze hops (e.g., gaze hops between gaze fixations (6 to 9)) have a line-breaking direction, the wearable device (101) can identify (or determine) that the gaze movement for gaze fixations (6 to 9) corresponds to a scanning pattern.

[0124] In one embodiment, the wearable device (101) can identify a binding phase (or similarity) between a gaze movement and a glance pattern based on a plurality of gaze fixations and / or a plurality of gaze hops representing gaze movements over a specified length of time (or time window). For example, if a glance pattern is determined based on two consecutive gaze fixations and one gaze hop, the wearable device (101) can identify eight gaze movements based on nine gaze fixations and eight gaze hops. For example, the wearable device (101) can identify a proportion of the eight gaze movements that correspond to the glance pattern as a binding phase (or similarity) to the glance pattern. For example, if a glance pattern is determined based on three consecutive gaze fixations and two gaze hops, the wearable device (101) can identify seven gaze movements based on nine gaze fixations and eight gaze hops. For example, the wearable device (101) can identify the proportion of movements corresponding to the glance pattern among the seven gaze movements as a combination step (or similarity) for the glance pattern.

[0125] FIG. 4c is a diagram illustrating gaze movements corresponding to a cross-reading pattern according to one embodiment.

[0126] Figure 4c can be explained with reference to Figures 1, 2, and 3.

[0127] Referring to FIG. 4C, the wearable device (101) can identify the movement of the user's (200) gaze while looking at content (430) (e.g., a document) aligned according to the writing direction. In one embodiment, the wearable device (101) can identify an area (440, 450) on the display (120) toward which the user's (200) gaze is directed (or on which the gaze is fixed) (or a change in an area of ​​interest (or an area of ​​viewing)) based on the eyes (211, 215) (or the positions of the pupils of the eyes (211, 215)) identified in successive images. For example, the area (440) can be an area including text (or an area not including an image or video). The area (450) can be an area including an image or video.

[0128] In one embodiment, the wearable device (101) can identify gaze fixations (1 to 13) based on the gaze (or gaze movement) of the user (200) through successive images. For example, the wearable device (101) can identify (or measure) characteristics of each of the gaze fixations (1 to 13) (e.g., gaze fixation time, type of object on which the gaze is fixed, and / or content of the object).

[0129] For example, the wearable device (101) can identify gaze hops between gaze fixations (1 to 13). For example, the wearable device (101) can identify (or measure) characteristics of each gaze hop (e.g., distance the gaze hopped, direction of the hop, and / or time of the hop). In one embodiment, the wearable device (101) can identify a pattern corresponding to gaze movement based on characteristics of two or more gaze fixations and characteristics of at least one gaze hop between two or more gaze fixations.

[0130] In one embodiment, the wearable device (101) can determine whether a specified number of consecutive fixations (e.g., four) and at least two gaze hops between the specified number of consecutive fixations (e.g., four) correspond to a perusal pattern. In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a cross-reading pattern if two or more of the at least three gaze hops are hops between multiple regions of the content (230). In one embodiment, the wearable device (101) can identify (or determine) that the gaze movement corresponds to a cross-reading pattern if the gaze repeatedly hops between one region of the content (230) and multiple other regions. If the gaze repeatedly jumps in a first direction (e.g., a line break direction (e.g., a vertical direction (or downward direction)) or a writing direction (e.g., a horizontal direction (or rightward direction))) and a second direction opposite to the first direction (e.g., a direction opposite to the line break direction (e.g., an upward direction) or a direction opposite to the writing direction (e.g., a leftward direction)), the gaze movement can be identified (or determined) as corresponding to a cross-reading pattern. In one embodiment, the wearable device (101) can identify (or determine) the gaze movement as corresponding to a cross-reading pattern if the gaze jumps are repeated between multiple areas of the content (230) and the gaze fixation time of the gaze fixation is longer than a reference fixation time.

[0131] For example, based on the fact that the gaze fixation time of the gaze fixations (1 to 4) is longer than the reference fixation time and the gaze leap between two neighboring gaze fixations (e.g., gaze fixations (1, 2), gaze fixations (2, 3), gaze fixations (3, 4)) is in the writing direction, the wearable device (101) can identify (or determine) that each of the gaze movements for the gaze fixations (1 to 4) corresponds to a reading pattern.

[0132] For example, based on the fact that consecutive gaze hops (e.g., gaze hops between fixations (4 to 8)) hop between two regions (440, 450), the wearable device (101) can identify (or determine) that the gaze movement for the fixations (4 to 8) corresponds to a cross-reading pattern. For example, based on the fact that consecutive gaze hops (e.g., gaze hops between fixations (9 to 12)) hop between two regions (440, 450), the wearable device (101) can identify (or determine) that the gaze movement for the fixations (9 to 12) corresponds to a cross-reading pattern.

[0133] In one embodiment, the wearable device (101) can identify a binding stage (or similarity) between a gaze movement and a cross-reading pattern based on a plurality of gaze fixations and / or a plurality of gaze hops representing gaze movements during a specified length of time (or time window). For example, if a cross-reading pattern is determined based on four consecutive gaze fixations and three gaze hops, the wearable device (101) can identify ten gaze movements based on thirteen gaze fixations and twelve gaze hops. For example, the wearable device (101) can identify a proportion of movements corresponding to a cross-reading pattern among the ten gaze movements as a binding stage (or similarity) to the cross-reading pattern. However, the present invention is not limited thereto. For example, if a cross-reading pattern is determined based on five consecutive gaze fixations and four gaze hops, the wearable device (101) can identify nine gaze movements based on 13 gaze fixations and 12 gaze hops. For example, the wearable device (101) can identify the ratio of movements corresponding to the cross-reading pattern among the nine gaze movements as a combination step (or similarity) for the cross-reading pattern.

[0134] FIG. 5 is a diagram illustrating changes between states of a reading pattern according to one embodiment. FIG. 6 is a diagram illustrating changes in the similarity between gaze movement and a reading pattern according to one embodiment.

[0135] FIGS. 5 and 6 can be described with reference to FIGS. 1, 2, 3, 4a, 4b, and 4c.

[0136] In one embodiment, the wearable device (101) can identify the state of each of the plurality of patterns based on the similarity of each of the plurality of patterns. In one embodiment, referring to the state transition diagram (500) of FIG. 5, the state of each of the plurality of patterns can include idle (510), ready (520), active (530), or transition (540). In one embodiment, the wearable device (101) can identify a pattern among the plurality of patterns whose state is other than idle (510) (i.e., ready (520), active (530), or transition (540)) as a pattern corresponding to the movement of the gaze.

[0137] Referring to the graph (600) of FIG. 6, the wearable device (101) can identify the state of the pattern based on the similarity (610). For example, during a time interval (e.g., a time interval before t1) in which the similarity (610) between the pattern and the movement of the gaze remains below a first reference similarity (620), the pattern can be determined to be idle (510). In one embodiment, while the pattern is idle (510), from a time point (t1) at which the similarity (610) rises above the first reference similarity (620), during a time interval (e.g., from t1 to t2) during the preparation time, the pattern can be determined to be ready (520). In one embodiment, from a time point (t2) when the pattern is ready (520), during a time interval after the preparation time when the similarity (610) is greater than or equal to a second reference similarity (630), the pattern may be determined to be active (530). In one embodiment, from a time point (t3) when the similarity (610) falls below the second reference similarity (630) while the pattern is active (630), during a time interval (e.g., from t3 to t4) during a transition time, the pattern may be determined to be transitional (530). In one embodiment, if the similarity (610) does not rise above the first reference similarity (620) during the transition time (e.g., from t3 to t4), the pattern may be determined to be idle (510) after the transition time (i.e., after t4).

[0138] In one embodiment, the wearable device (101) may set different first reference similarities (620) and second reference similarities (630) for each of a plurality of patterns. For example, the wearable device (101) may set different first reference similarities (620) and second reference similarities (630) for the cross-reading pattern to be lower than the different first reference similarities (620) and second reference similarities (630) for the reading pattern. However, the present invention is not limited thereto.

[0139] FIG. 7 is a diagram illustrating a change in state over time of each of a plurality of reading patterns according to one embodiment.

[0140] FIG. 7 can be explained with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5, and FIG. 6.

[0141] Figure 7 may represent an example (700) in which one pattern is activated.

[0142] In one embodiment, the wearable device (101) can identify eye movement of the user (200) while displaying content (230) (or in response to a request from an application (146) displaying content (230). In one embodiment, the wearable device (101) can identify eye movement of the user (200) based on the eyes (211, 215) (or the positions of the pupils of the eyes (211, 215)) identified in images continuously acquired through the sensor (130). In one embodiment, the eye movement can include a combination of fixation and / or saccade. In one embodiment, the eye movement can include a combination of fixation and / or saccade identified over a specified time interval. In one embodiment, the gaze movement may include a specified number of gaze fixations and / or leaps between the specified number of gaze fixations. In one embodiment, the specified number may be set for each of a plurality of patterns. For example, the specified number may be a number of gaze fixations and / or leaps between the specified number of gaze fixations to determine whether there is a correspondence with the pattern.

[0143] In one embodiment, the wearable device (101) can identify a pattern in which a user (200) views content (230) based on eye movement (or a combination of eye fixations and eye hops). In one embodiment, the wearable device (101) can identify at least one pattern corresponding to a characteristic of eye movement among a plurality of patterns based on eye movement.

[0144] In one embodiment, the wearable device (101) can identify the degree of association (or similarity) between gaze movements and each of a plurality of patterns. For example, the wearable device (101) can determine whether a combination of a minimum number of gaze fixations that allow interpretation of a pattern and a plurality of gaze hops between the plurality of gaze fixations corresponds to each of the plurality of patterns.

[0145] Referring to example (700) of FIG. 7, it can be determined that a combination of multiple gaze fixations and multiple gaze hops between the multiple gaze fixations does not correspond to a cross-reading pattern, a glance pattern, and a free-viewing pattern. Referring to example (700) of FIG. 7, it can be determined that the similarity for each of the cross-reading pattern, the glance pattern, and the free-viewing pattern is less than or equal to a first criterion similarity.

[0146] Referring to example (700) of FIG. 7, a combination of multiple gaze fixations and multiple gaze hops between the multiple gaze fixations can be determined to correspond to a reading pattern. Referring to example (700) of FIG. 7, the similarity to the reading pattern can be determined to exceed a first criterion similarity.

[0147] In one embodiment, the wearable device (101) may provide the user with an auxiliary function (e.g., auto-scroll) corresponding to the reading pattern while the reading pattern is in an active state. In one embodiment, the wearable device (101) may provide the user with an auxiliary function (e.g., auto-scroll) corresponding to the reading pattern while the reading pattern is in a transition state. In one embodiment, the wearable device (101) may provide the user with an auxiliary function (e.g., auto-scroll) of lower intensity (e.g., speed) than the auxiliary function (e.g., auto-scroll) provided in the active state while the reading pattern is in a transition state.

[0148] In Fig. 7, the pattern corresponding to the gaze movement is illustrated as a perusing pattern, but this is merely an example. For example, if the pattern corresponding to the gaze movement is a cross-reading pattern and the state of the cross-reading pattern is an active state (or a transition state), the wearable device (101) may provide the user with a visual cue and / or an auxiliary screen (e.g., a screen indicating one of the two areas (220, 225) to which the gaze jumps). For example, if the pattern corresponding to the gaze movement is a glance pattern and the state of the cross-reading pattern is an active state (or a transition state), the wearable device (101) may summarize the content (230). In one embodiment, the wearable device (101) may display the summarized content (230) the next time the user views the content (230). Hereinafter, auxiliary functions for each of the reading pattern, the cross-reading pattern, the skimming pattern, and the free viewing pattern can be specifically described with reference to FIGS. 11 to 15.

[0149] FIG. 8 is a diagram illustrating a change in state over time of each of a plurality of reading patterns according to one embodiment.

[0150] FIG. 8 can be explained with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5, and FIG. 6.

[0151] Fig. 8 may illustrate an example (800) where two patterns are activated simultaneously. Simultaneous activation of two patterns may indicate that the states of the two patterns are simultaneously active. A situation where the in-depth reading pattern and the cross-reading pattern are simultaneously activated, as illustrated in the example (800) of Fig. 8, may be illustrated when a user references multiple reading materials, or when multiple images exist in a single reading material.

[0152] In a situation such as FIG. 8, the wearable device (101) can select whether to provide a scroll function when the reading pattern is activated and a visual cue (or auxiliary screen) provision function when the cross-reading pattern is activated. For example, in a situation such as FIG. 8, the wearable device (101) can simultaneously provide a scroll function and a visual cue (or auxiliary screen) provision function. For example, in a situation such as FIG. 8, the wearable device (101) can provide a function selected from the scroll function or the visual cue (or auxiliary screen) provision function. For example, in a situation where the gaze movement according to the reading pattern is not fast, in case of many jumps to refer to various images, only the visual cue (or auxiliary screen) can be provided without scrolling. For example, in a situation where the gaze movement according to the reading pattern is not fast, in case of referencing a small number of images through jumps, the visual cue (or auxiliary screen) can be provided together with scrolling. However, the present invention is not limited thereto.

[0153] FIG. 9 is a diagram illustrating a change in state over time of each of a plurality of reading patterns according to one embodiment.

[0154] FIG. 9 can be explained with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5, and FIG. 6.

[0155] Figure 9 illustrates an example (900) where two patterns are sequentially activated. The sequential activation of two patterns may indicate that one of the two patterns is in an active state while the other is in a transition state. A situation where the reading pattern and the browsing pattern are sequentially activated, as in the example (900) of Figure 9, may be used when a user extracts a desired portion and reads it carefully (e.g., when viewing a web page).

[0156] In a situation such as FIG. 9, the wearable device (101) can select whether to perform a scroll function when the reading pattern is activated and a summary function of the content (230) when the browsing pattern is activated. For example, in a situation such as FIG. 9, the wearable device (101) can simultaneously perform the scroll function and the summary function of the content (230). For example, in a situation such as FIG. 9, while providing the scroll function, the wearable device (101) can summarize the content (230) based on the first areas where the user's gaze is located and the second areas where the user's gaze is not located (or where the user's gaze jumps) (or where the user's gaze is not fixed). For example, the wearable device (101) can summarize the content (230) based on the contents of the first areas. Accordingly, when the user re-views the content (230), the user can check the summary of the content that he or she focused on. For example, the wearable device (101) can summarize content (230) based on the contents of the second areas. Accordingly, when the user re-views the content (230), the user can check a summary of unconfirmed content.

[0157] FIG. 10 is a diagram illustrating a change in state over time of each of a plurality of reading patterns according to one embodiment.

[0158] FIG. 10 can be explained with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5, and FIG. 6.

[0159] FIG. 10 may illustrate an example (1000) in which a free-view pattern is activated simultaneously (or sequentially) with other patterns. A situation in which a glance pattern and a free-view pattern are activated simultaneously (or sequentially) as in example (1000) of FIG. 10 may occur when a user is viewing content (e.g., a comic book or a movie) that includes short text and multiple images.

[0160] In a situation such as FIG. 10, the wearable device (101) may provide auxiliary functions corresponding to the glance pattern and the free view pattern simultaneously when the glance pattern and the free view pattern are activated simultaneously (or sequentially). In one embodiment, the wearable device (101) may provide auxiliary functions corresponding to the glance pattern and the free view pattern simultaneously in addition to the auxiliary functions corresponding to the glance pattern or the free view pattern. For example, the wearable device (101) may display subtitles of a movie at the position of the user's gaze when the glance pattern and the free view pattern are activated simultaneously (or sequentially). However, the present invention is not limited thereto.

[0161] FIG. 11 is a diagram illustrating an operation of an electronic device providing a reading assistance function corresponding to a reading pattern according to one embodiment.

[0162] FIG. 11 can be explained with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5, and FIG. 6.

[0163] In one embodiment, the wearable device (101) can scroll content (230) while the reading pattern is activated (or while in a transition state after the reading pattern is activated).

[0164] For example, referring to FIG. 11, the wearable device (101) may scroll the content (230) at a scroll speed to cause the user's (200) gaze to look at a designated area (1110) (or a comfortable area). In one embodiment, the designated area (1110) may be located at the center between an upper area (1120) of a designated size and a lower area (1130) of a designated different size of the display (120). However, the present invention is not limited thereto. The designated area (1110) may be an area that includes a position of a gaze identified when the user previously viewed the content (230). The designated area (1110) may be an area where the gaze was mainly located by scrolling by user input when the user previously viewed the content (230).

[0165] In one embodiment, the wearable device (101) can identify a scroll speed based on the speed at which the user (200) reads the content (230) (e.g., the length of time for gaze fixation (1, 2, 3, 4)). For example, the wearable device (101) can identify a scroll speed that is proportional to the speed at which the user (200) reads the content (230). In one embodiment, the wearable device (101) can identify a scroll speed based on the distance (or speed) at which the gaze jumps in the direction of a line change. For example, the wearable device (101) can identify a scroll speed that is proportional to the distance (or speed) of the gaze jump in the direction of a line change (1111). In one embodiment, the wearable device (101) can identify a scroll speed based on the degree of similarity. For example, the wearable device (101) can identify a higher scroll speed as the similarity increases.

[0166] In one embodiment, the wearable device (101) can identify the scroll direction based on the direction of the gaze hop (e.g., the line-changing direction (1111), or the opposite direction (1115) of the line-changing direction (1111)). For example, the wearable device (101) can determine the scroll direction based on the direction of the gaze hop.

[0167] FIG. 12 is a diagram illustrating an operation of an electronic device providing a reading assistance function corresponding to a scanning pattern according to one embodiment.

[0168] FIG. 12 can be explained with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5, and FIG. 6.

[0169] In one embodiment, the wearable device (101) can summarize content (230) based on the activation of a scanning pattern.

[0170] For example, referring to FIG. 12, the wearable device (101) can summarize the content (230) based on the area (1210) where the user (200) focused on viewing the content (230) (or the area with a high heat map) (or the area where the user's gaze is located) and the area (1220, 1225) where the user (200) did not view (or the area where the gaze jumped) (or the area where the gaze is not located) (or the area where the user's gaze is not fixed).

[0171] In one embodiment, the wearable device (101) may summarize a portion of the content (230) corresponding to the area (1210) where the user (200) viewed the content (230) if the area (1210) of the content (230) viewed by the user (200) is greater than or equal to a reference ratio of the entire area of ​​the content (230). In one embodiment, the wearable device (101) may summarize a portion of the content (230) corresponding to areas (1220, 1225) other than the area (1210) where the user (200) viewed the content (230) if the area (1210) of the content (230) viewed by the user (200) is less than a reference ratio of the entire area of ​​the content (230). However, the present invention is not limited thereto.

[0172] For example, referring to FIG. 12, the wearable device (101) can summarize a portion (1230) of the content (230) that includes an area (1210) where the content (230) is viewed and areas adjacent to the area (1210) where the content is viewed. For example, the adjacent area may be an area that includes content related to the area (1210) where the content (230) is viewed and a summary of the area where the content is viewed. For example, the adjacent area may include an area of ​​a sentence that is simultaneously included in the area (1210) where the content (230) is viewed and the adjacent area.

[0173] For example, referring to FIG. 12, the wearable device (101) can summarize portions (1241, 1245) of content (230) including areas (1220, 1225) and areas adjacent to areas (1220, 1225). For example, the adjacent areas may be areas that include relevant content for summarizing areas (1220, 1225) that were not viewed among areas (1210) that were viewed of content (230). For example, the adjacent areas may include areas of sentences that are simultaneously included in areas (1220, 1225) that were not viewed and areas that were viewed (1210) of content (230).

[0174] According to an embodiment, the wearable device (101) may display a heat map indicating an area (1210) where content (230) was viewed on top of the content (230), but is not limited thereto.

[0175] FIG. 13 is a diagram illustrating an operation of an electronic device providing a read assistance function corresponding to a cross-read pattern according to one embodiment.

[0176] FIG. 13 can be explained with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5, and FIG. 6.

[0177] In one embodiment, the wearable device (101) may provide a visual cue (1340) and / or an auxiliary screen (1330) based on the activation of the cross-reading pattern.

[0178] For example, referring to FIG. 13, the wearable device (101) may provide a visual cue (1340) and / or a secondary screen (1330) for a portion of the content (230) based on two or more areas (1310, 1320) in which the user's (200) gaze hop occurs relative to each other, according to a cross-reading pattern.

[0179] In one embodiment, the wearable device (101) may provide a visual cue (1340) and / or a secondary screen (1330) for the area containing the image (1320) when the gaze hop between the area containing the text (1310) and the area containing the image (1320) is repeated (e.g., two or more times).

[0180] For example, the wearable device (101) may provide a visual cue (1340) (e.g., an image flashing) around an area (1320) containing an image. For example, the wearable device (101) may crop an area (1320) containing an image and then paste the cropped image around an area (1310) containing text. For example, the wearable device (101) may display a secondary screen (1330) for an area (1320) containing an image around an area (1310) containing text.

[0181] In FIG. 13, the wearable device (101) is shown as displaying a secondary screen of an image around text, but this is merely an example. According to an embodiment, the wearable device (101) may display a secondary screen of text around an image.

[0182] FIG. 14 is a diagram illustrating an operation of an electronic device providing a reading assistance function corresponding to a free view pattern according to one embodiment.

[0183] FIG. 14 can be explained with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5, and FIG. 6.

[0184] In one embodiment, the wearable device (101) may activate (or provide) an auxiliary function based on the type of content (230) based on the activation of the free view pattern. For example, the wearable device (101) may determine the display location of the additional content based on the location of the user's (200) gaze based on whether the content (230) includes additional content. For example, the wearable device (101) may determine the display location of the subtitles based on the location of the user's (200) gaze based on whether the content (230) is a movie including subtitles. For example, referring to FIG. 14, the wearable device (101) may display subtitles (1420) at the location of the user's (200) gaze (1410). However, the present invention is not limited thereto. The wearable device (101) may determine the location of the additional content of the content (230) to be away from the user's (200) line of sight when the user (200) does not view the additional content.

[0185] FIG. 15 is a diagram illustrating an operation of an electronic device generating a lifelog according to gaze reading patterns according to one embodiment.

[0186] FIG. 15 can be explained with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5, and FIG. 6.

[0187] In FIGS. 2 to 14, it is exemplified that the wearable device (101) provides an auxiliary function to the user (200) when the user (200) of the wearable device (101) views content (230). The provision of the auxiliary function of the wearable device (101) is not limited to the viewing of content (230). For example, the wearable device (101) may provide a designated function to the user (200) based on the movement of the user's (200) gaze while the user (200) is performing his / her daily life.

[0188] For example, the wearable device (101) may summarize screens (1510, 1520, 1530) displayed through the display (120) based on the concentration of the user (200) while using the wearable device (101). For example, the wearable device (101) may summarize screens (1510, 1520, 1530) displayed through the display (120) to summarize (or life log) the day of the user (200). For example, the wearable device (101) may summarize the contents of the screen of the display (120) viewed by the user (200) as text and / or images. For example, the wearable device (101) may summarize the screen (1510, 1520) of the display (120) into text when the user (200) recognizes the screen (1510, 1520) of the display (120) as semantic information (e.g., a reading pattern and / or a scanning pattern). For example, the wearable device (101) may summarize the screen into text (e.g., 1541, 1543) when the gaze movement according to the gaze fixations (1511, 1515, 1521, 1525) while looking at the screen (1510, 1520) of the display (120) corresponds to a reading pattern and / or a scanning pattern.

[0189] For example, the wearable device (101) may summarize the screen (1530) of the display (120) into an image when the user (200) recognizes the screen (1530) of the display (120) as visual information (e.g., cross-reading pattern and / or free viewing pattern). For example, the wearable device (101) may summarize the screen into an image (e.g., 1545 and 1547) when the gaze movement according to the gaze fixations (1531, 1533, and 1535) looking at the screen (1530) of the display (120) corresponds to the cross-reading pattern and / or the free viewing pattern.

[0190] For example, a wearable device (101) can provide content (1540) summarizing a day (or life logging) to a user (200).

[0191] According to an embodiment, the wearable device (101) may be a device for providing an extended reality (XR) service that displays images generated by a computer. In one embodiment, the XR service may include a service for augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines AR and VR. For example, the wearable device (101) may be AR glasses and / or a head-mounted device (HMD). In this case, the screens (1510, 1520, 1530) of FIG. 15 may be screens on which images representing real reality are displayed on the display (120). In this case, the screens (1510, 1520, 1530) of FIG. 15 may be screens for pass through (or VST (video see through)).

[0192] In one embodiment, when the wearable device (101) is a device for providing an XR service, the content of an image (or an image representing the front of the user (200)) acquired through the camera module (180) while the user (200) is wearing the wearable device (101) can be summarized as text and / or an image. For example, when the wearable device (101) recognizes an image representing a real space to be passed through (or VST) as semantic information (e.g., a perusing pattern and / or a scanning pattern), the wearable device (101) can summarize the image as text. For example, when the wearable device (101) recognizes an image representing a real space to be passed through (or VST) as visual information (e.g., a cross-reading pattern and / or a free viewing pattern), the wearable device (101) can summarize the image as an image.

[0193] FIG. 16 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0194] FIG. 16 can be explained with reference to FIG. 1, FIG. 2, FIG. 3, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5, and FIG. 6.

[0195] Referring to FIG. 16, in operation 1610, the wearable device (101) can identify the movement of the gaze.

[0196] In one embodiment, the wearable device (101) can identify a gaze (or gaze movement) characteristic based on the position of the pupils of the eyes (211, 215) and the distance between the eyes (211, 215) of the user (200). In one embodiment, the gaze characteristic (or gaze movement characteristic) may include a fixation and a saccade of the gaze. In one embodiment, a fixation of the gaze may refer to a situation in which the eyes (211, 215) are focused on a specific location for a predetermined period of time (e.g., 200 milliseconds) or longer. For example, a saccade of the gaze may refer to a situation in which the eyes (211, 215) move rapidly between two adjacent gaze fixations.

[0197] In operation 1620, the wearable device (101) can identify similarities with reading patterns.

[0198] In one embodiment, the wearable device (101) can identify at least one pattern corresponding to the movement of the user's (200) gaze based on characteristics of gaze spurts and / or characteristics of fixations of the user's (200). In one embodiment, the wearable device (101) can identify a degree of association (or similarity) between the gaze movement and each of the plurality of patterns based on characteristics of a plurality of gaze fixations representing the movement of the gaze over a specified length of time (or time window) and / or characteristics of a plurality of gaze spurts. For example, the wearable device (101) can determine whether a combination of a minimum number of gaze fixations that allow interpretation of a pattern and a plurality of gaze spurts between the plurality of gaze fixations corresponds to each of the plurality of patterns.

[0199] For example, if a reading pattern is determined based on two gaze fixations and one gaze spurt, the wearable device (101) can determine whether each of nine gaze movements combined based on ten gaze fixations and nine gaze spurts corresponds to the reading pattern. For example, the wearable device (101) can identify the ratio of movements corresponding to the reading pattern among the nine gaze movements as a combination stage (or similarity) for the reading pattern.

[0200] For example, if a glance pattern is determined based on three gaze fixations and two gaze hops, the wearable device (101) can determine whether each of eight gaze movements combined based on ten gaze fixations and nine gaze hops corresponds to the glance pattern. For example, the wearable device (101) can identify the proportion of movements corresponding to the glance pattern among the eight gaze movements as a combination step (or similarity) for the glance pattern.

[0201] For example, if a cross-reading pattern is determined based on four gaze fixations and three gaze hops, the wearable device (101) can determine whether each of the seven gaze movements combined based on ten gaze fixations and nine gaze hops corresponds to the cross-reading pattern. For example, the wearable device (101) can identify the proportion of movements corresponding to the cross-reading pattern among the seven gaze movements as a combination step (or similarity) for the cross-reading pattern.

[0202] Identifying a combination phase (or similarity) based on the rate of movement is only an example. For example, the wearable device (101) can identify the distance (or cosine similarity) between the values ​​(or vector values) represented by the characteristics of a plurality of gaze fixations representing the movement of the gaze during a specified time period (or time window) and / or the characteristics of a plurality of gaze hops and the values ​​(or vector values) represented by the characteristics of the gaze fixations and gaze hops set for each of the plurality of patterns as a similarity. However, the present invention is not limited thereto. In one embodiment, the wearable device (101) can identify the probability for each of the plurality of patterns as a similarity by inputting the values ​​(or vector values) represented by the characteristics of a plurality of gaze fixations representing the movement of the gaze during a specified time period (or time window) and / or the characteristics of a plurality of gaze hops into an AI (artificial intelligence) model (e.g., a model employing a convolutional neural network (CNN)) for classifying classes (or patterns).

[0203] At operation 1630, the wearable device (101) can determine whether the reading pattern is activated.

[0204] In one embodiment, the wearable device (101) can identify the state of each of the plurality of patterns based on the similarity of each of the plurality of patterns. In one embodiment, the state of each of the plurality of patterns can include idle, ready, active, or transition. In one embodiment, the wearable device (101) can identify a pattern among the plurality of patterns whose state is other than idle (i.e., ready, active, or transition) as a pattern corresponding to the movement of the gaze.

[0205] In one embodiment, after the state of the pattern changes from transition to idle, during a time period during which the similarity between the pattern and the gaze movement remains below a first criterion similarity, the pattern may be determined to be in an idle state. In one embodiment, from a time period during which the similarity between the pattern and the gaze movement rises above a first criterion similarity while the state of the pattern is idle, during a time period during a preparation time, the pattern may be determined to be in a ready state. In one embodiment, from a time period after the preparation time when the state of the pattern is ready, during a time period during which the similarity is above a second criterion similarity, the pattern may be determined to be in an active state. In one embodiment, during a time period during a transition time, from a time period during which the similarity between the pattern and the gaze movement falls below a second criterion similarity while the state of the pattern is active, the pattern may be determined to be in a transition state. In one embodiment, if the similarity of the pattern does not rise above the first criterion similarity during the transition time, the pattern may be determined to be in an idle state after the transition time.

[0206] At operation 1630, based on determining that the reading pattern is activated, the wearable device (101) may perform operation 1640. At operation 1630, based on determining that the reading pattern is not activated, the wearable device (101) may perform operation 1620 again.

[0207] In operation 1640, the wearable device (101) may provide a reading assistance function corresponding to the activated reading pattern. In one embodiment, the wearable device (101) may identify an additional function (or a reading assistance function) corresponding to a pattern corresponding to eye movement. For example, if there is only one pattern corresponding to eye movement, the wearable device (101) may identify one additional function. In one embodiment, the wearable device (101) may provide (or perform) at least one identified additional function.

[0208] FIG. 17 is a diagram illustrating a situation in which an electronic device identifies a user's gaze according to one embodiment.

[0209] The electronic device (1701) of FIG. 17 may be an electronic device that is not worn on the head of the user (1700), but is not limited thereto.

[0210] Referring to FIG. 17, an electronic device (1701) may include a display (1760) that displays content (1730) and a sensor (1780). In one embodiment, the display (1760) may correspond to the display (120) of FIG. 1 or the display module (1860) of FIG. 18. In one embodiment, the sensor (1780) may correspond to the sensor (130) of FIG. 1. In one embodiment, the sensor (130) may correspond to the camera module (1880) and / or the sensor module (1876) of FIG. 18. In FIG. 17, the sensor (1780) is illustrated as one, but this is merely an example. In one embodiment, the sensor (1780) may be two or more hardware pieces. For example, the sensor (1780) may include a camera module for acquiring images that are physically separated from each other (or distinct) and a sensor module for acquiring spatial information of the surroundings. For example, the sensor (1780) may be arranged to face the face of the user (1700) when the user (1700) looks at the display (1760) of the electronic device (1701) so as to capture at least a portion of the face of the user (1700) (e.g., eyes (1721, 1715)).

[0211] In one embodiment, a user (1700) can look at a display (1760) of an electronic device (1701) through eyes (1711, 1715). A user (200) can view content (1730) displayed through the display (1760). In one embodiment, the electronic device (1701) can provide a convenience function when the user (1700) views the content (1730). For example, the electronic device (1701) can provide a convenience function corresponding to a reading pattern of the user (1700) (or a gaze movement pattern of the user (1700)) when the user (200) views the content (1730). In one embodiment, the convenience function corresponding to the reading pattern (or the eye movement pattern of the user (1700)) may correspond to the convenience function provided by the wearable device (101) described with reference to FIGS. 1 to 16.

[0212] FIG. 18 is a block diagram of an electronic device (1801) within a network environment (1800), according to various embodiments.

[0213] Referring to FIG. 18, in a network environment (1800), an electronic device (1801) may communicate with an electronic device (1802) via a first network (1898) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (1804) or a server (1808) via a second network (1899) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (1801) may communicate with the electronic device (1804) via the server (1808). According to one embodiment, the electronic device (1801) may include a processor (1820), a memory (1830), an input module (1850), an audio output module (1855), a display module (1860), an audio module (1870), a sensor module (1876), an interface (1877), a connection terminal (1878), a haptic module (1879), a camera module (1880), a power management module (1888), a battery (1889), a communication module (1890), a subscriber identification module (1896), or an antenna module (1897). In some embodiments, the electronic device (1801) may omit at least one of these components (e.g., the connection terminal (1878)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1876), camera module (1880), or antenna module (1897)) may be integrated into a single component (e.g., display module (1860)).

[0214] The processor (1820) may control at least one other component (e.g., a hardware or software component) of the electronic device (1801) connected to the processor (1820) by executing, for example, software (e.g., a program (1840)), and may perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1820) may store commands or data received from other components (e.g., a sensor module (1876) or a communication module (1890)) in a volatile memory (1832), process the commands or data stored in the volatile memory (1832), and store result data in a non-volatile memory (1834). According to one embodiment, the processor (1820) may include a main processor (1821) (e.g., a central processing unit or an application processor) or an auxiliary processor (1823) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1821). For example, when the electronic device (1801) includes the main processor (1821) and the auxiliary processor (1823), the auxiliary processor (1823) may be configured to use less power than the main processor (1821) or to be specialized for a given function. The auxiliary processor (1823) may be implemented separately from the main processor (1821) or as a part thereof.

[0215] The auxiliary processor (1823) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1860), the sensor module (1876), or the communication module (1890)) of the electronic device (1801), for example, on behalf of the main processor (1821) while the main processor (1821) is in an inactive (e.g., sleep) state, or together with the main processor (1821) while the main processor (1821) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1823) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1880) or a communication module (1890)). In one embodiment, the auxiliary processor (1823) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1801) where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1808)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0216] The memory (1830) can store various data used by at least one component (e.g., the processor (1820) or the sensor module (1876)) of the electronic device (1801). The data can include, for example, software (e.g., the program (1840)) and input data or output data for commands related thereto. The memory (1830) can include volatile memory (1832) or non-volatile memory (1834).

[0217] The program (1840) may be stored as software in memory (1830) and may include, for example, an operating system (1842), middleware (1844), or an application (1846).

[0218] The input module (1850) can receive commands or data to be used in a component of the electronic device (1801) (e.g., a processor (1820)) from an external source (e.g., a user) of the electronic device (1801). The input module (1850) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0219] The audio output module (1855) can output audio signals to the outside of the electronic device (1801). The audio output module (1855) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0220] The display module (1860) can visually provide information to an external party (e.g., a user) of the electronic device (1801). The display module (1860) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (1860) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0221] The audio module (1870) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (1870) can acquire sound through the input module (1850), output sound through the sound output module (1855), or an external electronic device (e.g., electronic device (1802)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1801).

[0222] The sensor module (1876) can detect the operating status (e.g., power or temperature) of the electronic device (1801) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1876) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0223] The interface (1877) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1801) with an external electronic device (e.g., the electronic device (1802)). In one embodiment, the interface (1877) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0224] The connection terminal (1878) may include a connector through which the electronic device (1801) may be physically connected to an external electronic device (e.g., the electronic device (1802)). In one embodiment, the connection terminal (1878) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0225] The haptic module (1879) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1879) may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0226] The camera module (1880) can capture still images and moving images. In one embodiment, the camera module (1880) may include one or more lenses, image sensors, image signal processors, or flashes.

[0227] The power management module (1888) can manage the power supplied to the electronic device (1801). According to one embodiment, the power management module (1888) can be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

[0228] A battery (1889) may power at least one component of the electronic device (1801). In one embodiment, the battery (1889) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0229] The communication module (1890) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1801) and an external electronic device (e.g., electronic device (1802), electronic device (1804), or server (1808)), and the performance of communication through the established communication channel. The communication module (1890) may operate independently from the processor (1820) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1890) may include a wireless communication module (1892) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1894) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1804) via a first network (1898) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1899) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1892) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1896) to identify or authenticate the electronic device (1801) within a communication network such as the first network (1898) or the second network (1899).

[0230] The wireless communication module (1892) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1892) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1892) may support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1892) may support various requirements specified in the electronic device (1801), an external electronic device (e.g., the electronic device (1804)), or a network system (e.g., the second network (1899)). According to one embodiment, the wireless communication module (1892) can support a peak data rate (e.g., 20 Gbps or more) for eMBB implementation, a loss coverage (e.g., 164 dB or less) for mMTC implementation, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC implementation.

[0231] The antenna module (1897) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1897) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1897) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1898) or the second network (1899), may be selected from the plurality of antennas by, for example, the communication module (1890). A signal or power may be transmitted or received between the communication module (1890) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1897).

[0232] According to various embodiments, the antenna module (1897) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0233] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0234] According to one embodiment, commands or data may be transmitted or received between the electronic device (1801) and an external electronic device (1804) via a server (1808) connected to a second network (1899). Each of the external electronic devices (1802 or 1804) may be the same or a different type of device as the electronic device (1801). According to one embodiment, all or part of the operations executed in the electronic device (1801) may be executed in one or more of the external electronic devices (1802, 1804, or 1808). For example, when the electronic device (1801) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1801) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1801). The electronic device (1801) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1801) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1804) may include an Internet of Things (IoT) device. The server (1808) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1804) or server (1808) may be included within the second network (1899). The electronic device (1801) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0235] As described above, the wearable device (101) may include a display (120). The wearable device (101) may include a sensor (130) that obtains data for identifying a user's gaze. The wearable device (101) may include at least one processor (110) including a processing circuit. The wearable device (101) may include a memory (130) that stores instructions and includes one or more storage media. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify a movement of the user's gaze through the sensor (130) while displaying content (230) through the display (120). The movement may include a combination of a sprint of the gaze and a fixation of the gaze. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify at least one reading pattern corresponding to the movement among a plurality of reading patterns having different jumping characteristics and / or fixation characteristics based on the jumping characteristics and / or fixation characteristics of the movement. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to provide, through the display (120), a reading assistance function corresponding to the at least one reading pattern.

[0236] The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify a reading pattern as the at least one reading pattern based on the direction of the leap of the movement corresponding to the writing direction of the content (230) and the fixation of the movement being maintained for a reference time or longer. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to scroll the content (230) based on the identification of the reading pattern as the at least one reading pattern.

[0237] The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to determine a scroll speed corresponding to the reading pattern based on the position of the gaze within the content (230) and / or the period of the leap in the movement. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to scroll the content (230) based on the determined speed.

[0238] The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify a glance pattern as the at least one reading pattern based on a distance of the leap of the gaze included in the movement being greater than or equal to a reference distance and a time length of the fixation of the gaze being less than or equal to a reference time length. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to summarize the content (230) based on the identification of the glance pattern as the at least one reading pattern. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to provide the summarized content (230) through the display (120).

[0239] The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify first areas within the content (230) where the gaze is fixed and second areas where the gaze is skipped. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to summarize the content (230) based on portions of the content (230) included in the first areas.

[0240] The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify a cross-read pattern as the at least one read pattern based on the movement repeatedly jumping between the first area and the second area. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to display a portion of the content (230) displayed in the second area around the first area based on the cross-read pattern being identified as the at least one read pattern.

[0241] The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify a cross-read pattern as the at least one reading pattern based on the movement repeatedly jumping between the first region and the second region. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to display a visual cue for the second region while the gaze is positioned in the first region based on the cross-read pattern being identified as the at least one reading pattern.

[0242] The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify a similarity between each of the plurality of reading patterns and the movement. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify the at least one reading pattern whose similarity with the movement exceeds a first criterion similarity.

[0243] The above instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to provide the read assistance function corresponding to the at least one read pattern after a reference time has elapsed since the similarity exceeds the first reference similarity.

[0244] The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify that the similarity is less than or equal to a second reference similarity while the read assist function is provided. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to stop providing the read assist function corresponding to the at least one read pattern after another reference time has elapsed while the similarity is less than or equal to the second reference similarity.

[0245] The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify two or more read assistance functions corresponding to each of the two or more read patterns, based on the at least one read pattern being two or more read patterns. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to provide, through the display (120), at least one read assistance function selected from among the two or more read assistance functions.

[0246] The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify a read assist function set for two or more read patterns based on the at least one read pattern being two or more read patterns. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to provide the one read assist function through the display (120).

[0247] As described above, the method can be performed in a wearable device (101) including a display (120) and a sensor (130) for acquiring data for identifying a user's gaze. The method can include an operation of identifying a movement of the user's gaze through the sensor (130) while displaying content (230) through the display (120). The movement can include a combination of a leap of the gaze and a fixation of the gaze. The method can include an operation of identifying at least one reading pattern corresponding to the movement among a plurality of reading patterns having different leap characteristics and / or fixation characteristics based on a characteristic of the leap of the movement and / or the fixation. The method can include an operation of providing a reading assistance function corresponding to the at least one reading pattern through the display (120).

[0248] The method may include an operation of identifying a reading pattern as at least one reading pattern based on the direction of the leap of the movement corresponding to the writing direction of the content (230) and the fixation of the movement being maintained for a reference time or longer. The method may include an operation of scrolling the content (230) based on the identification of the reading pattern as at least one reading pattern.

[0249] The method may include an operation of identifying a scanning pattern as at least one reading pattern based on a distance of the leap of the gaze included in the movement being greater than or equal to a reference distance and a time length of the fixation of the gaze being less than or equal to a reference time length. The method may include an operation of summarizing the content (230) based on the identification of the scanning pattern as the at least one reading pattern. The method may include an operation of providing the summarized content (230) through the display (120).

[0250] The method may include an operation of identifying a cross-reading pattern as the at least one reading pattern based on the movement repeatedly jumping between the first area and the second area. The method may include an operation of displaying a portion of the content (230) displayed in the second area around the first area based on the cross-reading pattern being identified as the at least one reading pattern.

[0251] The method may include an operation of identifying a cross-reading pattern as the at least one reading pattern based on the movement repeatedly jumping between the first region and the second region. The method may include an operation of displaying a visual cue for the second region while the gaze is positioned in the first region based on the cross-reading pattern being identified as the at least one reading pattern.

[0252] The method may include an operation of identifying a similarity between each of the plurality of reading patterns and the movement. The method may include an operation of identifying at least one reading pattern whose similarity with the movement exceeds a first criterion similarity.

[0253] The method may include an operation of providing the reading assistance function corresponding to the at least one reading pattern after a reference time has elapsed from the point in time when the similarity exceeds the first reference similarity.

[0254] The method may include an operation of identifying that the similarity is less than or equal to a second reference similarity while the reading assistance function is provided. The method may include an operation of stopping the provision of the reading assistance function corresponding to the at least one reading pattern after another reference time has elapsed while the similarity is less than or equal to the second reference similarity.

[0255] As described above, a non-transitory computer readable storage medium can store a program including instructions. The instructions, when individually or collectively executed by at least one processor (110) of a wearable device (101) including a display (120) and a sensor (130) for obtaining data for identifying a user's gaze, can cause the wearable device (101) to identify a movement of the user's gaze through the sensor (130) while displaying content (230) through the display (120). The movement can include a combination of a sprint of the gaze and a fixation of the gaze. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to identify at least one reading pattern corresponding to the movement among a plurality of reading patterns having different jumping characteristics and / or fixation characteristics based on the jumping characteristics and / or fixation characteristics of the movement. The instructions, when individually or collectively executed by the at least one processor (110), may cause the wearable device (101) to provide, through the display (120), a reading assistance function corresponding to the at least one reading pattern.

[0256] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0257] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0258] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0259] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., a wearable device (101)). For example, a processor (e.g., a processor (110)) of the machine (e.g., a wearable device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0260] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0261] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable device (101), Display (120), A sensor (130) that acquires data to identify the user's gaze; At least one processor (110) comprising a processing circuit; and A wearable device (101) comprises a memory (130) storing instructions and including one or more storage media, wherein the instructions, when individually or collectively executed by the at least one processor (110), While displaying content (230) through the display (120), the movement of the user's gaze is identified through the sensor (130), and the movement includes a combination of a leap of the gaze and a fixation of the gaze. Based on the characteristics of the jump of the movement and / or the fixation, at least one reading pattern corresponding to the movement is identified among a plurality of reading patterns having different characteristics of the jump and / or the fixation, Causing the display (120) to provide a reading assistance function corresponding to at least one reading pattern; Wearable devices.

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: The direction of the jump of the above movement corresponds to the writing direction of the above content (230), and based on the fixation of the above movement being maintained for a reference time or longer, the reading pattern is identified as at least one reading pattern, Causing the content (230) to be scrolled based on the above-mentioned reading pattern being identified as at least one reading pattern, Wearable devices.

3. In claim 2, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: Based on the position of the gaze within the content (230) and / or the period of the jump of the movement, a scroll speed corresponding to the reading pattern is determined, Based on the above-determined speed, causing the content (230) to be scrolled, Wearable devices.

4. In any one of claims 1 to 3, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: Identifying a scanning pattern as at least one reading pattern based on the distance of the leap of the gaze included in the movement being greater than or equal to a reference distance and the time length of the fixation of the gaze being less than or equal to a reference time length, Summarizing the content (230) based on the above-mentioned scanning pattern being identified as at least one reading pattern, Causing the above summarized content (230) to be provided through the above display (120), Wearable devices.

5. In claim 4, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: Identifying the first areas where the gaze is fixed and the second areas where the gaze is skipped within the above content (230), Based on the portions of the content (230) included in the first areas, causing the content (230) to be summarized, Wearable devices.

6. In any one of claims 1 to 5, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: Based on the above movement repeatedly jumping between the first and second regions, the cross-reading pattern is identified as at least one reading pattern, Causing a part of the content (230) displayed in the second area to be displayed around the first area based on the cross-reading pattern being identified as at least one reading pattern; Wearable devices.

7. In any one of claims 1 to 6, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: Based on the above movement repeatedly jumping between the first and second regions, the cross-reading pattern is identified as at least one reading pattern, Causing a visual cue to be displayed for the second area while the gaze is positioned in the first area, based on the cross-reading pattern being identified as the at least one reading pattern. Wearable devices.

8. In any one of claims 1 to 7, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: Identifying the similarity between each of the above multiple reading patterns and the movement, causing said at least one reading pattern to be identified whose similarity with said movement exceeds a first criterion similarity, Wearable devices.

9. In claim 8, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: Causing the reading assistance function corresponding to the at least one reading pattern to be provided after a reference time has elapsed from the point in time when the similarity exceeds the first reference similarity, Wearable devices.

10. In claim 9, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: While the above reading assistance function is provided, it is identified that the similarity is less than or equal to the second reference similarity, Causing the provision of the reading assistance function corresponding to the at least one reading pattern to be stopped after another reference time has elapsed while the similarity is lower than or equal to the second reference similarity, Wearable devices.

11. In any one of claims 1 to 10, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: Based on the fact that the at least one reading pattern is two or more reading patterns, two or more reading assistance functions corresponding to each of the two or more reading patterns are identified, Causing at least one reading assistance function selected from among the two or more reading assistance functions to be provided through the display (120). Wearable devices.

12. In any one of claims 1 to 11, The above instructions, when individually or collectively executed by the at least one processor (110), cause the wearable device (101) to: Based on the fact that the at least one reading pattern is two or more reading patterns, identifying one reading assistance function set for the two or more reading patterns, Causing the above one reading assistance function to be provided through the display (120), Wearable devices.

13. A method of a wearable device (101) including a display (120) and a sensor (130) for acquiring data for identifying a user's gaze, An operation of identifying the movement of the user's gaze through the sensor (130) while displaying content (230) through the display (120), wherein the movement includes a combination of a leap of the gaze and a fixation of the gaze. An operation of identifying at least one reading pattern corresponding to the movement among a plurality of reading patterns having different jumping characteristics and / or fixation characteristics based on the characteristics of the jumping and / or fixation of the movement, and An operation including providing a reading assistance function corresponding to at least one reading pattern through the display (120). method.

14. In claim 13, An operation of identifying a reading pattern as at least one reading pattern based on the direction of the jump of the movement corresponding to the writing direction of the content (230) and the fixation of the movement being maintained for a reference time or longer, and An operation of scrolling the content (230) based on the above-mentioned reading pattern being identified as at least one reading pattern. method.

15. In claim 13 or claim 14, An operation of identifying a scanning pattern as at least one reading pattern based on the distance of the leap of the gaze included in the movement being greater than or equal to a reference distance and the time length of the fixation of the gaze being less than or equal to a reference time length; An operation of summarizing the content (230) based on the above-mentioned scanning pattern being identified as at least one reading pattern, and Including an operation of providing the above summarized content (230) through the display (120). method.

Images