Screen interaction using EOG coordinates

Abstract

A method comprising: obtaining a set of voltage signals from a set of electrodes placed adjacent to a user's ears; determining an EOG gaze vector in egocentric coordinates based on the set of voltage signals; determining the user's head pose in display coordinates using a sensor device worn by the user; combining the EOG gaze vector and the head pose to obtain a gaze vector in display coordinates; and determining a gaze point by calculating the intersection of the gaze vector with an imaging plane having a known position in the display coordinates.

Description

Technical Field

[0001] Cross-reference with related applications This application claims priority based on U.S. Provisional Application No. 63 / 224,062 filed on July 21, 2021 and European Patent Application No. 21190807.4 filed on August 11, 2021, the entire contents of which are incorporated herein by reference.

[0002] Technical field to which the invention belongs The present invention relates to eye tracking using electrooculogram (EOG). Specifically, the present invention relates to determining a fixation point on a display screen using such eye tracking.

Background Art

[0003] In many situations, eye tracking can be used to understand where a user's attention is focused. In particular, eye tracking can improve user control of peripheral devices.

[0004] The most common approach to eye tracking is to acquire an image of the user's eyes. Using appropriate image processing and algorithms based on numerical analysis or deep learning, the direction of the user's line of sight can be determined. The drawback of such video-based eye tracking is that it is necessary to direct a camera at the user's face or attach it to the head, which significantly limits its applications.

[0005] Recently, as an alternative to video-based eye tracking, a method using electrooculogram (EOG) has been introduced. Electrooculogram (EOG) is the measurement of the corneal-retinal dipole potential (the difference in charge between the cornea and the retina) of the eyeball. When the eyeball moves within the eye socket, the dipole rotates. This potential can be measured using a set of electrodes placed near the eye socket and can be used to estimate the position of the eyeball. The accuracy of EOG by current technology is estimated to be about 0.5 degrees, but future improvements are expected.

[0006] Compared to video-based eye tracking, EOG-based eye tracking offers several advantages, including the following: Because it eliminates the need for camera optics and image processing, hardware costs can be reduced. Because there is no need to position the camera to align with the user's line of sight, a more flexible design is possible. Robustness and accuracy are improved under harsh lighting conditions. Reducing processing and memory requirements leads to a reduction in power consumption, which is particularly important for portable / wearable devices. There is no need to point the camera at the user, and therefore no associated privacy issues.

[0007] Recent improvements in electrooculographic field of view determination (referred to herein as EOG-based eye tracking) have made numerous applications of such eye tracking possible.

[0008] However, compared to more conventional camera-based eye tracking, a challenge of EOG-based eye tracking is that EOG detection is performed relative to the user's head (known as an ego-centric coordinate system).

[0009] In many applications, such as augmented reality (AR) and virtual reality (VR), egocentricity is not a problem. In fact, egocentric eye tracking is very well suited to such applications. However, egocentricity has hindered the success of EOG-based eye tracking in many image processing applications to date. [Overview of the project] [Problems that the invention aims to solve]

[0010] The objective of the present invention is to overcome or mitigate the above-mentioned problems and enable EOG-based eye tracking in various image processing applications.

[0011] According to a first aspect of the present invention, the above and other objectives are achieved by a method comprising: acquiring a set of voltage signals from a set of electrodes positioned close to the user's ears; determining an EOG line of sight vector in egocentric coordinates based on the set of voltage signals; determining the user's head posture in display coordinates using a sensor device worn by the user; obtaining a line of sight vector in display coordinates by combining the EOG line of sight vector and the head posture; and determining a point of fixation by calculating the intersection of the line of sight vector and an imaging plane having a known position in display coordinates.

[0012] As those skilled in the art will understand, egocentric coordinates describe a position relative to the user's position (orientation), for example, the position relative to the user's head. Similarly, display coordinates describe a position relative to the display device (or a part of it). The position of the image plane of the display device is known in display coordinates.

[0013] The above method preferably further includes calibrating the sensor device to obtain its position in display coordinates. Such calibration may be performed not only during operation but also before determining the head posture in order to recalibrate the system. Calibration of the head-mounted sensor serves to determine its position in display coordinates. In a typical case, the calibration may have six degrees of freedom, but in more restrictive applications, fewer degrees of freedom may suffice. In some embodiments, the display coordinates have only two degrees of freedom (e.g., x and y). In some embodiments, the calibration does not involve using or measuring the rotational element of the degrees of freedom of head movement. For example, in applications where the viewing distance is far relative to the display width and the viewer is unlikely to rotate their head to see different parts of the display image surface (i.e., they only rotate their eyes to direct their gaze to different display areas). In this case, the display has a relatively small field of view (FOV), an example of such viewing is a smartphone viewed at arm's length.

[0014] It should be noted that calibration may be performed with respect to the image plane itself (e.g., with interaction with the display as outlined below) or with respect to other parts of the display device (e.g., the projector device).

[0015] A head-mounted sensor is configured to monitor the relative motion of the head. After initial calibration, the head-mounted sensor can thus provide head orientation in display coordinates. The head-mounted sensor may include one or more of an accelerometer, gyroscope, and magnetometer. One type of sensor useful in this context is an inertial measurement unit (IMU).

[0016] By combining the EOG (Eyew Field of View) line-of-view vector and head pose, the line-of-view vector in display coordinates can be obtained. Subsequently, the point of fixation can be determined as the intersection of the line-of-view vector and the imaging plane (which, as mentioned earlier, is also expressed in display screen coordinates). For example, a step is needed to convert from physical units on the display (e.g., mm) to pixel positions.

[0017] In some embodiments, calibration of the head-mounted sensor is achieved by synchronizing the head-mounted sensor with a second sensor device fixedly positioned relative to the display system (and thus the image plane). This embodiment is particularly useful for non-fixed display screens (e.g., smartphones) that generally include an orientation sensor device such as an IMU.

[0018] In some embodiments, spatial calibration is achieved by determining the distance between the user and the image plane. This approach may be more useful for stationary displays, such as televisions, which generally do not have an IMU. In some embodiments, the distance is determined using a suitable sensor located near or mounted within the display system (e.g., a remote control with an IR transceiver, or a LiDAR sensor on the display, which is becoming common in smartphones).

[0019] In some embodiments, the calibration includes displaying a graphic element on the imaging surface and receiving user input to confirm that the user is looking at the graphical element. Such calibration has the advantage of not only determining the position of the head-mounted sensor in display coordinates, but also providing calibration for the entire process including EOG gaze vector calculation.

[0020] In some embodiments, the method further includes offline calibration to handle any drift that may occur in the EOG gaze vector detection process. In some embodiments, such offline calibration includes, for example, statistical analysis of the user's fixation points over time, taking into account factors such as the size of the imaging surface, the regions of interest expected over time, and the like.

[0021] The method of the first aspect enables the modification of audio data and / or visual data in an audiovisual data rendering system. This modification provides an improved user experience when viewing or listening to an audiovisual presentation of a system such as a television, a projector display system, or a mobile handheld device. Examples of such improved user experiences are summarized below.

[0022] Determining the fixation point on the imaging surface based on EOG can be used, for example, to determine the gaze depth as the depth within the image associated with the fixation point, calculate the relative depth for each pixel in the image as the difference between the pixel's depth and the gaze depth, and blur the pixels according to a function of the relative depth to render the image based on depth. After blurring, the image is rendered with depth on the imaging surface. Such blurring can be done to simulate the natural depth of field that occurs in a 3D scene due to the optical system of the eye.

[0023] As another example, when depth-based image rendering is combined with a 3D sound field, the point of gaze and line of sight depth can be used to identify at least one audio object associated with the current point of interest, and such identified audio object can be highlighted. The current point of interest can be determined as a function of the point of gaze.

[0024] In yet another embodiment, the point of fixation on the imaging plane can be monitored over time using the method summarized above. Based on this monitoring, the average line of sight position and line of sight radius are determined, the line of sight radius is compared to a radius threshold, and if the line of sight radius is less than the radius threshold, the image data on the imaging plane is zoomed in. Such a procedure improves the user viewing experience, for example, in the context of rendering high spatial resolution (e.g., 4K or 8K) data on a small imaging plane such as a mobile device. To provide such an improved user viewing experience even when the average line of sight position is not at the center of the imaging plane, it may also be advantageous to determine the minimum distance between the average line of sight position and one or more edges of the imaging plane. Next, the minimum distance is compared to a distance threshold, and according to the determination that the minimum distance is less than the distance threshold, an offset is applied to the image data to increase the minimum distance. That is, such a procedure translates the image data so that even when zooming in on an object at the average line of sight position on an edge of the imaging plane, the object does not go out of view.

[0025] According to a second aspect of the present invention, the above and other objectives are achieved by a system comprising: a set of electrodes positioned close to the user's ears and configured to acquire a set of voltage signals; an EOG processing unit that determines an EOG gaze vector in egocentric coordinates based on the set of voltage signals; a user-worn sensor device that determines the user's head posture in display coordinates; and a processing unit configured to combine the EOG gaze vector and the head posture to obtain a gaze vector in display coordinates, and to determine a point of fixation by calculating the intersection of the gaze vector and an imaging plane having a known position in display coordinates.

[0026] According to a third aspect of the present invention, the above and other objects are achieved by a non - transitory computer - readable medium storing computer program code configured to execute the steps of the method according to the first aspect of the present invention when executed on a computer processor.

Brief Description of the Drawings

[0027] Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings showing presently preferred embodiments of the present invention. [Figure 1] FIG. 1 schematically shows a system for EOG - based fixation point determination according to an embodiment of the present invention. [Figure 2] FIG. 2 shows a flowchart of a method for EOG - based fixation point determination according to an embodiment of the present invention. [Figure 3] FIG. 3 shows a process for processing depth - based image data and associated audio data according to an embodiment of the present invention. [Figure 4] FIG. 4 shows a process of zooming and panning of image data according to an embodiment of the present invention.

Modes for Carrying Out the Invention

[0028] Figure 1 shows the basic elements of a system for EOG-based gaze point detection according to an embodiment of the present invention. This system is implemented in relation to a display device having an imaging surface 1. In the illustrated case, the display device is a portable device 2 such as a smartphone, but this system can be implemented in any display device having a dedicated imaging surface. For example, the display device may be a stationary display screen such as a television. Alternatively, the display device may be a projection display system including an image forming unit (projector) and an imaging surface (projection screen) located away from the image forming unit. In some embodiments, the imaging surface of the display device is integrated with an eyeglass component (e.g., a contact lens or eyeglass lens 15).

[0029] The system includes a pair of EOG electrodes 3 positioned on or adjacent to the skin of the user 8, preferably close to the ears, and an EOG processing unit 4 connected to the electrodes 3 and configured to determine the line of sight vector in egocentric coordinates (i.e., relative to the user's head). The line of sight vector may have two degrees of freedom (2DOF), which are horizontal and vertical field of view angles called azimuthal angle (change in direction of the left and right eyeballs) and elevation angle (vertical). For simple applications, only one degree of freedom (e.g., horizontal field of view) is required. In some embodiments, the EOG processing unit 4 is further configured to determine the line of sight vector partially based on rotation, which is a twisting eye movement known as binocular vergence that may occur at close viewing distances (e.g., as used in knitting).

[0030] The system further includes a head-mounted sensor unit 5, such as an inertial measurement unit (IMU), which can determine the relative position (head posture) of the head. The sensor unit 5 determines the relative movement of the object to which it is attached in 6 degrees of freedom (6DOF). These 6 degrees of freedom include three angular measurements: pitch, yaw, and roll, and three translation measurements: x, y, and z distances. The yaw angle corresponds to the azimuth angle (a term used to describe the spatial position of sound relative to the listener), and the pitch (not the sound frequency) corresponds to the elevation angle. The EOG processing unit 4 and the IMU 5 may be integrated into the same physical unit, for example, by being located in a headset 9, earphones, headphones, glasses, etc.

[0031] Electrode 3 is generally a transducer configured to convert the flow of ionic current within the human body into electric current, and examples include biopotential sensors, biopotential electrodes, and other sensor devices. Electrode 3 can also be integrated into the same physical unit as the EOG processing unit 4 and IMU 5. In particular, the electrode may be embedded in on-ear headphones or in-ear headphones. Furthermore, electrodes can also be provided using a technology known as "electronic skin," which is a soft, flexible material that adheres to the skin like a bandage.

[0032] In the illustrated embodiment, the system further includes a second sensor unit, such as a second IMU 6, which is located in the portable display device 2 and connected to the central processing unit 7. The CPU 7 is also connected to the display circuit (not shown) of the display device 2.

[0033] Both the EOG processing unit 4 and the IMU 5 are preferably connected to the CPU 7 via a wireless connection such as Bluetooth.

[0034] Figure 2 shows the various calculations performed by CPU7 to transform the EOG line of sight vector into a single point on the imaging plane 1. The calculations can be decomposed into three coordinate systems labeled egocentric, relative world, and display coordinates. Here, relative means an absolute physical distance but an unknown absolute position.

[0035] First, in block 10, a geometric relationship is determined that describes the relative position of the world coordinate system relative to the display coordinate system. Ideally, this relationship is a complete X, Y, Z position description that covers cases where the viewing angle is shifted relative to the image plane. Alternatively, this relationship is a simplified position description, for example, consisting only of distances and ignoring the aspect of viewing angle shift. Various methods for providing this calibration are described below.

[0036] In block 11, the EOG processing unit 4 provides the line of sight vector in egocentric 2DOF coordinates, and in block 12, the sensor unit 5 provides the head pose in relative world coordinates.

[0037] In block 13, the EOG gaze vector is combined with the head pose and the geometric relationship between relative world coordinates and display coordinates to obtain the 6DOF gaze vector in display coordinates. Typically, quaternion calculations are used, which may first be combined in an egocentric system and then transformed into relative and display coordinates, or vice versa.

[0038] The position of the image plane 1 and the line of sight vector are both derived in a common coordinate system (display coordinates), and in block 14, the intersection of the line of sight vector and the image plane is calculated. This calculation can be performed using standard 3D geometry. The intersection is the point of fixation, which can be converted to an XY position, expressed in physical units, and / or to pixel position units. The viewer's point of fixation is typically represented by a pixel position and can then be used as a key input item in various display-related applications, including those described below.

[0039] Calibration in block 10 can be determined in various ways. In some embodiments, the display system (e.g., portable device 2) includes a second relative position sensor unit 6, e.g., a second IMU. Since sensors 5 and 6 describe only differential changes, calibration involves a synchronous operation between the two sensors. This can be achieved by instructing the viewer to hold the head-mounted IMU 5 close to the display IMU 6 and then return it to the viewing position.

[0040] In some display systems, such as televisions, the display IMU8 is located on the television, some distance from the viewer, but in the case of mobile displays, it is generally at hand-held distance.

[0041] In some display systems, such as projector systems, the display IMU 8 may be located on the projector and not near the image plane 1. However, as long as the position of the image plane 1 is known in display coordinates, the position of the display IMU 8 is not a problem.

[0042] Currently, most televisions do not have an IMU, and in such cases, a more limited calibration may be employed, which simply detects the distance from the user to the center of the image plane, ignoring the angular aspect.

[0043] In one embodiment, distance may be determined using a remote control. The distance traveled is determined by transmitting an encrypted pattern from the remote control to the television and evaluating the travel time using the television system's internal clock cycle. In this case, the distance between the (handheld) remote control and the head is also ignored. Typically, this distance has a small component relative to the image plane.

[0044] When a mobile phone is connected to this system, there are several options for determining the distance from the viewer to the image plane. For example, the television image can be acquired from the viewing position. Knowing the focal length and pixel dimensions of the mobile phone's camera, and the television screen size (which can be determined from the model number), the distance from the mobile phone to the television can be calculated from the size of the television in the captured image.

[0045] Other techniques for determining the distance to the image plane include equipping the television with an infrared transmitter and a corresponding sensor for the head-mounted sensor 5, as well as wireless positioning techniques including active electromagnetic coils.

[0046] More sophisticated calibration may involve user interaction with the display system via certain input devices. These input devices may be user-worn devices such as headsets, other input devices connected to or communicating with user-worn devices, or input devices within the display system. Input devices may include touch interfaces, physical buttons, audio interfaces, or inputs based on the detection of flashing (in a specific manner) of any of the above devices.

[0047] In one example, the user has a pointing remote (also known as a magic wand, a remote that can control a visible cursor on the screen). Before performing EOG gaze point detection, the user is prompted to wear a device (e.g., headphones, earphones, or glasses) that includes a head-mounted sensor and an EOG transducer. Once the head-mounted device and display device are powered on, the user is prompted to point the cursor to the point on the screen where they are looking. This can be done indirectly, such as by displaying a text icon with a prompt like "Click if you can read this message" at the cursor's position. This method works because reading text becomes extremely difficult even if it's only a few degrees off the fovea, and the gaze point (fovea position) always corresponds to the position of the text being read. In other words, combining the click action with the cursor text position indicates the gaze point, and the rest of the calculations can be calibrated from this basic information. The best calibration can be obtained by testing multiple points on the screen (such as the four corners and the center), but depending on the application, a few reading positions, such as one reading position, may suffice.

[0048] Please note that the interactive calibration described is also useful for projector display systems.

[0049] This type of interactive calibration (sometimes called "offline" calibration) not only determines the position of the head-mounted sensor 5 in display coordinates, but also calibrates the EOG detection process.

[0050] A challenge with EOG-based gaze detection is its susceptibility to drift (e.g., the detection error changing over time). Several methods for addressing this challenge are disclosed in concurrently pending application EP 20153054.0, incorporated herein by reference. In addition to these methods for avoiding the drift problem, some form of "online" calibration, i.e., calibration performed while the system is in use, is desirable. Another reason for requiring online calibration is the drift associated with the detection of translational motion. Typically, a user-worn sensor 5 is based on some type of accelerometer. Double integrating the acceleration to obtain position can be susceptible to noise and drift, which may necessitate online calibration.

[0051] Such online calibration may be based on a statistical analysis of gaze points over time and knowledge of expected user viewing patterns. For example, in the case of subtitled content, viewing patterns corresponding to subtitle reading can be used for statistical calibration. For instance, modal values ​​in a gaze point histogram are expected to align with the center of the text block, so if a consistent offset is measured from this position, it can be assumed to be due to drift and corrected. The same applies to a definable discrete region of interest (ROI). The imaging region itself can also be used. If the size and shape of the imaging region are known, calibration can be performed simply by assuming that the user maintains their gaze within the imaging region.

[0052] Depth-based rendering with simulated adjustment blur In displays capable of displaying depth information (e.g., stereoscopic 3D (S3D), automated stereoscopic 3D (AS3D)), it is crucial to simulate the eye's depth of field (based on focal length). In conventional 3D images, the entire image is in focus regardless of depth (ignoring camera-based depth-of-field effects) and where the viewer is fixated. However, in natural vision, the eye's accommodation distance coincides with the fixation point. As a result, depths corresponding to the fixation point are perceived sharply, while closer and farther depths are out of focus, with the degree of defocus depending on the distance from the accommodation distance (in diopters). Consequently, for a more natural, realistic (and perhaps more comfortable) 3D display, it is important to simulate an accommodative blur effect, which is a blur that increases as the distance from the accommodation distance increases. To achieve this, it is necessary to know the eye's fixation point in the image, unlike in normal vision. However, using the EOG technology described herein, a point of focus can be determined, which can then be applied to depth-based rendering in, for example, S3D and AS3D displays.

[0053] Figure 3 shows an example of a process for depth-based image rendering. This process takes image data 31 containing depth information, also known as a 3D image representation, and a point of focus 32 in display coordinates as input.

[0054] Image data may be a voxel set I(x,y,z) or a pair of 2D images (L-view and R-view) often combined with a depth map of the image. In depth imaging, x is typically the horizontal direction (LR), y is the vertical direction, and z is the distance from the viewer to the object being imaged. The point of fixation can be determined using the process described above, with reference to Figures 1 and 2.

[0055] In block 33, the point of fixation (in display coordinates) is used to calculate the corresponding line-of-sight image position xG, yG and line-of-sight depth zG, i.e., the image depth at this image position. This can be achieved via the input depth plane, via the voxel index z, or via calculation from the LR stereo pair.

[0056] Next, in block 34, a local spatial blur function is calculated for the image. This function may be a point spreading function (PSF) that varies as a function of the relative depth Δz at a particular pixel location. The relative depth Δz is defined as the depth difference between the line-of-sight depth zG and the depth z at a particular pixel. In one example, the PSF has the same shape at all pixel locations, but its width is expanded or contracted to increase or decrease the blur, respectively. Instead of a PSF, a more advanced approach can be applied to determine the amount of blur. In one embodiment, a human visual system (HVS) model of the eye's optical system is used.

[0057] Next, in block 35, a position-shift blur function is applied to all 2D pixel positions, thereby filtering the input image as a function of relative depth. If the input is an LR pair, both images are filtered. For a single 2D + depth map input, a single 2D image is filtered.

[0058] Regardless of the technique specifications based on the input format, the result is a depth-rendered image 36 with simulated adjusted blur based on the viewer's gaze point. This image is then rendered in block 37 to match the specifications of a 3D display, for example, an S3D display consisting of LR image pairs, or an AS3D display using a multiview.

[0059] Depth-based image and audio rendering In some applications, depth-based images are combined with 3D audio fields (e.g., Dolby Atmos). In such applications, a gaze point may be used to identify one or more objects within the viewer's line of sight, and line-of-sight depth (e.g., gaze point depth) is used to evaluate a fixed depth position. An example of such a depth-based audio rendering process is shown in Figure 3, where it is used in combination with the depth-based images described above.

[0060] Audio data 41, each containing a spatial audio object consisting of an audio signal and metadata for spatial rendering, is received along with the gaze image position xG, yG and gaze depth zG determined in block 33. In block 42, at least one audio object associated with the current point of interest is identified based on the gaze image position and gaze depth.

[0061] In block 43, the audio data 41 is processed to isolate or emphasize identified audio objects (representing the audio objects being focused on). For example, audio objects closer to the current point of interest may be amplified to resolve the "cocktail party effect" of dialogue confusion. Other types of adjustments, such as changes in loudness or frequency distribution, may also be applied depending on the content creator's intent and the preferences of the audience or listener. Finally, in step 44, the processed audio data is rendered.

[0062] While the audio object processing described herein is illustrated in combination with simulated adjustment blur, it should be noted that gaze-point-based audio object processing can also be implemented independently.

[0063] Point-dependent zoom and pan Higher spatial resolution formats (4K, and especially 8K) create problems when content is viewed on various devices. For example, with 8K, the optimal viewing distance is 0.8 image height, which corresponds to an FOV of 80 degrees or more. This generally requires a very large display. When viewing the same content on a mobile phone, the FOV can drop to as low as 15 degrees. This means that all objects in the image become proportionally smaller on the retina, making it difficult to see all the features of the objects. As a simple example, facial expressions rendered for an 8K display may become unrecognizable when viewed on a mobile phone.

[0064] The solution to this problem is to zoom in on the object of interest (or region of interest, ROI). Such zooming requires a zoom factor and translation (image offset). Since most zoom functions are performed from the center of the image, no offset is needed in the special case where the ROI is already in the center of the image. However, if the object is not in the center of the image, center-based zooming can place the object of interest at the edge of the image or even completely off-screen. Making such adjustments regardless of the object's position is difficult when it needs to be done in real time, such as with video. A historical version of this problem is "pan and scan," which was used to convert content from wider-screen movie formats (16:9 aspect ratio) to previously narrower-screen television formats (4:3). Panning refers to the horizontal offset before scanning the movie into the video format. This solution was generally performed by human operators and usually did not include a zoom option. More advanced technologies that work automatically by analyzing image content have not become widespread without human supervision. While new automated algorithms exist that include zoom and offset for analyzing image content and metadata, they fail to effectively capture viewer interest. For example, ROI metadata may exist, but it is defined by the content creator. Often, viewer interest can deviate from the ROI predicted by the creator. Furthermore, existing technologies typically do not account for the changes in FOV (and thus the changes in the size of the imaged object) across the wide range of displays in today's ecosystem (from cinemas with over 80 degrees to smartwatches with less than 5 degrees).

[0065] Using the EOG-based gaze point determination discussed herein, improved viewer-determined zoom and pan can be provided. An example of such a process is shown in Figure 4.

[0066] In this process, the current gaze point 51 determined as described above is received by the gaze point monitor block 52. In block 53, the average line of sight position μ x, μ y However, for a predetermined duration t win This is determined using a sliding time window. Furthermore, the variance scale (e.g., standard deviation σ) of the average gaze position is determined. This variance measurement is referred to here as the "gaze radius" of the average gaze position and indicates the degree of focus of the viewer.

[0067] A small line of sight radius indicates that the viewer is focusing on only a small portion of the rendered image (for example, due to displaying 8K widescreen content on a mobile phone). Because human visual-spatial bandwidth and sensitivity decrease sharply as one moves away from the point of fixation, the rest of the image displayed is largely wasted, and it may be appropriate to zoom in on the image the viewer is fixating on. Conversely, a relatively large line of sight radius means the viewer is scanning a large portion of the image, and everything may be equally important to them. In such cases, zooming in on the image may not be appropriate.

[0068] In block 54, the line of sight radius is compared to a first threshold rth. The first threshold rth represents an implicit model of visual attention, greater than the orbit (~5 degrees) and less than the periorbital area (~18 degrees). If the determined line of sight radius is less than the threshold, the zoom coefficient is determined in block 55. The zoom coefficient is temporally low-pass filtered in block 56 to achieve a generally smooth and imperceptible change (except in special applications with a direct eye-based user interface). Next, in block 57, the zoom coefficient is applied to the image data 58 before it is rendered in block 59. The image data 58 may be a 2D video image, but the process in Figure 4 can also be applied to 3D image data.

[0069] Rendering is achieved using conventional methods, including other format conversions, color and tone mapping, etc.

[0070] In block 54, if the line of sight radius is determined to be greater than the threshold rth, the zoom coefficient can be maintained at its current value because viewing is a continuous process.

[0071] The process in Figure 4 also includes adjusting the spatial offset. In block 61, the distance d is equal to the minimum between the average line of sight position and one of the image edges. edge,min This is determined. This distance is determined in block 62 by the second threshold d threshold It is compared to the following. If the distance to the edge is less than the threshold, the viewer's point of interest is considered too close to the edge of the image, and an offset (in x and y) is calculated in block 63 to move the point of interest closer to the center of the displayed image. This offset is temporally low-pass filtered in block 64 to ensure a smooth, unobtrusive, and subtle change to the content. The offset is then applied in block 57 along with the zoom factor before the image data 58 is rendered in block 59.

[0072] In block 62, if the distance to the edge is determined to be greater than the threshold, the offset is maintained at the previous frame.

[0073] The offset serves at least two purposes. First, it ensures that the zoom, influenced by the applied zoom factor, does not extend beyond the image plane into the region of interest (the area around the average gaze position). Second, it limits the visibility of image boundaries to enhance immersion in the content.

[0074] Figure 4 shows two types of image metadata.

[0075] First, the ROI metadata 65 determined by the content creator / provider may indicate areas of imagery that do not align with the viewer's interests. If the creator's intent (communicated through the ROI metadata) is deemed more important than the viewer's immediate interest, the override option 66 can either replace the offset and zoom factors or blend the offset and zoom factors provided by the metadata 65 with those determined from the EOG process described above.

[0076] Secondly, scene cut metadata 67 may be used to reset the calculation of the average gaze position in block 53. This is because a scene cut results in greater eye movement as the viewer changes direction to the scene (perhaps to a new, small localized area of ​​the image). The term "scene cut" primarily refers to cases where the scene actually changes, and is different from "camera cut," which includes cases where the same scene is viewed from different perspectives, such as interactions between characters.

[0077] The process in Figure 4 does not demonstrate zooming out. Zooming out can be implemented by comparing the line of sight radius to another radius threshold. If the line of sight radius is greater than this threshold, the image is zoomed out by decreasing the zoom factor (assuming the zoom factor is greater than 1). Note that no spatial offset is required to zoom out (reduce the zoom magnification) because all content is pushed towards the center during zooming out. The other steps are the same as described above.

[0078] A special case of zooming out is a reset via scene cut metadata 67. In a scene cut, it may be appropriate to reset the zoom factor to full image view (zoom factor = 1), thereby creating a momentary zoom out.

[0079] generalization In this specification, unless otherwise specified, the ordinal numbers "first," "second," and "third" used to refer to common objects simply indicate that different instances of the same object are being referred to, and are not intended to imply that the objects described in this way must be in a predetermined order, such as temporally, spatially, or sequentially.

[0080] In the following claims and description herein, the terms “consisting of” or “composed of” are inclusive terms meaning that include at least the following elements / features, and do not exclude other elements / features. Therefore, when used in the claims, the term “consisting of” should not be interpreted as being limited only to the means, elements, or steps listed thereafter. For example, the expression “apparatus consisting of A and B” should not be limited to an apparatus consisting only of elements A and B. Similarly, as used herein, the term “including” is also inclusive terms meaning that include at least the following elements / features, and does not exclude other elements / features. Therefore, “including” is synonymous with “consisting of” and means “consisting of.”

[0081] In this specification, the term "exemplary" is used to mean "example" rather than to describe a property. That is, "exemplary embodiment" does not necessarily mean an embodiment that has an exemplary nature, but rather an embodiment given as an example.

[0082] In the above description relating to exemplary embodiments of the present invention, it should be understood that various features of the present invention may be summarized in a single embodiment, drawing, or description for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various embodiments of the invention. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires features beyond those explicitly stated in each claim. Rather, as reflected in the following claims, the embodiments of the invention have fewer features than all the features of one embodiment disclosed above. Therefore, the claims following the detailed description are explicitly incorporated into the detailed description, and each claim stands alone as an individual embodiment of the present invention.

[0083] Furthermore, some embodiments described herein may include some features included in other embodiments and may not include other features, but as will be understood by those skilled in the art, each combination of features of different embodiments is within the scope of the present invention and means that it forms another embodiment. For example, any combination of embodiments described in the following claims may be used.

[0084] Furthermore, some embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other means of performing a function. Thus, a processor having the instructions necessary to carry out such a method or method element forms means for carrying out the method or method element. Furthermore, the elements of the apparatus embodiments described herein are examples of means for realizing the functions performed by those elements for the purpose of carrying out the present invention.

[0085] Numerous specific details are described in the description provided herein. However, it should be understood that embodiments of the present invention can be carried out without these specific details. In order to avoid complicating the understanding of this specification, well-known methods, well-known structures and well-known techniques are not described in detail in other examples.

[0086] While specific embodiments of the present invention have been described, those skilled in the art will recognize that further modifications can be made without departing from the spirit of the invention. All such changes and modifications that fall within the scope of the invention are intended to be claimed. For example, the formulas shown above are merely representative examples of possible procedures. It is also possible to add or remove functions from the block diagram, or to swap operations between function blocks. It is also possible to add or remove steps to the methods described within the scope of the invention.

[0087] Various aspects of the present invention can be understood from the embodiments listed below (EEE).

[0088] EEE1. To acquire a set of voltage signals from a set of electrodes placed close to the user's ear, Based on the aforementioned set of voltage signals, the EOG line of sight vector in egocentric coordinates is determined, The user's head posture in display coordinates is determined using the sensor device worn by the user, By combining the aforementioned EOG line of sight vector and head pose, the line of sight vector in display coordinates is obtained, A method comprising determining a point of fixation by calculating the intersection of the line of sight vector and an imaging plane having a known position in display coordinates.

[0089] EEE2. The method according to EEE1, further comprising calibrating the sensor device in order to obtain the position of the sensor device in display coordinates.

[0090] EEE3. The above calibration is, Displaying graphic elements on the aforementioned imaging surface, The method of EEE2, comprising receiving user input confirming that the user is looking at the position on the imaging plane corresponding to the graphical element.

[0091] EEE4. The method according to EEE2, wherein the user-worn sensor device is synchronized with a second sensor device fixedly positioned with respect to at least a portion of the display system including the imaging surface.

[0092] EEE5. The method of EEE2, wherein the calibration includes determining the distance between the user and the imaging plane.

[0093] EEE6. The method according to EEE5, wherein the distance is determined using one or more sensors in the display system.

[0094] EEE7. The method of any of the EEEs, further comprising online statistical calibration, which includes a statistical analysis of the aforementioned gaze points over time and knowledge of expected user viewing patterns.

[0095] EEE8. The method according to any of the EEEs, wherein the egocentric coordinate system includes only one degree of freedom.

[0096] EEE9. The method according to any of the EEEs, wherein the display coordinates include only two degrees of freedom.

[0097] EEE10. The method according to any of the EEEs, wherein the display coordinates include 6 degrees of freedom.

[0098] EEE11. A method for processing image data including depth information for display on an imaging plane, The fixation point on the imaging plane is determined using the method of any of the above EEEs, Determining the line of sight depth based at least partially on the depth information associated with the point of fixation, The relative depth associated with the first portion of the image data is calculated as the difference between the depth information associated with the first portion of the image data and the line-of-sight depth. A method comprising generating modified image data by modifying pixel data associated with the first portion of the image data according to a function of the relative depth associated with the first portion of the image data.

[0099] EEE12. The method according to EEE11, wherein the modification of the pixel data includes changing one or more of the hue, brightness, gamma, and contrast of the pixel data.

[0100] EEE13. The method according to EEE11, wherein the modification of the pixel data includes changing one or more of the sharpness, blur, or spatial filtering of the pixel data.

[0101] EEE14. A method for processing audio objects associated with image data containing depth information for display on an imaging plane, The fixation point on the imaging plane is determined using the method described in any of EEE1 to 10, Determining the line of sight depth based at least partially on the depth information associated with the point of fixation, Identifying at least one audio object associated with the current point of interest, based at least partially on the point of gaze and the depth of gaze, A method comprising modifying an audio object such that the identified audio object is modified differently from other audio objects.

[0102] EEE15. The method according to EEE13, wherein modifying the identified audio object includes changing one of the volume, loudness, and frequency distribution of the identified audio object.

[0103] EEE16. The method according to EEE13 or 14, wherein the current point of interest is determined as a function of the point of gaze.

[0104] EEE17. A method for processing image data to be displayed on an imaging surface, The fixation point on the imaging plane, determined using one of the methods described in EEE1 to EEE10, is monitored over time. Determining the average line of sight position and line of sight radius, Comparing the aforementioned line of sight radius with a radius threshold, A method comprising: applying zoom to the image data in accordance with the determination that the line of sight radius is smaller than the radius threshold.

[0105] EEE18. The method according to EEE17, further comprising applying low-pass filtering of the zoom before applying it to the image data.

[0106] EEE19. Determining the minimum distance between the average line of sight position and one or more edges of the image plane, Comparing the aforementioned minimum distance with a distance threshold, The method according to EEE17 or 18, further comprising applying an offset to the image data to increase the minimum distance in accordance with the determination that the minimum distance is less than the distance threshold.

[0107] EEE20. The method according to EEE19, further comprising low-pass filtering the offset before applying it to the image data.

[0108] EEE21. The method according to EEE17, characterized in that the line of sight radius is based on the standard deviation around the average line of sight position.

[0109] EEE22. The method according to EEE17, characterized in that the average line of sight position and the line of sight radius are determined based on the change in the point of gaze within a predetermined time window.

[0110] EEE23. A pair of electrodes positioned close to the user's ear and configured to acquire a set of voltage signals, An EOG processing unit that determines the EOG line of sight vector in egocentric coordinates based on the aforementioned set of voltage signals, A user-worn sensor device that determines the user's head posture in display coordinates, A processing unit, By combining the aforementioned EOG line of sight vector and head pose, we obtain the line of sight vector in display coordinates. The point of focus is determined by calculating the intersection of the line-of-sight vector and the image plane having a known position in the display coordinates. A system including a processing unit configured in such a way.

[0111] EEE24. A non-temporary, computer-readable medium for storing computer program code configured to perform any of the steps described in EEE1-22 when executed on a computer processor.

Claims

1. This involves acquiring a set of voltage signals from a set of electrodes placed close to the user's ear, Based on the aforementioned set of voltage signals, the EOG line of sight vector in egocentric coordinates is determined, The user's head posture in display coordinates is determined using the sensor device worn by the user, By combining the aforementioned EOG line of sight vector and the aforementioned head pose, the line of sight vector in display coordinates is obtained, The point of focus is determined by calculating the intersection of the line of sight vector and the imaging plane having a known position in the display coordinates, To obtain the position of the sensor device in display coordinates, the sensor device is calibrated, Includes, A method wherein the sensor device is synchronized with a second sensor device fixedly positioned with respect to at least a portion of the display system including the imaging surface.

2. The aforementioned calibration is Displaying graphic elements on the aforementioned imaging surface, The method according to claim 1, comprising receiving user input confirming that the user is looking at the position on the imaging plane corresponding to the graphic element.

3. To acquire a set of voltage signals from a set of electrodes positioned close to the user's ear, Based on the aforementioned set of voltage signals, the EOG line of sight vector in egocentric coordinates is determined, The user's head posture in display coordinates is determined using the sensor device worn by the user, By combining the aforementioned EOG line of sight vector and the aforementioned head pose, the line of sight vector in display coordinates is obtained, The point of focus is determined by calculating the intersection of the line of sight vector and the imaging plane having a known position in the display coordinates, To obtain the position of the sensor device in display coordinates, the sensor device is calibrated, Includes, The calibration method includes determining the distance between the user and the imaging surface using one or more sensors in the display system.

4. The method according to any one of claims 1 to 3, further comprising online statistical calibration, which includes a statistical analysis of the aforementioned gaze points over time and knowledge of expected user viewing patterns.

5. A method for processing image data including depth information for display on an imaging plane, Determining the point of fixation on the imaging plane using the method described in any one of claims 1 to 3, Determining the line of sight depth based at least partially on the depth information associated with the point of fixation, The relative depth associated with the first portion of the image data is calculated as the difference between the depth information associated with the first portion of the image data and the line-of-sight depth. A method comprising generating modified image data by modifying pixel data associated with the first portion of the image data according to a function of the relative depth associated with the first portion of the image data.

6. The method according to claim 5, wherein modifying the pixel data includes changing one or more of the hue, brightness, gamma, contrast, sharpness, blur, or spatial filtering of the pixel data.

7. A method for processing audio objects associated with image data containing depth information for display on an imaging plane, Determining the point of fixation on the imaging plane using the method described in any one of claims 1 to 3, Determining the line of sight depth based at least partially on the depth information associated with the point of fixation, Identifying at least one audio object associated with the current point of interest, based at least partially on the point of gaze and the depth of gaze, A method comprising modifying the identified audio object such that it is modified differently from other audio objects.

8. The method according to claim 7, wherein modifying the identified audio object includes changing one of the volume, loudness, and frequency distribution of the identified audio object.

9. A method for processing image data to be displayed on an imaging surface, Monitoring the point of fixation on the imaging plane, determined by the method of any one of claims 1 to 3, over time, Determining the average line of sight position and line of sight radius, Comparing the aforementioned line of sight radius with a radius threshold, A method comprising: applying zoom to the image data in accordance with the determination that the line of sight radius is smaller than the radius threshold.

10. Determining the minimum distance between the average line of sight position and one or more edges of the image-forming surface, Comparing the aforementioned minimum distance with a distance threshold, The method according to claim 9, further comprising: applying an offset to the image data to increase the minimum distance in accordance with the determination that the minimum distance is less than the distance threshold.

11. The method according to claim 9, further comprising applying low-pass filtering of the zoom before applying it to the image data.

12. The method according to claim 10, further comprising low-pass filtering the offset before applying it to the image data.

13. A pair of electrodes positioned close to the user's ear and configured to acquire a set of voltage signals, An EOG processing unit that determines the EOG line of sight vector in egocentric coordinates based on the aforementioned set of voltage signals, A sensor device worn by the user that determines the user's head posture in display coordinates, By combining the EOG line of sight vector and head pose, we obtain the line of sight vector in display coordinates. The point of fixation is determined by calculating the intersection of the line-of-sight vector and the image plane having a known position in the display coordinates. To obtain the position of the sensor device in display coordinates, the sensor device is calibrated. A processing unit configured as follows, A system in which the sensor device is synchronized with a second sensor device fixedly positioned with respect to at least a portion of the display system including the imaging surface.

14. A computer program configured to perform the method described in any one of claims 1 to 3 when executed on a computer processor.

Images