Eye tracking method and device for rotating screen terminal equipment

Abstract

The present invention discloses an eye tracking method and apparatus for a rotating screen terminal device. The above method includes: determining the operating state of the rotating screen terminal device; the operating state at least includes the current device posture of the rotating screen terminal device; according to the operating state, determining a corresponding eye tracking scheme (target eye tracking scheme) from a plurality of preset eye tracking schemes; and using the target eye tracking scheme to perform eye tracking. The embodiment of the present invention is configured with a plurality of preset eye tracking schemes. After determining the real-time operating state of the rotating screen terminal device, a corresponding eye tracking scheme can be determined from a plurality of preset eye tracking schemes to perform eye tracking. Taking the device posture as an example, when the device posture changes, the embodiment of the present invention can switch between eye tracking schemes according to the change in the device posture. Compared with the use of a single eye tracking scheme, the eye tracking effect of the embodiment of the present invention under each device posture is better than the effect of using a single eye tracking scheme.

Description

Technical Field

[0001] The present invention relates to the field of eye tracking technology, and in particular to an eye tracking method and device for a rotating screen terminal device. Background Art

[0002] Most of the existing terminal-based eye tracking solutions require the terminal to be in a certain state to provide more accurate eye tracking services.

[0003] However, eye tracking solutions are gradually being applied to rotating screen terminal devices such as mobile phones and tablets, and the device postures of mobile phones, tablets and other terminal devices (or the user's posture relative to the rotating screen device) can vary. If eye tracking is only supported in a certain state, it will limit its promotion. In fact, some eye tracking solutions are completely unusable in certain device postures. Summary of the invention

[0004] In view of the above problems, the present invention provides an eye tracking method and apparatus for a rotating screen terminal device, so as to achieve the purpose of eye tracking by adaptively adjusting an eye tracking scheme.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] An eye tracking method for a rotating screen terminal device, the method comprising:

[0007] Determine the operating state of the rotating screen terminal device; the operating state at least includes the current device posture of the rotating screen terminal device;

[0008] According to the running state, a corresponding eye tracking scheme is determined from a plurality of preset eye tracking schemes; the determined eye tracking scheme is a target eye tracking scheme;

[0009] Eye tracking was performed using the target eye tracking scheme.

[0010] Optionally, the rotating screen terminal device has multiple device postures, each device posture corresponds to at least one eye tracking scheme; determining the corresponding eye tracking scheme from multiple preset eye tracking schemes according to the operating state includes: determining the target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture.

[0011] Optionally, the operating status also includes: the current load of the rotating screen terminal device; determining the target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture includes: when the current load is higher than a preset load threshold, determining the eye tracking scheme with the lowest system resource load from at least one eye tracking scheme corresponding to the current device posture.

[0012] Optionally, the operating status also includes: the remaining power of the rotating screen terminal device; determining the target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture includes: when the remaining power is lower than a preset power threshold, determining the eye tracking scheme with the smallest power demand from at least one eye tracking scheme corresponding to the current device posture.

[0013] Optionally, the running status also includes: the software type of the software currently running in the rotating screen terminal device; determining the target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture includes: determining an eye tracking scheme that meets the accuracy requirements of the software type from at least one eye tracking scheme corresponding to the current device posture.

[0014] Optionally, different eye tracking schemes differ in one or more of the following aspects: device usage strategy; eye tracking algorithm; wherein the eye tracking algorithm is used to calculate gaze information.

[0015] Optionally, the device usage strategy includes at least one of a light source usage strategy, a light source flicker strategy, a camera usage strategy, a camera frame rate setting strategy, and a camera resolution setting strategy; wherein: the light source usage strategy is used to indicate: a light source device, the power and lighting intensity of the light source device; the light source device includes a light source or a light source group; the light source flicker strategy is used to indicate: the frequency of switching different light sources in the light source group for lighting; the camera usage strategy is used to indicate: the usage status of the camera group; the camera group includes at least one target camera; the target camera includes at least one of a camera for capturing eye diagrams and a depth camera; the camera frame rate setting strategy is used to indicate: the frame rate of the target camera; the camera resolution setting strategy is used to indicate: the resolution of the target camera.

[0016] Optionally, the lighting device indicated by the light source usage strategy in the target eye tracking scheme is the target lighting device, the indicated power is the target power, and the indicated lighting intensity is the target lighting intensity; the camera usage strategy in the target eye tracking scheme is the camera usage target strategy; the light source flickering strategy in the target eye tracking scheme is the light source flickering target strategy; the eye tracking algorithm in the target eye tracking scheme is the target eye tracking algorithm; and the use of the target eye tracking scheme for eye tracking includes: controlling the target lighting device to use the target power, the target lighting intensity and the light source flickering target strategy for lighting; controlling the target camera to use the camera usage target strategy to capture images in real time; using the target eye tracking algorithm to extract feature data from the captured image, and calculate gaze information based on the feature data.

[0017] Optionally, the current device posture includes: the current screen posture of the rotating screen terminal device and the relative distance between the rotating screen terminal device and the user's face.

[0018] Optionally, the current screen posture of the rotating screen terminal device includes: the posture of the screen of the rotating screen terminal device relative to the user's face and / or the posture of the screen of the rotating screen terminal device itself; the posture of the screen of the rotating screen terminal device relative to the user's face is determined by a first posture analysis method; the posture of the screen of the rotating screen terminal device itself is determined by a second posture analysis method; the first posture analysis method has a higher priority than the second posture analysis method.

[0019] Optionally, the posture of the screen of the rotating screen terminal device relative to the user's face is determined by a first posture analysis method; the first posture analysis method includes: using a wide-area camera to capture an image containing the user's face; obtaining position data of a specific part from the image; the specific part includes: at least one of the face, pupil, forehead, nose, ear, and mouth; using the position data to calculate the relative angle between the wide-area camera and the face; and converting the relative angle into the posture of the screen relative to the user's face based on the positional relationship between the wide-area camera and the screen.

[0020] Optionally, the posture of the screen of the rotating screen terminal device itself is determined by a second posture analysis method; the second posture analysis method includes: obtaining the current orientation output by a posture determination module; and determining the posture of the screen itself according to the current orientation.

[0021] An eye tracking device for a rotating screen terminal device, comprising:

[0022] A determination unit, configured to determine an operating state of the rotating screen terminal device; the operating state at least includes a current device posture of the rotating screen terminal device;

[0023] A selection unit, configured to determine a corresponding eye tracking scheme from a plurality of preset eye tracking schemes according to the operating state; the determined eye tracking scheme is a target eye tracking scheme;

[0024] A tracking unit is used to perform eye tracking using the target eye tracking solution.

[0025] Optionally, the rotating screen terminal device has multiple device postures, each device posture corresponds to at least one eye tracking scheme; in the aspect of determining the corresponding eye tracking scheme from multiple preset eye tracking schemes according to the operating state, the selection unit is specifically used to: determine the target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture.

[0026] An eye tracking device for a rotating screen terminal device comprises: a processor, a memory and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the eye tracking method for a rotating screen terminal device as described in any one of the above items are implemented.

[0027] A storage medium storing computer executable instructions, wherein the computer executable instructions, when loaded and executed by a processor, implement the eye tracking method steps for a rotating screen terminal device as described in any of the above items.

[0028] Compared with the prior art, the embodiment of the present invention is configured with multiple preset eye tracking schemes. After determining the real-time operating state of the rotating screen terminal device (exemplarily including the current device posture), the corresponding eye tracking scheme can be determined from multiple preset eye tracking schemes to perform eye tracking, thereby achieving the purpose of adaptively adjusting the eye tracking scheme for eye tracking.

[0029] Taking the device posture as an example, when the device posture changes, the embodiment of the present invention can switch between eye tracking schemes according to the change of the device posture. In this way, compared with the use of a single eye tracking scheme, the eye tracking effect of the embodiment of the present invention under various device postures is better than that of the use of a single eye tracking scheme, which improves the user experience and promotes the promotion of eye tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0031] Figure 1a to Figure 1h A schematic diagram of camera and light source deployment provided in an embodiment of the present invention;

[0032] Figure 2 An exemplary process of an eye tracking method provided by an embodiment of the present invention;

[0033] Figure 3An exemplary process for determining the current device posture of a rotating screen terminal device provided by an embodiment of the present invention;

[0034] Figure 4 A schematic diagram of the relationship between the target surface and the length direction of a human face provided by an embodiment of the present invention;

[0035] Figure 5a to Figure 5c A schematic diagram of the relationship between the length direction of the screen and the face provided by an embodiment of the present invention;

[0036] Figure 6 Another exemplary process for determining the current device posture of a rotating screen terminal device provided in an embodiment of the present invention;

[0037] Figure 7a A schematic diagram of a user using a vertical position in a side-lying position provided by an embodiment of the present invention;

[0038] Figure 7b Another exemplary process for determining the current device posture of a rotating screen terminal device provided by an embodiment of the present invention;

[0039] Figure 7c Another exemplary process for determining the current device posture of a rotating screen terminal device provided by an embodiment of the present invention;

[0040] Figure 8 Another exemplary process of the eye tracking method provided by an embodiment of the present invention;

[0041] Figures 9a to 9c A schematic diagram of the use of the device provided in the embodiment of the present invention;

[0042] Fig.10 Another exemplary process of the eye tracking method provided by the embodiment of the present invention;

[0043] Fig.11 Another exemplary process of the eye tracking method provided by the embodiment of the present invention;

[0044] Fig.12 Another exemplary process of the eye tracking method provided by the embodiment of the present invention;

[0045] Fig.13 Another exemplary process of the eye tracking method provided by the embodiment of the present invention;

[0046] Fig.14 An exemplary structure of an eye tracking device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The terms "first" and "second" and the like in the specification and claims of the present invention and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0049] The eye tracking method and eye tracking device for a rotating screen terminal device provided in the embodiments of the present invention are applied to the field of eye tracking. Eye tracking, which may also be called line of sight tracking, is a technology that estimates the line of sight and / or gaze point of an eye by measuring eye movement.

[0050] Let’s first introduce the rotating screen terminal device: the rotating screen terminal device may include mobile phones, tablets and other terminal devices. The page displayed on the screen of the rotating screen terminal device can be rotated according to the device posture and the currently running software. The more common display modes include horizontal screen display and vertical screen display. The rotating screen terminal device generally has a built-in attitude determination module, including an inertial measurement unit (IMU) and an operation unit. The IMU is a device for measuring the three-axis attitude angle (or angular velocity) and acceleration of an object. Generally, an IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the object in the independent three axes of the carrier coordinate system, and the gyroscope detects the angular velocity signal of the carrier relative to the coordinate system, measures the angular velocity and acceleration of the object in three-dimensional space, and uses this to calculate the attitude of the object.

[0051] The eye tracking device involved in the embodiment of the present invention is deployed in a rotating screen terminal device, and can determine a suitable eye tracking scheme according to the operating state of the rotating screen terminal device, and use the determined eye tracking scheme to perform eye tracking.

[0052] An eye tracking solution includes an eye tracking algorithm and a usage strategy of a hardware device. Different eye tracking solutions differ in at least one aspect of the device usage strategy and the eye tracking algorithm.

[0053] The eye tracking algorithm is used to calculate the gaze information. Specifically, the eye tracking algorithm can calculate the gaze direction and the eyeball position, and obtain the gaze point coordinates on the screen according to the gaze direction and the eyeball position.

[0054] Further examples of hardware devices may include cameras, light sources, etc. The hardware devices are also deployed on the rotating screen terminal device, and the use of the hardware serves the eye tracking algorithm.

[0055] A set of eye tracking algorithms and corresponding hardware devices can also be regarded as an eye tracking system.

[0056] More specifically, the device usage policy may include at least one of: a light source usage policy, a light source flicker policy, a camera usage policy, a camera frame rate setting policy, and a camera resolution setting policy.

[0057] If the same eye tracking algorithm is combined with different device usage strategies (at least one of the light source usage strategy, light source flicker strategy, camera usage strategy, camera frame rate setting strategy, and camera resolution setting strategy), multiple eye tracking solutions can be obtained.

[0058] Here are some of them:

[0059] 1. Camera usage strategy:

[0060] The camera usage policy is used to indicate: the usage status of the camera group.

[0061] The camera group includes at least one target camera, which includes a camera for capturing eye images (such as an environment camera, an infrared camera) and a depth camera.

[0062] The eye image here refers to an image containing eyes, such as a frontal or side profile image, or an image containing only eyes.

[0063] Environmental cameras (wide-area cameras), depth cameras, and infrared cameras are all deployed on rotating screen terminal devices.

[0064] In one example, an infrared camera may be deployed in the middle of a short side of the screen, for example, at least one infrared camera may be deployed near a home button; meanwhile, at least one infrared camera may be deployed in the middle of a long side of the screen.

[0065] Some eye tracking algorithms (such as eye tracking algorithms based on pupil position and morphology) can calculate gaze information only by relying on images taken by environmental cameras, some eye tracking algorithms can use both environmental cameras and infrared cameras, and some eye tracking algorithms (such as gaze methods) cannot calculate gaze information using images taken by environmental cameras, and need to rely on images taken by one or more infrared cameras to calculate gaze information.

[0066] For further information, see Figure 1a ,If the selectable eye tracking algorithm includes binocular positioning to determine the eye distance, two infrared cameras need to be deployed on the same side in hardware.

[0067] Deploying two infrared cameras on the same side may also have other applications. For example, in order to obtain a larger available range, the two infrared cameras may serve different eye tracking algorithms respectively. That is, the embodiment of the present invention may support the simultaneous use of two or more eye tracking schemes to calculate gaze information. The gaze information calculated by different eye tracking schemes may be fused later to obtain a better tracking effect.

[0068] Those skilled in the art can flexibly set the number of infrared cameras deployed on the same side according to algorithm requirements, which will not be elaborated here.

[0069] Depth cameras are usually deployed near infrared cameras to obtain depth images. Depth images can be used to construct the user's head posture and serve eye tracking algorithms that need to use head posture data. Of course, depth images can also be used to obtain the distance between various parts of the face (such as the pupil) and the terminal.

[0070] The camera usage strategy paired with the eye tracking algorithm dictates which camera group or groups to use, and which target camera or cameras in the camera group to capture.

[0071] In one example, cameras can be grouped according to camera type and deployment location. For example, environmental cameras form a group (which may be called camera group 1), infrared cameras and depth cameras deployed near the home button form a camera group (which may be called camera group 2), and infrared cameras and depth cameras deployed in the middle of the long side of the screen form a camera group (which may be called camera group 3).

[0072] Of course, if all eye tracking algorithms used by the terminal do not require the use of a depth camera, then a depth camera may not be deployed.

[0073] The camera group usage status in the camera usage policy may include: the usage status of each camera group.

[0074] 0 or 1 can indicate whether it is used (0 can be understood as the first state, 1 can be understood as the second state). Taking camera group 2 as an example, if the usage status of camera group 2 is "1", it means that camera group 2 is used.

[0075] Furthermore, the usage status of the camera group may also include: the usage status of each camera in the camera group whose usage status is the second status.

[0076] Still taking camera group 2 as an example, assuming that camera group 2 includes infrared camera 1 and infrared camera 2, where the usage status of infrared camera 1 is "0" and the usage status of infrared camera 2 is "1", it means that infrared camera 1 is not used and infrared camera 2 is used, then infrared camera 2 is the target camera.

[0077] Second, camera frame rate setting strategy:

[0078] The camera frame rate setting strategy is used to indicate: the frame rate of the target camera.

[0079] The unit of frame rate is frames per second, abbreviated as FPS (Frame per Second). If the frame rate of a camera is 30FPS, it will capture 30 frames per second.

[0080] It should be noted that the algorithm frame rate can be the same as or different from the camera frame rate. For example, if the camera frame rate is set to 30FPS, the algorithm frame rate can be set to be less than or equal to 30FPS, such as 15FPS. In this case, although the camera shoots 30 frames per second, the algorithm processes 15 frames of images in 1 second.

[0081] The higher the algorithm frame rate, the more computing and storage resources are required.

[0082] 3. Camera resolution setting strategy:

[0083] The camera resolution setting strategy is used to indicate: the resolution of the target camera.

[0084] Resolution mainly affects the processor load. If the hardware bottleneck is not reached, the higher the resolution, the heavier the processor load.

[0085] In addition, if the resolution in the camera resolution setting strategy (which may be referred to as the target resolution) is smaller than the maximum resolution of the target camera, the camera resolution setting strategy may also include a specific sampling method:

[0086] For example, the maximum resolution of the target camera is 5 million pixels. If the resolution in the camera resolution setting strategy is 1 million pixels, there are different sampling methods to achieve 1 million pixels:

[0087] Method 1: Downsample the entire captured image, and the pixels of the downsampled image are the target resolution.

[0088] Still taking the maximum resolution of 5 million as an example, the 5-megapixel image can be taken at intervals through downsampling to obtain a 1-megapixel image.

[0089] Method 2: a portion of the captured image is captured, and the number of pixels of the captured portion (final image) is the same as the target resolution.

[0090] Still taking the maximum resolution of 5 million as an example, a certain area of ​​the captured 5-megapixel image can be selected, and the number of pixels in the area is 1 million.

[0091] In one example, a wide-angle lens may be used to capture an image, and the eye area may be determined as the target area. Then, the target area in the 5-megapixel image may be selected as the final image.

[0092] Fourth, light source usage strategy:

[0093] The light source usage strategy is used to indicate the light source device, the power of the light source device, and the lighting intensity.

[0094] The light source device here includes an (infrared) light source or an (infrared) light source group.

[0095] For some eye tracking algorithms based on pupil-corneal reflection (PCCR), infrared cameras and infrared light sources are required, and infrared light will not affect the vision of the eyes.

[0096] Its working principle is: infrared light source shines on the eye, forming a reflection point on the cornea, namely a light spot (also called Purkinje spot). The infrared camera takes a picture of the eye and obtains an eye image with the light spot.

[0097] As the eyeball rotates, the relative position relationship between the pupil center and the light spot will change accordingly. Correspondingly, several eye images with light spots collected by the infrared camera will also reflect such position change relationship. The line of sight / fixation point can be estimated based on the position change relationship.

[0098] The infrared light source group also serves the eye tracking algorithm, and its deployment location also needs to consider the needs of the eye tracking algorithm.

[0099] For example, among the eye tracking algorithms based on pupil-corneal reflection (PCCR), some use the bright-dark pupil technique. The bright-dark pupil technique requires controlling the light source to alternately turn on and off to produce a video sequence in which the bright pupil and the dark pupil appear alternately every other frame. The pupil can be effectively extracted by taking the difference between adjacent bright pupil and dark pupil images and detecting the pupil in the thresholded difference image.

[0100] The infrared light source group serving the eye tracking algorithm using bright and dark pupil technology can be deployed as follows:

[0101] See also Figure 1a, multiple infrared light sources a in a group of infrared light source groups are deployed around the infrared camera in a circle to form a stable bright pupil light spot. The infrared light source a surrounding the infrared camera can be called a bright pupil infrared light source group; a group of infrared light source groups (can be called a dark pupil infrared light source group) or an infrared light source b (can be called a dark pupil infrared light source) is deployed at a position far away from the infrared camera (for example, the upper left, lower left, upper right, and lower right corners of the screen) to form a stable dark pupil light spot.

[0102] The light source usage strategy can indicate the usage status of the light source device. It can be 0 or 1 to indicate whether it is used (0 can be understood as the first state, 1 can be understood as the second state). Taking an infrared light source as an example, if its usage status is "1", it means that the infrared light source is used.

[0103] As for the light source group, the light source usage policy may indicate the usage status of the light source group and the usage status of each light source in the light source group.

[0104] For example, assuming there are two light source groups (light source group 1 and light source group 2), the use status of light source group 1 is 0, and the use status of light source group 2 is 1, which means that light source group 1 is not used and light source group 2 is used.

[0105] Furthermore, the light source group 2 includes three infrared light sources 1-3, and their use states are respectively "010", which means that the infrared light sources 1 and 3 are not used, and the infrared light source 2 is used.

[0106] There are many combinations of light source usage strategies and camera usage strategies to meet the hardware requirements of different eye tracking algorithms.

[0107] For example, Figure 1b or Figure 1c The combinations shown include: a single (infrared) camera, a bright-pupil infrared light source group surrounding the single camera, a left-right symmetrical dark-pupil infrared light source group, and a wide-angle camera that does not use light source illumination (in order to highlight the combination, other cameras and infrared light sources that are not involved in the work deployed on the rotating screen terminal device are not shown in the figure, and the same applies to the following);

[0108] For another example, Figure 1d or Figure 1e The combinations shown include: a single (infrared) camera, a bright-pupil infrared light source group surrounding the single camera, a left-right symmetrical dark-pupil infrared light source group, and a depth camera that does not use light source illumination;

[0109] For another example, Figure 1f A single environment camera that does not require light source illumination is shown;

[0110] For another example, Figure 1g or Figure 1hThe combination shown is: dark pupil infrared light source group 1, infrared camera 1, infrared camera 2, and dark pupil infrared light source group 2 arranged in the same direction, with the four being close in distance and occupying one side.

[0111] 5. Light source flickering strategy:

[0112] The light source flashing strategy is used to indicate: the frequency and switching order of switching different light sources in the light source group for lighting.

[0113] For some eye tracking algorithms that use bright-dark-pupil technology, it is necessary to set the frequency and switching order of switching different light sources for illumination.

[0114] Still taking the three infrared light sources 1-3 in the light source group 2 as an example, the switching sequence 123 and the switching sequence 132 are different light source flashing strategies.

[0115] The embodiments of the present invention will be further described in detail below based on the common aspects of the present invention described above.

[0116] Embodiment 1

[0117] To solve the above problems, in the first embodiment of the present application, an eye tracking method for a rotating screen terminal device is provided to adaptively adjust the eye tracking scheme for eye tracking. Figure 2 , the eye tracking method for a rotating screen terminal device includes:

[0118] S1: Determine the operating status of the rotating screen terminal device.

[0119] The above-mentioned operation status may include the current device posture, current load, remaining power, software type of the currently running software, etc. of the rotating screen terminal device. The above-mentioned current device posture, current load, remaining power, software type of the currently running software can also be understood as operation status parameters.

[0120] S2: According to the above operating status, a corresponding eye tracking scheme is determined from a plurality of preset eye tracking schemes.

[0121] For the sake of convenience, the determined eye tracking scheme may be referred to as a target eye tracking scheme.

[0122] S3: Eye tracking was performed using the target eye tracking scheme.

[0123] Compared with the prior art, the embodiment of the present invention is configured with multiple preset eye tracking schemes. After determining the real-time operating state of the rotating screen terminal device (exemplarily including the current device posture), the corresponding eye tracking scheme can be determined from multiple preset eye tracking schemes to perform eye tracking, thereby achieving the purpose of adaptively adjusting the eye tracking scheme for eye tracking.

[0124] Embodiment 2

[0125] In the first embodiment, it is introduced that the operating state may include the current device posture of the rotating screen terminal device.

[0126] The current device posture may specifically include: the current screen posture of the rotating screen terminal device and the relative distance between the rotating screen terminal device and the user's face.

[0127] The current screen posture of the rotating screen terminal device includes: at least one of the posture of the screen of the rotating screen terminal device relative to the user's face and the posture of the screen of the rotating screen terminal device itself.

[0128] The current screen posture can be determined in a number of ways.

[0129] Optionally, the posture of the screen of the rotating screen terminal device relative to the user's face is determined by a first posture analysis method;

[0130] Optionally, the posture of the screen of the rotating screen terminal device itself is determined by a second posture analysis method; the first posture analysis method has a higher priority than the second posture analysis method.

[0131] The following will specifically introduce the first posture analysis method, the second posture analysis method, the priority relationship between them, and the specific method for determining the relative distance between the rotating screen terminal device and the user's face according to the embodiments.

[0132] Embodiment 3

[0133] Figure 3 That is, an exemplary process for determining the current device posture of the rotating screen terminal device may include:

[0134] S31: Use a wide-area camera to capture an image including the user's face.

[0135] The wide-area camera is deployed on the rotating screen terminal device, which may exemplarily include a front camera of the rotating screen terminal device.

[0136] S32: Acquire location data of a specific part from the captured image.

[0137] The specific part may include at least one of: face, pupil, forehead, nose, ear and mouth.

[0138] S33: Calculate the relative angle between the wide-area camera and the face using the above position data.

[0139] Specifically, the angle between the direction from the center of the target surface of the image sensor in the wide-area camera to the optical center and the length direction of the face can be calculated as the relative angle between the wide-area camera and the face.

[0140] Figure 4The orthogonal directions of the target plane are set as the x-axis and the y-axis, wherein the rightward direction of the target plane is set as the positive direction of the x-axis, the upward direction of the target plane is set as the positive direction of the y-axis, and the direction perpendicular to the xy plane and pointing forward (or backward) is set as the positive direction of the z-axis. The length direction of the face in this coordinate system can be represented by a (x, y, z) direction vector.

[0141] Similarly, the direction from the center of the target surface to the optical center can also be represented by a (x, y, z) direction vector, and the angle between these two direction vectors can be calculated.

[0142] In one example, the length direction of the face may be the direction from the lowest point of the chin along the bridge of the nose to the forehead. The length direction of the face may be determined at least based on the positions of the pupils of both eyes and the nose, and may also refer to the positions of the forehead, mouth, and ears.

[0143] S34: According to the positional relationship between the wide-area camera and the screen, the relative angle is converted into a posture of the screen relative to the user's face.

[0144] Since the position of the wide-area camera on the rotating screen terminal device is fixed, the positional relationship between the wide-area camera and the screen is also fixed. Therefore, the relative angle between the wide-area camera and the face can be converted into the posture of the screen relative to the user's face.

[0145] In one example, the posture of the screen relative to the user's face may include:

[0146] 1. Vertical posture:

[0147] The vertical posture means that the long side of the screen is parallel to the length of the face (from the chin to the forehead).

[0148] For example, when the user is in an upright position using portrait mode, the screen is in a vertical position relative to the user's face; or, when the user is in a side-lying position (with the head and torso basically in a straight line) using landscape mode, the screen is in a vertical position relative to the user's face.

[0149] Among them, the above-mentioned vertical screen mode is a display form in which the long side of the screen of the rotating screen terminal device is perpendicular to the ground; the above-mentioned horizontal screen mode is a display form in which the long side of the screen of the rotating screen terminal device is parallel to the ground.

[0150] See also Figure 5a , set the upward direction of the long side of the screen as the positive direction of the y-axis, the rightward direction of the short side as the positive direction of the x-axis, set the direction perpendicular to the xy plane and forward (or backward) as the positive direction of the z-axis, and the length direction of the face in this coordinate system can be represented by a (x, y, z) direction vector.

[0151] The angle between the long side of the screen and the length direction of the face can be in any of the four quadrants. In one example, if the angle between the long side of the screen and the length direction of the face is near 0 degrees or near 180 degrees, the long side can be regarded as parallel to the length direction of the face.

[0152] The so-called nearby, taking 0 degree as an example, a threshold can be designed. If the difference between the angle and 0 degree is less than or equal to the threshold, it can be determined to be nearby 0 degree.

[0153] The situation around 180 degrees is similar and will not be elaborated here.

[0154] 2. Horizontal posture:

[0155] The horizontal posture means that the entire screen is parallel to the length of the face, and the long side of the screen is perpendicular to the length of the face.

[0156] For example, when a user uses the landscape mode in an upright position, the screen is in a landscape orientation relative to the user's face.

[0157] In an example, see Figure 5b If the angle between the long side of the screen and the length direction of the face is around 90 degrees or 270 degrees, the long side can be regarded as perpendicular to the length direction of the face.

[0158] The so-called nearby determination can be found in the above introduction and will not be elaborated here.

[0159] 3. Oblique posture:

[0160] See also Figure 5c The oblique posture means that the entire screen is parallel or not parallel to the length direction of the face, and the long side of the screen is not parallel and not perpendicular to the length direction of the face. For example, the angle between the long side of the screen and the length direction of the face is 45 degrees (abbreviated as oblique).

[0161] It should be noted that the vertical posture, horizontal posture and diagonal posture can all be represented by the angle between the long side of the screen and the length direction of the face. For example, an angle of 0 or 180 degrees represents a vertical posture, an angle of 90 or 270 degrees represents a vertical posture, and other angles represent a diagonal posture.

[0162] S35: Calculate the relative distance between the rotating screen terminal device and the user's face.

[0163] There are many ways to calculate the relative distance between the rotating screen terminal device and the user's face. For example, the relative distance between the rotating screen terminal device and the user's face is calculated using the depth image introduced above.

[0164] For another example, a distance sensor provided on the rotating screen terminal device is used to emit a light pulse, and the time from the emission of the light pulse to its reflection by an object is measured, and the relative distance between the rotating screen terminal device and the face is calculated by measuring the time interval.

[0165] Another example is the calculation of relative distance based on the mathematical principle of binocular stereo vision: two images of the face are obtained from two cameras at the same time from different angles, and a triangle is formed between the image planes of the two cameras and the face. Given the positional relationship between the two cameras, the three-dimensional coordinates of each point on the face in the camera coordinate system can be calculated based on the triangular geometric relationship, and then the relative distance can be calculated based on the three-dimensional coordinates.

[0166] Those skilled in the art may also choose other methods of calculating the relative distance between the rotating screen terminal device and the face, which will not be elaborated here.

[0167] Steps S31 - S35 may be referred to as a first posture analysis method.

[0168] S36: Determine a target eye tracking scheme from a plurality of preset eye tracking schemes according to the posture of the screen relative to the user's face (ie, the current screen posture) and the relative distance.

[0169] This article will later describe in detail how to determine the target eye tracking solution based on the posture and relative distance of the screen relative to the user's face.

[0170] S37: Eye tracking was performed using the target eye tracking scheme.

[0171] In this embodiment, the posture of the screen of the rotating screen terminal device relative to the user's face is determined by taking pictures and analyzing image data, which can adapt to scenarios where the user uses different screen modes (horizontal screen, vertical screen) in various states (upright, sideways).

[0172] When the posture changes, the embodiment of the present invention can switch between eye tracking schemes according to the change of posture. In this way, compared with the use of a single eye tracking scheme, the eye tracking effect of the embodiment of the present invention under various device postures is better than that of the use of a single eye tracking scheme, which improves the user experience and promotes the promotion of eye tracking.

[0173] Embodiment 4

[0174] In the second embodiment, a first posture analysis method is introduced to determine the current device posture. This embodiment introduces another posture analysis method to determine the current device posture. Figure 6 An exemplary process is shown to include:

[0175] S61: Obtain the current position output by the attitude determination module.

[0176] The attitude determination module is used to calculate the current position of the rotating screen terminal device relative to the fixed coordinate system. The fixed coordinate system is a user-defined coordinate system that can be defined as needed, for example: the positive direction of the Y axis is parallel to the ground and due north, the positive direction of the X axis is due east, and the direction perpendicular to the ground and away from the ground is the positive direction of the Z axis.

[0177] The attitude determination module includes an inertial measurement unit (IMU), which is a device built into the rotating screen terminal device to measure the three-axis attitude angle (or angular velocity) and acceleration of the device. Generally, an IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometer detects the acceleration signal of the object in the independent three-axis of the carrier coordinate system, and the gyroscope detects the angular velocity signal of the carrier relative to the coordinate system, measures the angular velocity and acceleration of the object in three-dimensional space, and uses this to calculate the attitude of the object, that is, the current position.

[0178] S62: Determine the posture of the screen itself according to the current position.

[0179] Specifically, the posture determination module built into the rotating screen terminal device can monitor the current placement posture of the device. For example, relative to the ground (assuming the ground is a plane), when the screen of the rotating screen terminal device is perpendicular to the ground and the angle between the short side and the ground is around 0 degrees or 180 degrees, the posture of the device screen itself is determined to be a vertical posture; when the screen of the rotating screen terminal device is perpendicular to the ground and the angle between the short side and the ground is around 90 degrees or 270 degrees, the posture of the device screen itself is determined to be a horizontal posture; when the screen of the rotating screen terminal device is perpendicular to the ground and the angle between the short side and the ground is around 45 degrees or 135 degrees, the posture of the device screen itself is determined to be a diagonal posture. For the so-called nearby, taking 0 degrees as an example, a threshold can be designed. If the difference between the angle and 0 degrees is less than or equal to the threshold, it can be determined to be near 0 degrees.

[0180] The angles around 90 degrees, 180 degrees and 270 degrees are similar and will not be described in detail here.

[0181] It can also be defined that: when the screen of the rotating terminal device is perpendicular to the ground, if the device is neither in the horizontal nor in the vertical posture, the device is determined to be in the oblique posture.

[0182] S63: Calculate the relative distance between the rotating screen terminal device and the user's face.

[0183] There are many ways to calculate the relative distance between the rotating screen terminal device and the user's face, which will not be described in detail here.

[0184] Steps S61 - S63 may be referred to as a second posture analysis method.

[0185] S64: According to the posture of the screen itself (ie, the current screen posture) and the relative distance, a target eye tracking scheme is determined from a plurality of preset eye tracking schemes.

[0186] This article will later describe in detail how to determine the target eye tracking solution based on the posture and relative distance of the screen relative to the user's face.

[0187] S65: Eye tracking using the target eye tracking solution.

[0188] In this embodiment, the use posture is determined directly based on the data output by the posture determination module. Compared with the photo taking method in the third embodiment, the analysis method of the posture determination module does not require taking and analyzing pictures. When the user is standing upright, the calculation result is more accurate, so there is no need to install other calculation software on the terminal device, which is more convenient and saves computing power.

[0189] Embodiment 5

[0190] Although the analysis method of the posture determination module is more convenient, it is not applicable in some scenarios. For example, see Figure 7a When the user uses the terminal device in a side-lying position (the head and torso are basically in a straight line and parallel to the ground) in a vertical posture (the long side of the terminal screen is parallel to the length of the face), the screen is determined to be in a horizontal posture (because the short side of the screen is perpendicular to the ground) based on the current orientation output by the posture determination module, but it should actually be in a vertical posture (because the long side of the screen is parallel to the length of the face).

[0191] To this end, when the device has both the first posture analysis method and the second posture analysis method, this embodiment limits the priority of the first posture analysis method and the second posture analysis method to determine the current device posture of the rotating screen terminal device. Figure 7b , which exemplarily includes the following process:

[0192] S71: If the rotating screen terminal device has both the first posture analysis method and the second posture analysis method, determining the current screen posture of the rotating screen terminal device according to the first posture analysis method;

[0193] As mentioned above, the posture determination module is not applicable in some special scenarios. Therefore, if the rotating screen terminal device has both the first posture analysis method and the second posture analysis method, the first posture determination method is more reliable, so the first posture analysis method will be used preferentially to determine the current screen posture of the rotating screen terminal device.

[0194] S72: Calculate the relative distance between the rotating screen terminal device and the user's face.

[0195] There are many ways to calculate the relative distance between the rotating screen terminal device and the user's face. For example, the relative distance between the rotating screen terminal device and the face is calculated using the depth image introduced above.

[0196] For another example, a distance sensor provided on the rotating screen terminal device is used to emit a light pulse, and the time from the emission of the light pulse to its reflection by an object is measured, and the relative distance between the rotating screen terminal device and the face is calculated by measuring the time interval.

[0197] Another example is the calculation of relative distance based on the mathematical principle of binocular stereo vision: two images of the face are obtained from two cameras at the same time from different angles, and a triangle is formed between the image planes of the two cameras and the face. Given the positional relationship between the two cameras, the three-dimensional coordinates of each point on the face in the camera coordinate system can be calculated based on the triangular geometric relationship, and then the relative distance can be calculated based on the three-dimensional coordinates.

[0198] Those skilled in the art may refer to existing methods to calculate the relative distance between the rotating screen terminal device and the face, which will not be elaborated here.

[0199] S73: Determine a target eye tracking scheme from a plurality of preset eye tracking schemes according to the current screen posture and relative distance of the rotating screen terminal device.

[0200] This article will later describe in detail how to determine the target eye tracking solution based on the posture and relative distance of the screen relative to the user's face.

[0201] S74: Eye tracking using the target eye tracking solution.

[0202] It can be seen that in this embodiment, when the device has both the first posture analysis method and the second posture analysis method, the first posture analysis method is used preferentially to determine the current screen posture of the rotating screen terminal device, which can adapt to the scenarios where users use different screen modes (horizontal screen, vertical screen) in multiple states (upright, sideways).

[0203] Embodiment 6

[0204] In addition to giving priority to the first posture analysis method, another processing method can be used to reflect the priority of the first posture analysis method. See Figure 7c , which exemplarily includes the following process:

[0205] S701: Determine the current screen posture (which may be referred to as the first posture) using a first posture analysis method;

[0206] For how to determine, please refer to the description of the aforementioned embodiment 3, which will not be elaborated here.

[0207] S702: using a second posture analysis method to determine the current screen posture (which may be referred to as the second posture);

[0208] For how to determine, please refer to the description of the aforementioned fourth embodiment, which will not be elaborated here.

[0209] S703: Compare the first posture and the second posture.

[0210] S704: If the first posture and the second posture are the same, determine the second posture as the final current screen posture.

[0211] When the two are the same, the data obtained by the second posture analysis method is more accurate, so the second posture shall prevail.

[0212] S705: If the first posture and the second posture are different, determine the first posture as the final current screen posture.

[0213] This reflects the priority of the first posture analysis method.

[0214] S706: Calculate the relative distance between the rotating screen terminal device and the user's face.

[0215] There are many ways to calculate the relative distance between the rotating screen terminal device and the user's face, which will not be described in detail here.

[0216] S707: Determine a target eye tracking scheme from a plurality of preset eye tracking schemes according to the final current screen posture and relative distance.

[0217] This article will later introduce in detail how to determine the target eye tracking solution based on the current posture and relative distance of the screen.

[0218] S708: Perform eye tracking using a target eye tracking solution.

[0219] It can be seen that in this embodiment, the advantages of the two analysis methods are combined to make the obtained posture more accurate.

[0220] Embodiment 7

[0221] This embodiment will focus on how to determine the target eye tracking solution according to the current device posture of the rotating screen terminal device.

[0222] See also Figure 8 , which may include the following steps:

[0223] S81: Determine the current screen posture and current relative distance of the rotating screen terminal device.

[0224] For how to determine the current screen posture and the current relative distance, please refer to Embodiments 2 to 6, which will not be described in detail here.

[0225] S82: Determine a target eye tracking scheme that is compatible with the current relative distance and the current screen posture from a plurality of preset eye tracking schemes.

[0226] As mentioned above, different eye tracking solutions differ in at least one aspect, namely, the device usage strategy and the eye tracking algorithm. Therefore, in the process of determining the target eye tracking solution, the eye tracking algorithm and the device usage strategy are determined. The following are introduced one by one:

[0227] When determining the eye tracking algorithm, some eye tracking algorithms have their appropriate distances. If the distance is too close or too far, the effect will not be optimal.

[0228] Therefore, in one example, an eye tracking algorithm that matches the calculated current relative distance may be determined from a plurality of eye tracking algorithms according to the calculated current relative distance.

[0229] For example, assuming that the calculated relative distance is A, and the relative distance range of the eye tracking algorithm ac matches it, that is, the relative distance A falls within the relative distance range of the eye tracking algorithm ac, then the eye tracking algorithm ac is an alternative algorithm, and the final eye tracking algorithm to be adopted can be further determined from the eye tracking algorithm ac.

[0230] When determining the final eye tracking algorithm, it can be determined randomly, or priorities can be set for each eye tracking algorithm, and alternative algorithms can be selected from the list according to the priority from high to low, or it can be determined according to other operating status parameters (which will be introduced later in this article).

[0231] As for the device usage strategy, this embodiment focuses on the camera usage strategy and the light source usage strategy.

[0232] Device usage strategies serve the eye tracking algorithm.

[0233] Taking the vertical posture as an example, for the eye tracking algorithm based on the pupil-corneal reflection method (PCCR), using a camera and light source deployed at a certain position has a better shooting effect and is easier to form a light spot compared to cameras and light sources deployed at other positions.

[0234] For example, taking the eye movement algorithm based on bright pupil technology as an example, in the vertical posture, see Figure 9a , if the current angle between the length direction of the face and the long side of the screen is 0 degrees, selecting infrared camera 1, dark pupil infrared light source group 3, dark pupil infrared light source group 4 and bright pupil infrared light source group 1 is less likely to be blocked by the eyelids, and it is easier to capture eye images with light spots.

[0235] Similarly, see Figure 9b,If the current angle between the length direction of the face and the long side of the screen is 180 degrees, selecting infrared camera 2, dark pupil infrared light source group 1, dark pupil infrared light source group 2 and bright pupil infrared light source group 2 is less likely to be blocked by the eyelids and it is easier to capture eye images with light spots.

[0236] For another example, in landscape orientation, see Fig.9c , if the current angle between the length direction of the face and the long side of the screen is 90 degrees, selecting infrared camera 4, dark pupil infrared light source group 1, dark pupil infrared light source group 3 and bright pupil infrared light source group 4 is less likely to be blocked by the eyelids and it is easier to capture eye images with light spots.

[0237] Similarly, if the current angle between the length direction of the face and the long side of the screen is 270 degrees, selecting infrared camera 3, dark pupil infrared light source group 2, dark pupil infrared light source group 4 and bright pupil infrared light source group 3 is less likely to be blocked by eyelids and it is easier to capture eye images with light spots.

[0238] For oblique postures, the current angle between the long side of the screen and the length of the face can be used to determine which angle is closer to 0, 180, 80, or 270 degrees. Then, the corresponding camera and infrared light source group can be determined based on the closest angle.

[0239] Of course, for eye tracking algorithms that only use an environment camera, infrared illumination is not necessary.

[0240] S83: Eye tracking was performed using the target eye tracking scheme.

[0241] It should be noted that, taking the vertical posture and the horizontal posture as an example, the resolution of the images taken in the vertical posture and the horizontal posture are different, and the user's eye characteristic data are also different in the vertical posture and the horizontal posture.

[0242] Therefore, the eye tracking algorithm finally adopted needs to fine-tune the algorithm parameters (such as resolution and eye characteristic data) according to the current posture.

[0243] It can be seen that in this embodiment, after obtaining the current posture and current relative distance of the screen relative to the user's face, a target eye tracking scheme that is compatible with the above relative distance and posture can be determined from multiple preset eye tracking schemes. In this way, if the posture and / or relative distance obtained this time changes compared to the posture and relative distance obtained last time, this embodiment can switch between eye tracking schemes based on the change. In this way, compared with the use of a single eye tracking scheme, the eye tracking effect of the embodiment of the present invention under various device postures is better than that of a single eye tracking scheme, which improves the user experience and can promote the promotion of eye tracking.

[0244] Embodiment 8

[0245] This embodiment will focus on how to determine the target eye tracking solution according to the current load of the rotating screen terminal device.

[0246] See also Fig.10 , which may exemplarily include the following steps:

[0247] S101: Determine the current device posture of the rotating screen terminal device, and determine at least one eye tracking solution (alternative solution) adapted to the current device posture.

[0248] How to determine the current device posture and determine at least one eye tracking solution adapted to the current device posture can be found in the aforementioned S81 and S82, which will not be elaborated here.

[0249] S102: Determine the current load of the rotating screen terminal device.

[0250] In an example, the current load may specifically refer to the load of a controller (eg, a CPU) of the rotating screen terminal device, and the load may be measured by the CPU occupancy rate.

[0251] Steps S102 and S101 may be executed in parallel or sequentially.

[0252] S103: When the current load is higher than a preset load threshold, an eye tracking scheme with the lowest system resource load is determined as a target eye tracking scheme from at least one eye tracking scheme (alternative scheme) corresponding to the current device posture.

[0253] Assuming that the preset load threshold is 0.7, when the current load is higher than 0.7, an eye tracking solution with the lowest system resource load is determined from the alternative solutions.

[0254] Specifically, the higher the algorithm frame rate, the greater the impact on the load. Therefore, a solution with the lowest algorithm frame rate can be selected from the alternative solutions as the target eye tracking solution.

[0255] Specifically, an eye tracking algorithm with the lowest frame rate can be selected from a variety of algorithm frame rates, and a corresponding camera use strategy and light source use strategy can be determined to obtain a target eye tracking solution.

[0256] Of course, if the current load does not exceed the preset load threshold, you can select the preset algorithm frame rate or the eye tracking algorithm with the best performance.

[0257] In addition, camera resolution, camera frame rate, algorithm accuracy requirements, etc. will also affect the CPU load, so you can consider them comprehensively and choose the eye tracking solution with the lowest load.

[0258] S104: Perform eye tracking using a target eye tracking solution.

[0259] S104 is the same as the aforementioned S83 and will not be described in detail here.

[0260] It can be seen that in this embodiment, the current device posture and the current load are combined to determine the target eye tracking scheme from multiple preset eye tracking schemes. If the current load exceeds the load threshold (that is, the load is too heavy), the eye tracking scheme with the lowest load can be selected as the target eye tracking scheme to reduce the device load as much as possible.

[0261] Embodiment 9

[0262] This embodiment will focus on how to determine the target eye tracking solution according to the remaining power of the rotating screen terminal device.

[0263] See also Fig.11 , which may exemplarily include the following steps:

[0264] S111: Determine a current device posture of a rotating screen terminal device, and determine at least one eye tracking solution adapted to the current device posture.

[0265] How to determine the current device posture and determine at least one eye tracking solution adapted to the current device posture can be found in the aforementioned S81 and S82, which will not be elaborated here.

[0266] S112: Determine the remaining power of the rotating screen terminal device.

[0267] The remaining power can be expressed as a percentage.

[0268] Steps S112 and S111 may be executed in parallel or sequentially.

[0269] S113: When the remaining power is lower than a preset power threshold, determine the eye tracking solution with the lowest power demand from at least one eye tracking solution (alternative solution) corresponding to the current device posture as a target eye tracking solution.

[0270] Assuming that the preset power threshold is 30%, when the remaining power is less than 30%, an eye tracking solution with the lowest power demand is determined from the alternative solutions.

[0271] Specifically, the use of infrared light sources and cameras has a greater impact on power consumption. Therefore, in one example, a solution with the least infrared light sources and the least number of target cameras can be selected from the alternative solutions as the target eye tracking solution. Alternatively, if there is an eye tracking solution using an environmental camera among the alternative solutions, the eye tracking solution using an environmental camera can be used as the target eye tracking solution.

[0272] In addition, factors that affect the CPU load, such as camera resolution, camera frame rate, algorithm processing frame rate, and algorithm accuracy requirements, will also affect power consumption. Therefore, comprehensive considerations should be made to select the eye tracking solution with the lowest power consumption.

[0273] Of course, if the remaining power is greater than the preset power threshold, the optimal target eye tracking solution can be determined based on other parameters (such as current load) and priority by randomly selecting from the alternative solutions, or selecting from high to low.

[0274] How to determine the target eye tracking solution based on the current load can be found in the previous description of this article and will not be elaborated here.

[0275] S114: Perform eye tracking using the target eye tracking solution.

[0276] S114 is the same as the aforementioned S83 and will not be described in detail here.

[0277] It can be seen that in this embodiment, the target eye tracking scheme is determined to be suitable from multiple preset eye tracking schemes in combination with the current device posture and the remaining power. If the remaining power is lower than the preset power threshold (i.e., the remaining power is too low), the eye tracking scheme with the lowest power consumption can be selected as the target eye tracking scheme to minimize power consumption.

[0278] Embodiment 10

[0279] This embodiment will focus on how to determine the target eye tracking solution according to the software type of the currently running software of the rotating screen terminal device.

[0280] See also Fig.12 , which may exemplarily include the following steps:

[0281] S121: Determine the current device posture of the rotating screen terminal device, and determine at least one eye tracking solution adapted to the current device posture.

[0282] How to determine the current device posture and determine at least one eye tracking solution adapted to the current device posture can be found in the aforementioned S81 and S82, which will not be elaborated here.

[0283] S122: Determine the software type of the software currently running in the rotating screen terminal device.

[0284] In one example, the software types may include: games, reading, browsers, learning, or videos, etc.

[0285] Alternatively, the software type may be represented by the level of positioning accuracy of the software, for example, divided into low, medium, and high accuracy level types.

[0286] Steps S122 and S121 may be executed in parallel or sequentially.

[0287] S123: From at least one eye tracking solution (alternative solution) corresponding to the current device posture, determine an eye tracking solution that meets the accuracy requirement of the above software type as a target eye tracking solution.

[0288] In an example, taking the software types including games, reading, browsers, etc., generally speaking, games and reading have higher requirements for positioning accuracy, while browsing with a browser has lower requirements for positioning accuracy. Therefore, the target eye tracking solution can be determined according to the accuracy requirements of different software types.

[0289] In another example, the software may be classified not according to its nature but according to its accuracy requirement level, that is, using the accuracy requirement level as the software type.

[0290] Quantitative indicators may be used to characterize different accuracy requirement levels, and exemplary indicators may include: the angle between the calculated sight line direction and the actual sight line direction, icon level, and the like.

[0291] Taking the included angle as an example, different accuracy requirement levels can be represented by 0.5 degrees, 2 degrees, etc.

[0292] Taking the icon level as an example, the number of icons that can be deployed on the screen represents different accuracy requirements. For example, 4*5 (4 icons are placed in the horizontal direction and 5 icons are placed in the vertical direction), 8*10, etc. represent different accuracy requirements.

[0293] Accordingly, the eye tracking algorithm and the corresponding device usage strategy that meet the accuracy requirement level of the currently running software can be determined.

[0294] In addition to considering the software type, if the remaining power is lower than the preset power threshold, you can also select an eye tracking algorithm and device usage strategy with low power consumption on the hardware based on the remaining power. Or, if the load is too heavy, you can also select an eye tracking algorithm and device usage strategy with the smallest load on the hardware, etc. I will not go into details here.

[0295] S124: Perform eye tracking using the target eye tracking solution.

[0296] S124 is the same as the aforementioned S83 and will not be described in detail here.

[0297] It can be seen that in this embodiment, in combination with the current device posture and the accuracy requirements of the currently running software, an adapted target eye tracking scheme is determined from multiple preset eye tracking schemes to achieve a better user experience.

[0298] In other embodiments of the present invention, a priority may be set for each operating status parameter, and a target eye tracking solution may be determined according to the priority.

[0299] Exemplarily, assuming that the accuracy requirement of the software type has the highest priority, the adaptation alternatives may be determined first according to the accuracy requirement of the software type of the currently running software.

[0300] The current device posture has the second highest priority, and the range of alternative solutions can be further narrowed down from the alternative solutions according to the current device posture, or the target eye tracking solution can be determined from the alternative solutions according to the current device posture.

[0301] The remaining power is second. If the remaining power is too low, the solution with the lowest power consumption is selected from the alternative solutions that meet the accuracy requirements and the current device posture.

[0302] The load is second. If there is no problem of low remaining power but a problem of excessive load, the solution with the lowest load can be selected from the alternative solutions that meet the accuracy requirements and the current device posture.

[0303] Priorities are set according to the operating status parameters of the device, so as to determine the appropriate target eye tracking solution from multiple preset eye tracking solutions, making the target eye tracking solution more intelligent and effective, so as to achieve a better user experience.

[0304] Embodiment 11

[0305] This embodiment will focus on how to use the target eye tracking solution to perform eye tracking. Fig.13 An exemplary process is shown to include:

[0306] S131: Determine the operating status of the rotating screen terminal device.

[0307] For related content, please refer to the previous introduction of this article and I will not go into details here.

[0308] S132: According to the above operating status, a target eye tracking scheme is determined from a plurality of preset eye tracking schemes.

[0309] For related content, please refer to the previous introduction of this article and I will not go into details here.

[0310] For the sake of convenience, the lighting device indicated by the device usage strategy in the target eye tracking solution is called the target lighting device, the indicated power is the target power, and the indicated lighting intensity is the target lighting intensity;

[0311] The camera usage strategy in the target eye tracking scheme is called the camera usage target strategy, the light source flicker strategy is called the light source flicker target strategy, and the eye tracking algorithm is called the target eye tracking algorithm.

[0312] S133: Control the target lighting device to perform lighting using a target power, a target lighting intensity, and a light source flicker target strategy;

[0313] For example, the target light sources include infrared light source 1 and infrared light source 2, the target power is A, the target lighting intensity is B, and the frequency of switching different light sources for lighting is C.

[0314] Then the infrared light source 1 and the infrared light source 2 flicker alternately at a frequency C. When illuminating, the power of the infrared light source 1 or the infrared light source 2 is A, and the illumination intensity is B.

[0315] S134: Control the target camera to collect images in real time according to the “camera use target strategy”.

[0316] Assuming that the frame rate set in the target strategy used by the camera is 30 FPS, the target camera is controlled to shoot a video sequence at a frame rate of 30 frames per second.

[0317] S135: Using a target eye tracking algorithm, feature data is extracted from the collected image, and gaze information is calculated based on the feature data.

[0318] The feature data may include: pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, light spot position, etc.

[0319] Different eye tracking algorithms may have different requirements for the type of feature data. For example, an eye tracking algorithm based on pupil position and morphology calculates gaze information through the main axis direction and position of the pupil. The feature data that the algorithm needs to extract include the main axis direction, pupil position, pupil major axis length, pupil minor axis length, etc. This algorithm only requires the environment camera to capture.

[0320] For another example, a certain feature vector fitting method works as follows:

[0321] Extract the center position of the pupil, the center position of the left corner of the eye, and the center position of the right corner of the eye;

[0322] The pupil center minus the left eye corner center is taken as the feature vector A;

[0323] The pupil center minus the right eye corner center is taken as feature vector B.

[0324] Construct a mapping function (a set of equations) between vector A, vector B and the gaze point.

[0325] Based on the given feature vectors A and B, and the known gaze information, a polynomial fitting is performed to obtain the unknown coefficients in the mapping function (the solution of the unknown coefficients can be completed during the calibration process).

[0326] After obtaining the unknown coefficients, the currently extracted feature vector is input into the mapping function to obtain the current gaze point (this process is a tracking process).

[0327] For the above-mentioned feature vector fitting method, the feature data to be extracted include the center position of the hole, the center position of the left eye corner and the center position of the right eye corner.

[0328] Since there are many types of eye tracking algorithms, we will not go into details here.

[0329] Embodiment 12

[0330] This embodiment provides an eye tracking device for a rotating screen terminal device. Fig.14 , the device comprises:

[0331] A determination unit 1 is used to determine the operating state of the rotating screen terminal device; the operating state at least includes the current device posture of the rotating screen terminal device;

[0332] A selection unit 2 is used to determine a corresponding eye tracking scheme from a plurality of preset eye tracking schemes according to the running state; the determined eye tracking scheme is a target eye tracking scheme;

[0333] The tracking unit 3 is used for performing eye tracking using a target eye tracking solution.

[0334] The eye tracking device involved in the embodiment of the present invention is deployed in a rotating screen terminal device, and can determine a suitable eye tracking scheme according to the operating state of the rotating screen terminal device, and use the determined eye tracking scheme to perform eye tracking.

[0335] Please refer to the above records for specific details and I will not go into details here.

[0336] In one example, the rotating screen terminal device has multiple device postures, and each device posture corresponds to at least one eye tracking solution.

[0337] In terms of determining a corresponding eye tracking scheme from a plurality of preset eye tracking schemes according to the operating state, the selection unit 2 is specifically used for:

[0338] A target eye tracking scheme is determined from at least one eye tracking scheme corresponding to the current device posture.

[0339] Please refer to the above records for specific details and I will not go into details here.

[0340] In another example, the above-mentioned operating status also includes: the current load of the rotating screen terminal device.

[0341] In terms of determining a target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture, the selection unit 2 is specifically configured to:

[0342] When the current load is higher than a preset load threshold, an eye tracking scheme with the lowest system resource load is determined from at least one eye tracking scheme corresponding to the current device posture.

[0343] Please refer to the above records for specific details and I will not go into details here.

[0344] In another example, the above-mentioned operating status also includes: the remaining power of the rotating screen terminal device.

[0345] In terms of determining a target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture, the selection unit 2 is specifically configured to:

[0346] When the remaining power is lower than a preset power threshold, an eye tracking scheme with the lowest power requirement is determined from at least one eye tracking scheme corresponding to the current device posture.

[0347] Please refer to the above records for specific details and I will not go into details here.

[0348] In yet another example, the above-mentioned running status also includes: the software type of the software currently running in the rotating screen terminal device.

[0349] In terms of determining a target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture, the selection unit 2 is specifically configured to:

[0350] An eye tracking scheme that meets the accuracy requirement of the software type is determined from at least one eye tracking scheme corresponding to the current device posture.

[0351] Please refer to the above records for specific details and I will not go into details here.

[0352] In yet another example, the above different eye tracking solutions differ in one or more of the following aspects:

[0353] Device usage policy;

[0354] Eye tracking algorithms;

[0355] Among them, the eye tracking algorithm is used to calculate the gaze information.

[0356] Please refer to the above records for specific details and I will not go into details here.

[0357] In yet another example, the device usage policy includes at least one of a light source usage policy, a light source flickering policy, a camera usage policy, a camera frame rate setting policy, and a camera resolution setting policy;

[0358] in:

[0359] The light source use strategy is used to indicate: light source equipment, power and lighting intensity of the light source equipment; the light source equipment includes a light source or a light source group;

[0360] The light source flashing strategy is used to indicate: the frequency and switching order of switching different light sources for lighting in the light source group;

[0361] The camera usage policy is used to indicate: the usage status of the camera group; the camera group includes at least one target camera; the target camera includes at least one of a camera for capturing an eye diagram and a depth camera;

[0362] The camera frame rate setting strategy is used to indicate: the frame rate of the target camera;

[0363] The camera resolution setting strategy is used to indicate: the resolution of the target camera.

[0364] Please refer to the above records for specific details and I will not go into details here.

[0365] In yet another example, the lighting device indicated by the light source usage strategy in the above target eye tracking solution is the target lighting device, the indicated power is the target power, and the indicated lighting intensity is the target lighting intensity;

[0366] The camera usage strategy in the target eye tracking scheme is the camera usage target strategy;

[0367] The light source flicker strategy in the target eye tracking scheme is the light source flicker target strategy;

[0368] The eye tracking algorithm in the target eye tracking solution is the target eye tracking algorithm;

[0369] In the aspect of using the target eye tracking solution for eye tracking, the tracking unit 3 is specifically used for:

[0370] Control the target lighting equipment to use target power, target lighting intensity and light source flicker target strategy for lighting;

[0371] Control the target camera in the camera group to use the "camera use target strategy" to collect images in real time;

[0372] Using the target eye tracking algorithm, feature data is extracted from the collected images, and gaze information is calculated based on the feature data.

[0373] Please refer to the above records for specific details and I will not go into details here.

[0374] In yet another example, the current device posture includes: the current screen posture of the rotating screen terminal device and the relative distance between the rotating screen terminal device and the user's face.

[0375] Please refer to the above records for specific details and I will not go into details here.

[0376] In yet another example, the posture of the screen of the rotating screen terminal device relative to the user's face is determined by a first posture analysis method;

[0377] The first posture analysis method includes:

[0378] Using a wide area camera to capture an image including the user's face;

[0379] Acquire location data of a specific part from the image; the specific part includes: at least one of: face, pupil, forehead, nose, ear and mouth;

[0380] Use the position data to calculate the relative angle between the wide-area camera and the face;

[0381] According to the positional relationship between the wide-area camera and the screen, the relative angle is converted into the posture of the screen relative to the user's face.

[0382] Please refer to the above records for specific details and I will not go into details here.

[0383] In yet another example, the posture of the screen of the rotating terminal device itself is determined by a second posture analysis method;

[0384] The second posture analysis method includes:

[0385] Get the current position output by the attitude determination module;

[0386] Determine the posture of the screen itself based on the current orientation.

[0387] Please refer to the above records for specific details and I will not go into details here.

[0388] In yet another example, the current screen posture of the rotating screen terminal device is determined in the following manner:

[0389] If the rotating screen terminal device has both the first posture analysis method and the second posture analysis method, the current screen posture of the rotating screen terminal device is determined according to the first posture analysis method;

[0390] Please refer to the above records for specific details and I will not go into details here.

[0391] Embodiment 13

[0392] An embodiment of the present invention provides a storage medium storing computer executable instructions. When the computer executable instructions are loaded and executed by a processor, the steps of the eye tracking method for a rotating screen terminal device described in any one of Embodiments 1 to 9 are implemented.

[0393] Embodiment 14

[0394] An embodiment of the present invention provides a processor, which is used to run a program, wherein when the program is run, the steps of the eye tracking method for a rotating screen terminal device described in any one of Embodiments 1 to 9 are executed.

[0395] Embodiment 15

[0396] An embodiment of the present invention provides a rotating screen terminal device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the eye tracking method steps for the rotating screen terminal device described in any one of Embodiments 1 to 9 can be executed.

[0397] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0398] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0399] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0400] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0401] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0402] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0403] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0404] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0405] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0406] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. An eye tracking method for a rotating screen terminal device, characterized in that: The method comprises: Determine the operating state of the rotating screen terminal device; the operating state at least includes the current device posture of the rotating screen terminal device; the rotating screen terminal device has multiple device postures, each device posture corresponds to at least one eye tracking scheme; the current device posture includes: the current screen posture of the rotating screen terminal device and the relative distance between the rotating screen terminal device and the user's face; Determining a corresponding eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture; The determined eye tracking scheme is a target eye tracking scheme; different eye tracking schemes differ in one or more of the following aspects: device usage strategy, eye tracking algorithm, wherein the eye tracking algorithm is used to calculate gaze information; Eye tracking was performed using the target eye tracking scheme.

2. The method according to claim 1, characterized in that The operating status also includes: the current load of the rotating screen terminal device; Determining the target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture includes: When the current load is higher than a preset load threshold, an eye tracking scheme with the lowest system resource load is determined from at least one eye tracking scheme corresponding to the current device posture.

3. The method according to claim 1, characterized in that The operating status also includes: the remaining power of the rotating screen terminal device; Determining the target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture includes: When the remaining power is lower than a preset power threshold, an eye tracking scheme with the minimum power requirement is determined from at least one eye tracking scheme corresponding to the current device posture.

4. The method according to claim 1, characterized in that The running status also includes: the software type of the software currently running in the rotating screen terminal device; Determining the target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture includes: An eye tracking scheme that meets the accuracy requirement of the software type is determined from at least one eye tracking scheme corresponding to the current device posture.

5. The method according to claim 1, characterized in that The device usage strategy includes at least one of a light source usage strategy, a light source flickering strategy, a camera usage strategy, a camera frame rate setting strategy, and a camera resolution setting strategy; in: The light source use strategy is used to indicate: a light source device, the power and lighting intensity of the light source device; the light source device includes a light source or a light source group; The light source flashing strategy is used to indicate: the frequency and switching order of switching different light sources in the light source group for lighting; The camera usage policy is used to indicate: the usage status of a camera group; the camera group includes at least one target camera; the target camera includes at least one of a camera for capturing an eye diagram and a depth camera; The camera frame rate setting strategy is used to indicate: the frame rate of the target camera; The camera resolution setting strategy is used to indicate: the resolution of the target camera.

6. The method according to claim 1, characterized in that The lighting device indicated by the light source usage strategy in the target eye tracking solution is the target lighting device, the indicated power is the target power, and the indicated lighting intensity is the target lighting intensity; The camera usage strategy in the target eye tracking solution is the camera usage target strategy; The light source flickering strategy in the target eye tracking scheme is a light source flickering target strategy; The eye tracking algorithm in the target eye tracking solution is a target eye tracking algorithm; The eye tracking using the target eye tracking scheme comprises: Controlling the target lighting device to perform lighting using the target power, the target lighting intensity and the light source flicker target strategy; Controlling a target camera in the camera group to use the camera to collect images in real time using a target strategy; The target eye tracking algorithm is used to extract feature data from the collected image, and gaze information is calculated based on the feature data.

7. The method according to claim 1, characterized in that The current screen posture of the rotating screen terminal device includes: the posture of the screen of the rotating screen terminal device relative to the user's face and / or the posture of the screen of the rotating screen terminal device itself; The posture of the screen of the rotating screen terminal device relative to the user's face is determined by a first posture analysis method; The posture of the screen of the rotating screen terminal device itself is determined by a second posture analysis method; The first posture analysis method has a higher priority than the second posture analysis method.

8. The method according to claim 7, characterized in that The first posture analysis method comprises: capturing an image including the user's face using a wide-area camera; Acquire location data of a specific part from the image; the specific part includes: at least one of face, pupil, forehead, nose, ear and mouth; calculating a relative angle between the wide area camera and the face using the position data; The relative angle is converted into a posture of the screen relative to the user's face according to the positional relationship between the wide-area camera and the screen.

9. The method according to claim 7, characterized in that The second posture analysis method comprises: Get the current position output by the attitude determination module; According to the current position, the posture of the screen itself is determined.

10. An eye tracking device for a rotating screen terminal device, characterized in that: The device comprises: A determination unit, configured to determine an operating state of the rotating screen terminal device; the operating state at least includes a current device posture of the rotating screen terminal device; the rotating screen terminal device has a plurality of device postures, each device posture corresponds to at least one eye tracking scheme; the current device posture includes: a current screen posture of the rotating screen terminal device and a relative distance between the rotating screen terminal device and a user's face; A selection unit is used to determine a corresponding eye tracking scheme from a plurality of preset eye tracking schemes according to the running state; the determined eye tracking scheme is a target eye tracking scheme; different eye tracking schemes are different in one or more of the following aspects: device use strategy, eye tracking algorithm, wherein the eye tracking algorithm is used to calculate gaze information; A tracking unit, configured to perform eye tracking using the target eye tracking scheme; The selection unit is specifically used to determine the target eye tracking scheme from at least one eye tracking scheme corresponding to the current device posture.

Images