Method and device for adjusting eye movement signal acquisition equipment and eyeball tracking module

Abstract

The invention discloses an adjusting method and device of eye movement signal acquisition equipment and an eyeball tracking module. The adjustment method of the eye movement signal acquisition equipment comprises the following steps: determining posture change information of the eye movement signal acquisition equipment; based on the posture change information, the eye movement signal acquisition equipment is adjusted, and a to-be-acquired object is included in an acquisition area of the adjusted eye movement signal acquisition equipment; the adjustment amount for adjusting the eye movement signal collecting device is obtained, calibration data of the eye movement signal collecting device are updated, and the calibration data are used for calculating human eye gazing information. According to the invention, the eye movement signal acquisition equipment is adjusted according to the posture change information of the eye movement signal acquisition equipment, and then the calibration data of the eye movement signal acquisition equipment is updated according to the adjustment amount of the eye movement signal acquisition equipment, so that the eye movement signal acquisition equipment can obtain clear and high-quality eye movement signals; therefore, more accurate eyeball tracking can be carried out based on the eye movement signal.

Description

Technical Field

[0001] The present invention relates to the technical field of eye tracking, and particularly to an adjustment method and device for an eye movement signal acquisition device and an eye tracking module. Background Art

[0002] An eye tracker is a device that can track and measure the position of the eyeball and the information of eye movement. When the eye tracker is set on an electronic device, the eye tracker is fixed to the display device of the electronic device so that the display device and the eye tracker maintain the same posture, facilitating the acquisition of clear human eye images or signals of the user.

[0003] However, during the use of the electronic device, a certain degree of jitter is inevitably brought, which makes the eye tracker unable to capture clear human eye images, resulting in a reduction in the accuracy of eye tracking. Summary of the Invention

[0004] The present invention provides an adjustment method and device for an eye movement signal acquisition device and an eye tracking module to solve the technical problem that the eye tracker cannot capture clear human eye images during eye tracking.

[0005] According to one aspect of the present invention, an adjustment method for an eye movement signal acquisition device is provided, and the method includes:

[0006] Determine the posture change information of the eye movement signal acquisition device;

[0007] Based on the posture change information, adjust the eye movement signal acquisition device, and the acquisition area of the adjusted eye movement signal acquisition device includes the object to be acquired;

[0008] Obtain the adjustment amount for adjusting the eye movement signal acquisition device, and update the calibration data of the eye movement signal acquisition device, where the calibration data is used to calculate the human eye fixation information.

[0009] According to another aspect of the present invention, an eye tracking module is provided, including:

[0010] An eye movement signal acquisition device, an adjustment device, a posture sensor, and a light source;

[0011] The eye movement signal acquisition device is used to acquire a target signal, the target signal is used to determine the posture change information of the eye movement signal acquisition device, and the posture change information is used to adjust the eye movement signal acquisition device;

[0012] The adjustment device is used to adjust the eye movement signal acquisition device; the posture sensor is used to determine posture data, the posture data is used to determine the adjustment amount for adjusting the eye movement signal acquisition device, the adjustment amount is used to update the calibration data of the eye movement signal acquisition device, and the calibration data is used to calculate the human eye fixation information.

[0013] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0014] An eye tracking module as provided by the present invention;

[0015] At least one processor communicatively connected to the eye tracking module; and

[0016] A memory communicatively connected to the at least one processor and the eye tracking module; wherein,

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the adjustment method of the eye movement signal acquisition device of the present invention.

[0018] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the adjustment method of the eye movement signal acquisition device according to any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention determines the attitude change information of the eye movement signal acquisition device; based on the attitude change information, adjusts the eye movement signal acquisition device, and the acquisition area of the adjusted eye movement signal acquisition device includes the object to be acquired; obtains the adjustment amount for adjusting the eye movement signal acquisition device, and updates the calibration data of the eye movement signal acquisition device, and the calibration data is used to calculate the human eye fixation information. The present invention solves the technical problem that in a jittery usage scenario, the eye tracker cannot capture clear eye movement signals, thereby reducing the accuracy of eye tracking. The present invention adjusts the eye movement signal acquisition device through the attitude change information of the eye movement signal acquisition device, and then updates the calibration data of the eye movement signal acquisition device according to the adjustment amount of the eye movement signal acquisition device, so that the eye movement signal acquisition device can be adjusted based on its own attitude change information, has an anti-shake function, and thus can obtain clear eye movement signals for more accurate eye tracking based on the eye movement signals.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0022] Figure 1 It is a flowchart of a method for adjusting an eye movement signal acquisition device provided in Embodiment 1 of the present invention;

[0023] Figure 2 It is a flowchart of a method for adjusting an eye movement signal acquisition device provided in Embodiment 2 of the present invention;

[0024] Figure 3 It is a schematic structural diagram of an adjustment device for an eye movement signal acquisition device provided in Embodiment 3 of the present invention;

[0025] Figure 4 It is a schematic structural diagram of an eyeball tracking module provided in Embodiment 4 of the present invention;

[0026] Figure 5 It is a schematic structural diagram of an electronic device for implementing the method for adjusting an eye movement signal acquisition device in Embodiment 5 of the present invention;

[0027] Figure 6 It is a usage scenario diagram of the eyeball tracking module on an electronic device in Embodiment of the present invention. Detailed implementation manners

[0028] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0029] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] Eye tracking, also known as gaze tracking, is a technique for estimating the line of sight and / or fixation point of the eye by measuring eye movements.

[0031] Among them, the line of sight can be understood as a three-dimensional vector, and the fixation point can be understood as the two-dimensional coordinates of the projection of the above three-dimensional vector on a certain plane.

[0032] There are various methods for applying eye tracking technology. One of them is the optical recording method: using a camera or video camera to record the eye movements of the subject, that is, obtaining an eye image reflecting the eye movements, and extracting eye features from the obtained eye image for establishing a model for estimating the line of sight and / or a model for estimating the fixation point. Among them, the eye features may include: pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, position of the light spot (also known as the Purkinje spot), etc.

[0033] In the optical recording method, the eye tracking method included is called the pupil-corneal reflection method. Other methods may also include methods not based on eye images, such as estimating eye movements based on contact / non-contact sensors (such as electrodes, capacitive sensors).

[0034] Among them, the pupil-corneal reflection method is outlined as follows:

[0035] 1. The working principle can be simply summarized as:

[0036] Obtain an eye image;

[0037] Estimate the line of sight and / or fixation point based on the eye image.

[0038] 2. Hardware requirements:

[0039] (1) Light source: Generally an infrared light source, because infrared light does not affect the vision of the eyes; and multiple infrared light sources can be used, arranged in a predetermined manner, such as in a triangular shape, a linear shape, etc.;

[0040] (2) Image acquisition device: such as an infrared camera device, an infrared image sensor, a camera or a video camera, etc. The image acquisition device can be considered as a device for image acquisition, such as a device for acquiring an eye diagram. In the present invention, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of relevant laws and regulations.

[0041] 3. The specific implementation is as follows:

[0042] 1. Acquisition of eye images:

[0043] The light source shines on the eye, and the image acquisition device takes a picture of the eye. The reflection point of the corresponding shooting light source on the cornea is the light spot (also called the Purkinje spot), and thus an eye image with the light spot is obtained.

[0044] 2. Estimation of line of sight and / or fixation point:

[0045] As the eyeball rotates, the relative position relationship between the pupil center and the light spot changes accordingly. A number of eye images with the light spot collected correspondingly reflect such a position change relationship;

[0046] The line of sight and / or fixation point is estimated according to the position change relationship.

[0047] Another method for realizing eye tracking is an eye tracking system based on the MEMS scheme. Through a pre-deployed optical signal transmitter and optical signal receivers at multiple positions, it is confirmed whether corneal reflection occurs in the corresponding direction, and the approximate direction of eyeball rotation is determined based on its orientation, and then the required line of sight information is obtained. According to the patent US20210303064A1, the eye tracking method of the MEMS system can obtain eye pupil data (pupil size and contour). The eye tracking system based on the MEMS scheme does not use a camera to collect the user's eye image, but uses an infrared diode to emit infrared light. After the infrared light is emitted, it is directed to the scanner plane mirror. At the same time, the scanner will control the infrared light to be directed to the user's eye, and then it is reflected from the corneal surface to the optical sensor. The optical sensor converts the optical signal into an electrical signal, and finally part of the eye data (information such as the user's cornea, iris and pupil information data) is recognized and generated.

[0048] In the prior art, eye tracking can be realized by an eye tracker. When the eye tracker is built into an electronic device or used as an accessory of an electronic device, the eye tracker and the display device of the electronic device are fixed together to maintain the same position and posture. Because the electronic device inevitably brings a certain degree of jitter during use, the eye tracker cannot capture the area of interest or cannot capture it completely, resulting in a decrease in the accuracy of eye tracking or it becoming unavailable.

[0049] To solve the above technical problems, a method for adjusting an eye movement signal acquisition device is publicly provided, which enables the eye tracking module to have an anti-shake function by adjusting the eye movement signal acquisition device.

[0050] Embodiment 1

[0051] Figure 1 FIG. is a flowchart of a method for adjusting an eye movement signal acquisition device according to Embodiment 1 of the present invention. This embodiment is applicable to the situation where the eye movement signal acquisition device is adjusted to enable the eye tracking device to have an anti-shake function. This method can be executed by an adjustment device of the eye movement signal acquisition device. The adjustment device of the eye movement signal acquisition device can be implemented in the form of hardware and / or software, and the adjustment device of the eye movement signal acquisition device can be configured in an electronic device. The electronic device can be an electronic device that generates jitter during use, such as a handheld electronic device. In one embodiment, the electronic device can be a mobile phone, a tablet computer, a notebook computer, etc.

[0052] As Figure 1 shown, the method includes:

[0053] S110. Determine the posture change information of the eye movement signal acquisition device.

[0054] The posture change information can be considered as information characterizing the posture change. The posture change information can be information characterizing the change from the posture of the eye movement signal acquisition device before adjustment; or it can be information characterizing the posture of the eye movement signal acquisition device during the calibration of the eye tracking module. In the present invention, the eye movement signal acquisition device can be an image acquisition device in the eye tracking module, a Dynamic Vision Sensor (DVS) camera, or an optical signal receiver in the MEMS system, for realizing eye tracking.

[0055] The calibration of the eye tracking module can also be called user calibration. The eye tracking module can be calibrated before use to ensure the accuracy rate. During calibration, the user is asked to fixate on a specific target point on the screen, and the device binds the user's eye line of sight to the target point. Do this multiple times to achieve calibration.

[0056] Specifically, during the process of line of sight and / or fixation point estimation, in order to determine some undetermined parameters (also called calibration parameters, generally corresponding to some internal parameters of the user's eyeball, such as the eyeball radius, etc.) in the model for line of sight estimation and / or the model for fixation point estimation, a common method is: let the user fixate on one or more target points, assuming that the information of the target points is the known line of sight (because the target points are preset in advance), and thus the above calibration parameters can be solved inversely.

[0057] The means for determining the pose change information is not limited. For example, taking an image acquisition device as an example: the pose change information of the image acquisition device can be determined through the target image acquired by the image acquisition device. When analyzing the pose change information of the image acquisition device through the target image, an object for analysis can be selected from the target image, such as an object to be acquired. The pose change information of the image acquisition device can be determined by analyzing the offset of the object to be acquired in the target image. The object to be acquired can be considered as an object to be acquired through the image acquisition device, such as the eye of a user using an eye tracking module.

[0058] In one embodiment, the pose change information includes one or more of the following:

[0059] Information indicating the adjustment direction of the eye tracking module; information indicating the adjustment amount of the eye tracking module.

[0060] When the pose change information includes information indicating the adjustment direction of the eye tracking module and information indicating the adjustment amount of the eye tracking module, the eye tracking module can be directly adjusted to a position where a clear eye image can be acquired. For example, along the adjustment direction of the eye tracking module, adjust the eye tracking module, and the adjustment amplitude is equal to the information indicating the adjustment amount of the eye tracking module. Clear can be measured by the clarity and the integrity of the human eye in the eye image.

[0061] When the pose change information includes information indicating the adjustment direction of the eye tracking module, the eye tracking module can be adjusted multiple times. After each adjustment, it can be determined whether to continue adjusting the eye tracking module by analyzing the target image acquired by the image acquisition device in the adjusted eye tracking module. The step size of each adjustment of the eye tracking module can be the same or different. When the step size of each adjustment of the eye tracking module is the same, the eye tracking module can be adjusted using a set step size. When the step size of each adjustment of the eye tracking module is different, the step size of each adjustment can be determined based on the position of the object to be acquired in the target image. The offset of the object to be acquired from the central region of the target image can be proportional to the step size of each adjustment of the eye tracking module. For example, the farther the object to be acquired deviates from the central region of the target image, the larger the adjustment step size.

[0062] S120. Based on the pose change information, adjust the eye movement signal acquisition device, and the acquisition area of the adjusted eye movement signal acquisition device includes the object to be acquired.

[0063] The acquisition area can be considered as the field of view of the eye movement signal acquisition device. In this operation, the pose change information is used to indicate how to adjust the eye movement signal acquisition device. Adjust the eye movement signal acquisition device based on the pose change information, so that the eye movement signal acquisition device can acquire the eye movement signal including the object to be acquired, to achieve the acquisition of a clear eye movement signal, and further achieve eye tracking.

[0064] The means for adjusting the eye movement signal acquisition device are not limited. For example, the position or parameters of the eye tracking module can be adjusted. In one example, the entire eye tracking module can be adjusted based on the attitude change information, such as rotating the eye tracking module. In another example, if the eye movement signal acquisition device is an image acquisition device, the lens parameters of the image acquisition device in the eye tracking module can be adjusted based on the attitude change information. The adjustment of the lens parameters can be achieved through the adjustment device of the image acquisition device.

[0065] By adjusting the eye movement signal acquisition device based on the attitude change information, the eye movement signal acquisition device has an anti-shake function, which improves the accuracy of the eye movement signal acquisition device to acquire eye movement signals for eye tracking.

[0066] S130. Obtain the adjustment amount for adjusting the eye movement signal acquisition device, and update the calibration data of the eye movement signal acquisition device. The calibration data is used to calculate the human eye fixation information.

[0067] The adjustment amount can be regarded as a value measuring the adjustment situation of the eye movement signal acquisition device. For example, the measurement value of the rotation of the eye movement signal acquisition device. The calibration data can be regarded as the data for calibrating the eye movement signal acquisition device, such as the internal parameters and / or external parameters of the eye movement signal acquisition device. Based on the updated calibration data, the eye movement signal acquisition device can obtain the eye movement signal, and then determine the human eye fixation information based on the eye movement signal. The fixation information can be regarded as the fixation point and / or fixation direction of a person.

[0068] The determination of the adjustment amount can be achieved through the devices separately arranged in the eye tracking module, or through the devices in the eye tracking module that realize the adjustment of the eye movement signal acquisition device. For example, the adjustment amount of the eye movement signal acquisition device is detected through the separately arranged devices; or when the devices that realize the adjustment are adjusting the eye movement signal acquisition device, the adjustment amount of the eye movement signal acquisition device is recorded.

[0069] Among them, the device that realizes the adjustment of the eye movement signal acquisition device can be the adjustment device. The separately arranged device can be a pose sensor. The pose sensor determines the adjustment amount for adjusting the eye movement signal acquisition device by detecting the pose of the eye movement signal acquisition device.

[0070] After obtaining the adjustment amount in this operation, the calibration data of the eye movement signal acquisition device can be updated based on the adjustment amount. The update means are not limited here.

[0071] In one embodiment, taking the image acquisition device as an example of the eye movement signal acquisition device, the adjustment amount may include the translation information and rotation angle information of the optical center after the rotation of the image acquisition device. By sending the translation information and rotation angle information to the eye movement algorithm, the pose information of the image acquisition device is compensated. The eye movement algorithm updates the internal parameters and / or external parameters of the image acquisition device, recalculates the eye movement information, and obtains a high-precision result.

[0072] In another embodiment, taking the optical signal receiver in the MEMS system as an example of the eye movement signal acquisition device, the adjustment amount may include the translation information and rotation angle information of the optical signal receiver. By sending the translation information and rotation angle information to the eye movement algorithm, the pose information of the optical signal receiver is compensated. The eye movement algorithm updates the internal parameters and / or external parameters of the optical signal receiver, recalculates the eye movement information, and obtains a high-precision result.

[0073] The technical solution of the embodiment of the present invention determines the attitude change information of the eye movement signal acquisition device; based on the attitude change information, adjusts the eye movement signal acquisition device, and the acquisition area of the adjusted eye movement signal acquisition device includes the object to be acquired; obtains the adjustment amount for adjusting the eye movement signal acquisition device, and updates the calibration data of the eye movement signal acquisition device, where the calibration data is used to calculate the human eye fixation information. The present invention solves the technical problem that in a jittery usage scenario, the eye tracker cannot obtain clear eye movement signals, thereby reducing the accuracy of eye tracking. The present invention adjusts the eye movement signal acquisition device through the attitude change information of the eye movement signal acquisition device, and then updates the calibration data of the eye movement signal acquisition device according to the adjustment amount of the eye movement signal acquisition device, so that the eye movement signal acquisition device can be adjusted based on its own attitude change information, has an anti-shake function, and thus can obtain clear eye movement signals for more accurate eye tracking based on the eye movement signals.

[0074] Embodiment 2

[0075] Figure 2 is a flowchart of a method for adjusting an eye movement signal acquisition device provided by Embodiment 2 of the present invention. In this embodiment, the operations of determining the attitude change information, obtaining the adjustment amount, and adjusting the eye movement signal acquisition device are refined based on the above embodiment. As Figure 2 shown, the method includes:

[0076] S210. Obtain the target signal collected by the eye movement signal acquisition device.

[0077] The target signal can be understood as the eye movement signal collected by the eye movement signal acquisition device, which can be the target image collected by the image acquisition device, the dynamic signal collected by the DVS camera, or the optical signal collected by the optical signal receiver of the MEMS system.

[0078] In this operation, the processor of the electronic device can obtain the target signal collected by the eye movement signal acquisition device to determine the attitude change information of the eye movement signal acquisition device by analyzing the target signal.

[0079] The target image can be the target image collected by the image acquisition device when the user uses the eye tracking module. The eye tracking module can be arranged on the electronic device. The user using the eye tracking module can be that the user uses the electronic device to realize eye tracking through the eye tracking module. The present invention performs eye tracking on the user after obtaining the user's authorization.

[0080] S220. Determine the attitude change information of the eye movement signal acquisition device according to the signal characteristics of the object to be collected in the target signal.

[0081] The representation method of the signal characteristics of the object to be collected in the target signal is not limited. For example, it can be represented by the orientation of the object to be collected in the target image, or by the relative position of the object to be collected and the center position of the target image, such as in the upper left, lower right, directly above or directly below the center position of the target image, etc.; it can also be represented by the orientation in the image converted from the dynamic signal collected by the DVS camera, or by the image converted from the optical signal collected by the optical signal receiver of the MEMS system.

[0082] This operation can determine the attitude change information by combining the correlation between the signal characteristics of the object to be collected in the target signal and the change in the pose of the eye movement signal acquisition device.

[0083] If the signal characteristics of the object to be collected in the target signal indicate that the object to be collected is biased to the left of the target signal, it reflects that the eye movement signal acquisition device has shifted to the right, thus causing the object to be collected to be biased to the left of the target signal. The attitude change information can be the information indicating that the eye movement signal acquisition device has shifted to the right.

[0084] S230. Adjust the pose information of the eye tracking module to which the eye movement signal acquisition device belongs based on the attitude change information; and / or adjust the lens parameters in the eye tracking module based on the attitude change information.

[0085] The pose information can be the information indicating the position and attitude. The lens parameters can be the parameters of the lens of the image acquisition device.

[0086] This operation can adjust the pose information of the eye tracking module in combination with the pose change information. For example, by adjusting the position information of the eye tracking module to compensate for the pose change information, the eye movement signal acquisition device can obtain the content of interest, such as eye movement signals. This step can also adjust the lens parameters of the eye tracking module in combination with the pose change information. For example, change the focal length of the image acquisition device. The means of adjusting the lens parameters are not limited here, as long as the pose change information can be compensated so that the adjusted image acquisition device can capture an image containing a clear eye.

[0087] This operation can directly adjust the pose information of the eye movement signal acquisition device to the appropriate position based on the pose change information, or adjust the pose information of the eye movement signal acquisition device based on the pose change information, and then determine whether the adjusted eye movement signal acquisition device still needs to be adjusted. If so, repeat the determination of the pose change information of the adjusted eye movement signal acquisition device, and then adjust the eye movement signal acquisition device based on the pose change information. By adjusting the eye movement signal acquisition device multiple times, the eye movement signal acquisition device reaches an ideal state, such as a state where it can capture a clear image of the eye.

[0088] S240. Obtain pose data, where the pose data represents the relative rotation amount of the adjusted position of the eye movement signal acquisition device relative to the reference position.

[0089] The reference position can be considered as the position for reference. At the reference position, the eye movement signal acquisition device can collect the eye signal of interest. The eye signal can be considered to include an image of the eye or other eye data. Eye tracking can be achieved based on the eye signal. The eye signal of interest can be considered to include the content required for eye tracking in the eye signal.

[0090] The pose sensor can directly determine the pose data, or collect the adjusted position of the eye movement signal acquisition device. After the pose sensor collects the adjusted position of the eye movement signal acquisition device, the relative rotation amount can be determined in combination with the reference position.

[0091] In this embodiment, the target signal collected by the eye movement signal acquisition device is used to determine how to adjust the eye movement signal acquisition device. The pose sensor is used to record the adjustment amount of the eye movement signal acquisition device to facilitate updating the calibration data of the eye movement signal acquisition device.

[0092] S250. Determine the relative rotation amount as the adjustment amount.

[0093] S260. Update the calibration data of the eye movement signal acquisition device.

[0094] In this embodiment, the means for determining the attitude change information, adjusting the eye movement signal acquisition device, and obtaining the adjustment amount are refined. Through the analysis of the target signal, accurate attitude change information can be determined. Furthermore, through the adjustment of the pose information and lens parameters, the adjustment of the eye movement signal acquisition device is realized. By determining the attitude data to update the calibration data of the eye movement signal acquisition device, the adjusted and calibrated eye movement signal acquisition device can perform accurate eye tracking.

[0095] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. Here, it should be noted that for the sake of brevity of description, only the differences from the above embodiment are described in the variant embodiment.

[0096] In one embodiment, adjusting the pose information of the eye movement signal acquisition device belonging to the eye tracking module based on the attitude change information includes:

[0097] Based on the attitude change information, determine whether to adjust the eye movement signal acquisition device belonging to the eye tracking module;

[0098] If so, based on the attitude change information, determine the target direction for adjusting the eye movement signal acquisition device belonging to the eye tracking module;

[0099] Control the adjustment device of the eye tracking module to adjust the pose information of the eye movement signal acquisition device belonging to the eye tracking module along the target direction according to the set step size;

[0100] Return to continue determining the attitude change information of the eye movement signal acquisition device, and based on the attitude change information, adjust the eye movement signal acquisition device belonging to the eye tracking module.

[0101] If the attitude change information indicates that the attitude of the eye movement signal acquisition device has no change or the change amount is less than the set threshold, it can be determined that there is no need to adjust the eye movement signal acquisition device belonging to the eye tracking module; otherwise, it can be determined that the eye movement signal acquisition device belonging to the eye tracking module needs to be adjusted.

[0102] The attitude change information can indicate the direction of the attitude change of the eye movement signal acquisition device. In this embodiment, based on the attitude change information, the target direction of the adjustment of the eye tracking module can be determined. The target direction can be the direction for adjusting the eye tracking module.

[0103] The adjustment device can be a device that realizes the adjustment function in the eye tracking module, such as a device for adjusting the pose of the eye movement signal acquisition device belonging to the eye tracking module.

[0104] The adjustment of the eye movement signal acquisition device can be realized by adjusting the eye movement signal acquisition device itself, or by adjusting the eye tracking module integrated with the eye movement signal acquisition device.

[0105] The set step size can be a pre-set step size. After determining the target direction, in this embodiment, the pose information of the eye movement signal acquisition device belonging to the eye tracking module can be adjusted along the target direction according to the set step size. After the adjustment, the attitude change information of the eye movement signal acquisition device can be continuously determined to continue to determine whether it is necessary to adjust the eye movement signal acquisition device belonging to the eye tracking module.

[0106] Embodiment III

[0107] Figure 3 It is a schematic structural diagram of an adjustment device for an eye movement signal acquisition device provided in Embodiment III of the present invention. As Figure 3 shown, the device includes:

[0108] A determination module 310, configured to determine the attitude change information of the eye movement signal acquisition device;

[0109] An adjustment module 320, configured to adjust the eye movement signal acquisition device based on the attitude change information, and the acquisition area of the adjusted eye movement signal acquisition device includes the object to be acquired;

[0110] An update module 330, configured to obtain the adjustment amount for adjusting the eye movement signal acquisition device, and update the calibration data of the eye movement signal acquisition device, where the calibration data is used to calculate the human eye fixation information.

[0111] The eye movement signal acquisition device provided by the present invention adjusts the eye movement signal acquisition device through the attitude change information of the eye movement signal acquisition device, and then updates the calibration data of the eye movement signal acquisition device according to the adjustment amount of the eye movement signal acquisition device, so that the eye movement signal acquisition device can be adjusted based on its own attitude change information, has an anti-shake function, and thus can capture clear eye movement signals for more accurate eye tracking based on the eye movement signals.

[0112] In one embodiment, the determination module 310 is specifically configured to:

[0113] Obtain the target signal collected by the eye movement signal acquisition device;

[0114] Determine the attitude change information of the eye movement signal acquisition device according to the signal characteristics of the object to be acquired in the target signal.

[0115] In one embodiment, the update module 330 is specifically configured to:

[0116] Obtain attitude data, where the attitude data represents the relative rotation amount of the position where the eye movement signal acquisition device is adjusted relative to the reference position;

[0117] Determine the relative rotation amount as the adjustment amount.

[0118] In one embodiment, the adjustment module 320 includes:

[0119] A first adjustment unit, configured to adjust the pose information of the eye movement signal acquisition device's affiliated eye tracking module based on the pose change information; and / or,

[0120] A second adjustment unit, configured to adjust the lens parameters in the eye tracking module based on the pose change information.

[0121] In one embodiment, the first adjustment unit is specifically configured to:

[0122] Determine whether to adjust the eye tracking module to which the eye movement signal acquisition device belongs based on the pose change information;

[0123] If so, determine the target direction for adjusting the eye tracking module to which the eye movement signal acquisition device belongs based on the pose change information;

[0124] Control the adjustment device of the eye tracking module to adjust the pose information of the eye movement signal acquisition device's affiliated eye tracking module along the target direction according to a set step size;

[0125] Return to continue determining the pose change information of the eye movement signal acquisition device, and adjust the eye movement signal acquisition device's affiliated eye tracking module based on the pose change information.

[0126] The adjustment device of the eye movement signal acquisition device provided by the embodiments of the present invention can execute the adjustment method of the eye movement signal acquisition device provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0127] The adjustment method of the eye movement signal acquisition device provided by the present invention can adjust the shooting angle of the eye tracking module in real time, so that the obtained region of interest is always at the best position in the acquired image, and the best eye tracking accuracy can be obtained.

[0128] Embodiment 4

[0129] Figure 4 It is a schematic structural diagram of an eye tracking module provided by Embodiment 4 of the present invention.

[0130] The eye tracking module includes:

[0131] An eye movement signal acquisition device 410, an adjustment device 420, a pose sensor 430, and a light source 440;

[0132] The eye movement signal acquisition device 410 is configured to acquire a target signal, the target image signal is used to determine the pose change information of the eye movement signal acquisition device, and the pose change information is used to adjust the eye movement signal acquisition device 410;

[0133] The adjustment device 420 is used to adjust the eye movement signal acquisition device 410; the attitude sensor 430 is used to determine attitude data, and the attitude data is used to determine the adjustment amount for adjusting the eye movement signal acquisition device 410, and the adjustment amount is used to update the calibration data of the eye movement signal acquisition device 410, and the calibration data is used to calculate the human eye fixation information.

[0134] In this embodiment, the eye movement signal acquisition device is adjusted according to the attitude change information of the eye movement signal acquisition device, and then the calibration data of the eye movement signal acquisition device is updated according to the adjustment amount of the eye movement signal acquisition device, so that the eye movement signal acquisition device can obtain clear and high-quality eye movement signals, so as to perform more accurate eye tracking based on the eye movement signals.

[0135] Embodiment Five

[0136] Figure 5 It is a schematic structural diagram of an electronic device for implementing the adjustment method of the eye movement signal acquisition device according to Embodiment Five of the present invention.

[0137] Figure 5 A schematic structural diagram of an electronic device 10 that can be used to implement any embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0138] As Figure 5 shown, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by the at least one processor 11, and the computer program is executed by the at least one processor 11, so that the at least one processor 11 can execute the method of any embodiment provided by the present invention.

[0139] The processor 11 can perform various appropriate actions and processes according to a computer program stored in the read-only memory (ROM) 12 or a computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0140] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0141] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the methods and processes of the various embodiments described above, such as the adjustment method of the image acquisition device.

[0142] In some embodiments, the adjustment method of the eye movement signal acquisition device can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the adjustment method of the eye movement signal acquisition device described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the adjustment method of the eye movement signal acquisition device by any other appropriate means (e.g., by means of firmware).

[0143] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0144] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine, or entirely on the remote machine or server.

[0145] Figure 6 It is a usage scenario diagram of the eye tracking module for implementing the embodiments of the present invention on an electronic device. Refer to Figure 6 , an eye tracking module is arranged on the electronic device 10, and the eye tracking module includes an eye movement signal acquisition device 2, an attitude sensor 3 for sensing external changes, and a light source 4. The attitude sensor 3 can be one or a combination of a gyroscope, an accelerometer, and a magnetometer. The attitude sensor 3 can record the changes of the eye movement signal acquisition device 2, such as the adjustment amount. The adjustment amount is transmitted to the electronic device 10, and then the electronic device 10 sends a command to the adjustment device of the eye tracking module to adjust the eye tracking module. It is possible to rotate the eye tracking module as a whole, change one or a combination of the lens parameters of the eye tracking module to achieve adjustment, compensate for the angle change to the greatest extent, and keep the eye movement signal acquisition device 2 at the best shooting position.

[0146] The eye movement signal acquisition device 2 can be configured directly below the screen of the electronic device 10. There is a light source 4, such as an LED lamp, at each of the left and right borders directly below the screen of the electronic device 10. The attitude sensor 3 needs to be close to the eye movement signal acquisition device 2 and fixed together to ensure that when the screen rotates, the rotation angles of the attitude sensor 3 and the eye movement signal acquisition device 2 are as exactly the same as possible, so as to adjust the angle of the eye movement signal acquisition device 2 to the correct position.

[0147] In one embodiment, the position deviation between the installation position of the attitude sensor in the eye tracking module on the electronic device and the installation position of the eye movement signal acquisition device in the eye tracking module on the electronic device is less than a set deviation value.

[0148] The set deviation value can be a pre-set deviation value. The position deviation can be the difference between the installation position of the attitude sensor on the electronic device and the installation position of the eye movement signal acquisition device on the electronic device. The position deviation being less than the set deviation value enables the rotation angles of the attitude sensor and the eye movement signal acquisition device to be as exactly the same as possible.

[0149] When the screen of the electronic device shakes, the adjustment device controls the eye movement signal acquisition device to rotate and compensate. The attitude sensor determines the compensation value, that is, the adjustment amount, and transmits the adjustment amount to the eye movement algorithm. After the eye movement algorithm completes the compensation, it calculates the accurate fixation point and / or fixation direction.

[0150] A specific example is as follows: Assume that in the optimal usage position state of the mobile phone (which can be assumed to be the calibrated state), a world coordinate system is set. According to the right-hand coordinate system rule, the positive x direction is to the right, the positive y direction is downward, and the positive z direction is perpendicular to the screen and outward. With the coordinate of the upper left corner point of the screen as the origin (0, 0) and the scale unit being mm, on the premise of knowing the mobile phone size and the position of the eye tracking module, the camera coordinates and the LED coordinates and the three-axis rotation of the camera (i.e., the eye movement signal acquisition device) relative to the origin can be obtained. When shaking occurs, the adjustment device rotates the eye movement module to adjust back to the optimal position. The attitude sensor records the adjustment amount and transmits the adjustment amount to the eye movement algorithm. The eye movement algorithm updates the calibration data of the eye movement module in real time, and calculates the real-time eye movement fixation point and / or fixation direction through the updated extrinsic parameters.

[0151] The up, down, left, and right described in this embodiment can be the orientations based on the illustrated angles.

[0152] The present invention adjusts the attitude of the eye tracking module in real time according to the signal characteristics of the human eye in the target signal, so that the human eye is in the optimal shooting position, and at the same time transmits the adjusted attitude information to the eye movement algorithm, and the eye movement algorithm completes the subsequent compensation.

[0153] In the context of the present invention, a computer-readable storage medium stores computer instructions for causing a processor to implement the adjustment method of the eye movement signal acquisition device according to any embodiment provided by the present invention when executed.

[0154] A computer-readable storage medium may be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide for interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN), blockchain network, and the Internet.

[0157] The computing system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and Virtual Private Server (VPS) services.

[0158] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and this is not limited herein.

[0159] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for adjusting an eye movement signal acquisition device, characterized in that, the method includes: determining the attitude change information of the eye movement signal acquisition device; based on the attitude change information, adjusting the eye movement signal acquisition device, and the acquisition area of the adjusted eye movement signal acquisition device includes the object to be acquired; obtaining the adjustment amount for adjusting the eye movement signal acquisition device, and updating the calibration data of the eye movement signal acquisition device, where the calibration data is used to calculate the human eye fixation information.

2. The method according to claim 1, characterized in that, the determining the attitude change information of the eye movement signal acquisition device includes: obtaining the target signal acquired by the eye movement signal acquisition device; determining the attitude change information of the eye movement signal acquisition device according to the signal characteristics of the object to be acquired in the target signal.

3. The method according to claim 1, characterized in that, obtaining the adjustment amount for adjusting the eye movement signal acquisition device includes: obtaining attitude data, where the attitude data represents the relative rotation amount of the position where the eye movement signal acquisition device is adjusted relative to the reference position; determining the relative rotation amount as the adjustment amount.

4. The method according to claim 1, characterized in that, the adjusting the eye movement signal acquisition device based on the attitude change information includes: adjusting the pose information of the eye tracking module to which the eye movement signal acquisition device belongs based on the attitude change information; and / or, adjusting the lens parameters in the eye tracking module based on the attitude change information.

5. The method according to claim 4, characterized in that, the adjusting the pose information of the eye tracking module to which the eye movement signal acquisition device belongs based on the attitude change information includes: determining whether to adjust the eye tracking module to which the eye movement signal acquisition device belongs based on the attitude change information; if so, determining the target direction for adjusting the eye tracking module to which the eye movement signal acquisition device belongs based on the attitude change information; controlling the adjustment device of the eye tracking module to adjust the pose information of the eye tracking module to which the eye movement signal acquisition device belongs along the target direction according to a set step size; returning to continue determining the attitude change information of the eye movement signal acquisition device, and adjusting the eye tracking module to which the eye movement signal acquisition device belongs based on the attitude change information.

6. An adjustment device for an eye movement signal acquisition device, characterized in that, includes: a determination module for determining the attitude change information of the eye movement signal acquisition device; an adjustment module for adjusting the eye movement signal acquisition device based on the attitude change information, and the acquisition area of the adjusted eye movement signal acquisition device includes the object to be acquired; an update module for obtaining the adjustment amount for adjusting the eye movement signal acquisition device and updating the calibration data of the eye movement signal acquisition device, where the calibration data is used to calculate the human eye fixation information.

7. An eye tracking module, characterized in that, includes: an eye movement signal acquisition device, an adjustment device, an attitude sensor and a light source; The eye movement signal acquisition device is used to acquire a target signal, the target signal is used to determine the attitude change information of the eye movement signal acquisition device, and the attitude change information is used to adjust the eye movement signal acquisition device; The adjustment device is used to adjust the eye movement signal acquisition device; the attitude sensor is used to determine attitude data, the attitude data is used to determine the adjustment amount for adjusting the eye movement signal acquisition device, the adjustment amount is used to update the calibration data of the eye movement signal acquisition device, and the calibration data is used to calculate the human eye fixation information.

8. An electronic device, characterized in that, the electronic device includes: the eye tracking module according to claim 7; at least one processor communicatively connected to the eye tracking module; and a memory communicatively connected to the at least one processor and the eye tracking module; wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the adjustment method of the eye movement signal acquisition device according to any one of claims 1-5.

9. The electronic device according to claim 8, characterized in that, the position deviation between the installation position of the attitude sensor in the eye tracking module on the electronic device and the installation position of the eye movement signal acquisition device in the eye tracking module on the electronic device is less than a set deviation value.

10. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the adjustment method of the eye movement signal acquisition device according to any one of claims 1-5 when executed by a processor.

Images