Eye tracking method, terminal and computer storage medium

Abstract

The application relates to an eyeball tracking method applied to an image sensor, wherein the image sensor comprises an image acquisition module, an artificial intelligence model reasoning module and an external interface module, and the method comprises the following steps: the image acquisition module acquires an eye image of a user; the artificial intelligence model reasoning module performs reasoning based on the eye image according to a preset algorithm to obtain a reasoning result; and the external interface module outputs corresponding eyeball recognition data according to the reasoning result. The application can track the eyeball state of the user in real time, realizes the control of a light source and the output of information only according to the eye image through the acquisition and reasoning of the eye image, and improves the use experience of the user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of eye tracking, in particular to an eye tracking method, a terminal and a computer storage medium. BACKGROUND

[0002] The existing eye tracking system can currently achieve the identification and tracking of human eye behavior hundreds of times per second. However, each component in the existing eye tracking system is in a separated state, so that there is a large delay from the user's eye making a specific behavior to the eye tracking system being able to identify the user's behavior. The current eye tracking system using images as input has a delay of more than 10ms, and the eye tracking system not using images as input can control the delay to be less than 3ms, but the recognition accuracy is lower than the system using images as input. Secondly, due to the use of more components in the existing eye tracking system, the overall power consumption of the system is high, and the installation and adaptation are not convenient due to the occupation of more space of the device. Limited by the processing capacity of the logic processor, the existing eye tracking system can usually only process the image information collected by 1 to 2 image sensors in real time, which makes it impossible to track the rapid eye movement behaviors such as the user closing his eyes and scanning at high accuracy and high frame rate. SUMMARY

[0003] Therefore, the purpose of the present application is to provide an eye tracking method, a terminal and a computer storage medium, which can track the eye state of the user in real time, control the light source and output information only according to the eye image through the collection and inference of the eye image, and improve the user experience.

[0004] To achieve the above purpose, the first aspect of the embodiment of the present application provides an eye tracking method applied to an image sensor, wherein the image sensor includes an image collection module, an artificial intelligence model inference module and an external interface module, and the method includes:

[0005] The image collection module collects the eye image of the user;

[0006] The artificial intelligence model inference module infers based on the eye image according to the preset algorithm to obtain an inference result;

[0007] The external interface module outputs corresponding eye recognition data according to the inference result.

[0008] As one of the implementation manners, the image sensor further includes an image processing module, and after the image collection module collects the eye image of the user, the method includes:

[0009] The image processing module processes the eye image into an input format corresponding to the preset algorithm.

[0010] As one of the implementations, the eye recognition data includes a gaze angle, and the gaze angle recognition includes:

[0011] extracting a key point in the eye image;

[0012] determining a gaze angle of the eye image according to the eye parameter of the user and the key point.

[0013] As one of the implementations, the key point includes at least one of a pupil, an iris, an eyeball, and an eyelid.

[0014] As one of the implementations, the eye recognition data includes an eye parameter, and the eye parameter recognition includes:

[0015] guiding the user's eyeball to a direction of a preset landmark point;

[0016] collecting an eye image of the user when it is detected that the user is gazing at the landmark point;

[0017] extracting a feature point in the eye image for calculating the eye parameter;

[0018] calculating the eye parameter according to the feature point.

[0019] As one of the implementations, the guiding the user's eyeball to the direction of the landmark point includes at least one of:

[0020] controlling a preset landmark point in a light source to flash;

[0021] controlling a preset landmark point displayed on a screen in front of the user to flash.

[0022] As one of the implementations, the way of detecting that the user is gazing at the landmark point includes: when an angle between a line of sight of the user and a line from the pupil to the landmark point is within a preset range.

[0023] As one of the implementations, the eye recognition data includes an eye movement state, and the eye movement state recognition includes:

[0024] performing eye feature extraction on the eye image;

[0025] determining a current eye movement state of the user according to the eye feature, the eye parameter of the user, and a historical eye movement state.

[0026] As one of the implementations, the image sensor further includes a central controller, and the method further includes:

[0027] the central controller outputs a light source control instruction according to the eye image, and the light source control instruction is used to adjust at least one of a brightness, a working frequency, and a switch of the light source.

[0028] As one of the implementations, the adjusting the brightness of the light source comprises:

[0029] The brightness of the light source is adjusted by changing the duty cycle of the light source.

[0030] As one of the implementations, the adjusting the brightness of the light source further comprises:

[0031] If the brightness of the eye image is lower than the lower limit of the target brightness interval, the brightness of the eye image is re-acquired by at least one of increasing the exposure time, the gain of the image acquisition, and increasing the brightness of the light source until the brightness of the re-acquired eye image is within the target brightness interval.

[0032] If the brightness of the eye image is higher than the upper limit of the target brightness interval, the brightness of the eye image is re-acquired by at least one of decreasing the brightness of the light source, reducing the gain, the exposure time of the image acquisition until the brightness of the re-acquired eye image is within the target brightness interval.

[0033] As one of the implementations, the adjusting the brightness of the light source further comprises:

[0034] If the brightness of the eye image is lower than the lower limit of the target brightness interval, the brightness of the eye image is re-acquired by sequentially increasing the exposure time, the gain of the image acquisition, and increasing the brightness of the light source until the brightness of the re-acquired eye image is within the target brightness interval.

[0035] If the brightness of the eye image is higher than the upper limit of the target brightness interval, the brightness of the eye image is re-acquired by sequentially decreasing the brightness of the light source, reducing the gain, the exposure time of the image acquisition until the brightness of the re-acquired eye image is within the target brightness interval.

[0036] To achieve the above object, a terminal is provided in a second aspect of the embodiments of the present application, and the terminal comprises at least one processor and at least one memory, the memory is coupled to the processor and stores instructions for execution by the processor, and the instructions, when executed by the processor, cause the terminal to perform the eye tracking method according to the first aspect.

[0037] To achieve the above object, a computer storage medium is provided in a third aspect of the embodiments of the present application, and the computer storage medium stores computer program instructions; the computer program instructions are executed by a processor to implement the eye tracking method according to the first aspect.

[0038] Compared with the prior art, the technical scheme has the following beneficial effects:

[0039] The application provides an eye tracking method, a terminal and a computer storage medium, the eye tracking method is applied to an image sensor, the image sensor comprises an image acquisition module, an artificial intelligence model reasoning module and an external interface module, and the method comprises the following steps: the image acquisition module acquires an eye image of a user; the artificial intelligence model reasoning module performs reasoning based on the eye image according to a preset algorithm, and obtains a reasoning result; and the external interface module outputs corresponding eye recognition data according to the reasoning result. The application can track the eye state of the user in real time, realizes the control of a light source and the output of information only according to the eye image through the acquisition and reasoning of the eye image, and improves the use experience of the user. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A structural schematic diagram of an image sensor for eye movement tracking is provided for the embodiment of the application.

[0041] Figure 2 A structural schematic diagram of an artificial intelligence model reasoning module is provided for the embodiment of the application.

[0042] Figure 3 An application scenario diagram of an image sensor for eye movement tracking is provided for the embodiment of the application.

[0043] Figure 4 A structural schematic diagram of an eye tracking device is provided for the embodiment of the application.

[0044] Figure 5 A flowchart of an eye tracking method is provided for the embodiment of the application.

[0045] Figure 6 A flowchart of line-of-sight angle recognition is provided for the embodiment of the application.

[0046] Figure 7 A flowchart of eye parameter recognition is provided for the embodiment of the application.

[0047] Figure 8 A flowchart of eye movement state recognition is provided for the embodiment of the application.

[0048] Figure 9 One of flowcharts of light source control is provided for the embodiment of the application.

[0049] Figure 10 The other of flowcharts of light source control is provided for the embodiment of the application.

[0050] Figure 11 A flowchart of a line-of-sight landing point estimation method is provided for the embodiment of the application.

[0051] Figure 12 An application scenario diagram of a line-of-sight landing point estimation method is provided for the embodiment of the application.

[0052] Figure 13 A specific flowchart of the line-of-sight landing point estimation method provided by the embodiment of the present application is shown in the figure.

[0053] Figure 14 A flowchart of the multi-sensor cooperative work provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0054] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application, which are only used to explain the present application, and do not limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. In the patent specification, “one embodiment” or “an embodiment” means that the specific features, structures or characteristics described in the examples are included in at least one embodiment of the present application. The specific features, structures or characteristics can be included in integrated circuits, electronic circuits, combination logic circuits or other suitable components that provide the described functions. In addition, only the contents associated with the point of the present application are described in the specification, and other contents can be understood by those skilled in the art in combination with the prior art.

[0055] First, in order to better understand the present application, the inventive concept of the present application is described in detail as follows.

[0056] First embodiment

[0057] Figure 1 A structural schematic diagram of the image sensor for eye tracking provided by the embodiment of the present application is shown in the figure. Figure 1 The image sensor for eye tracking of the embodiment of the present application includes an image acquisition module, an image processing module, an artificial intelligence model reasoning module, an external interface module and a central controller module.

[0058] It should be noted that when performing the eye tracking task, the central controller module distributes the tasks in different working states to each module to complete in order to realize the functions of light source control, image acquisition, line-of-sight angle recognition, eye parameter recognition, eye movement state recognition, multi-sensor data synchronization and three-dimensional line-of-sight landing point recognition. The image sensor for eye tracking of the present application can work cooperatively, or can work cooperatively with other ordinary image sensors. In the multi-sensor cooperative work state, the function of three-dimensional line-of-sight landing point recognition can be realized.

[0059] In an embodiment, the image acquisition module is configured to acquire the first image of the user's eyeball, and the first image of the user's eyeball can be acquired in real time. The image acquisition module acquires images according to the image acquisition instruction output by the external interface module.

[0060] It should be noted that the image acquisition module can be composed of a pixel array, a control unit, a pixel value reading unit, and an analog-to-digital signal conversion unit. In an embodiment, considering that the light source is close to the human eye, and in order to control the power consumption, the light source is controlled to be in a flickering state. In a further example, the control unit in the image acquisition module receives the control of the external interface module to ensure that the images meeting the dynamic range requirement are acquired in a synchronized manner.

[0061] In an embodiment, the image processing module is configured to process the first image into an input format corresponding to a preset inference algorithm, to obtain a second image, and subsequent artificial intelligence inference is based on the second image.

[0062] It should be noted that the images acquired by the image acquisition module will be transmitted to the image processing module, and the function of the image processing module is to process the user's eye images into the input format required by different algorithms according to the requirements of different algorithms. Preferably, the image processing module integrates hardware circuits of bad pixel compensation, image denoising, gamma correction, and lens distortion correction algorithms. The image processing module can use an existing image processing module ISP.

[0063] In an embodiment, the artificial intelligence model inference module is configured to infer the first image or the second image (when the image processing module exists) according to a preset inference algorithm, to obtain an inference result.

[0064] Specifically, different types of artificial intelligence models can be run on the artificial intelligence model inference module, and the inference of the artificial intelligence model is completed after receiving the input. According to the algorithm type, a specific output is given to the external interface module. The inference result includes eye recognition data. Specifically, the eye recognition data includes at least one of a gaze angle, an eyeball parameter, and an eye movement state. Since the user's private information such as the original human eye picture and the user's eyeball parameter are saved in the image sensor for eye movement tracking, only the light source control instruction or the eye recognition data is output to other devices, so as to achieve the purpose of protecting the user's privacy.

[0065] Figure 2 A structural diagram of the artificial intelligence model inference module provided for the embodiment of the present application is shown in FIG. 4. Figure 2 The artificial intelligence model inference module includes an on-chip storage unit and a neural network processor, and the neural network processor includes an instruction scheduling unit, a matrix calculation unit, a vector calculation unit, and a memory management unit.

[0066] An instruction scheduling unit is configured to send data transfer instructions and calculation instructions to corresponding units and manage execution sequences of the respective calculation units;

[0067] A matrix calculation unit is configured to perform matrix multiplication calculation;

[0068] A vector calculation unit is configured to perform vector addition and subtraction calculation and dot product calculation;

[0069] A memory management unit is configured to implement data transfer between the units;

[0070] An on-chip storage unit is configured to store weights, inputs, intermediate calculation results and outputs of the artificial intelligence model inference module.

[0071] It should be noted that the artificial intelligence model inference module includes the on-chip storage unit and the neural network processor. The on-chip storage unit can be a storage unit located on a chip and configured to store a preset inference algorithm program. The neural network processor is configured to execute the preset inference algorithm program pre-stored in the on-chip storage unit to perform inference on the first image information.

[0072] In an embodiment, the external interface module is configured to output the inference result and receive data sent by other image sensors.

[0073] It should be noted that the external interface module controls the on-off of the specific light source, the duty cycle of the specific light source or uploads the eye movement state of the user to the CPU of the virtual reality / augmented reality in real time according to the output of the artificial intelligence model inference module, or exchanges information with other image sensors. The external interface module executes the instructions of the central controller according to the current system working state, thereby performing light source control, data sending and receiving operations.

[0074] In an embodiment, the central controller module is configured to schedule and control other modules.

[0075] It should be noted that the image acquisition module acquires images through the central controller module according to the image acquisition instructions output by the external interface module. In addition to adjusting the exposure parameters of the image acquisition module, the central controller module of the present application can also adjust the brightness of the light source by controlling the duty cycle of the light source through the external interface module, thereby ensuring the frame rate and image quality of image acquisition at the same time.

[0076] Specifically, the central controller module adjusts the image brightness based on at least one of the exposure parameters and the external interface module. The method of adjusting the image brightness based on the external interface module includes at least one of controlling the on-off of the light source, adjusting the brightness of the light source and adjusting the working frequency of the light source.

[0077] In addition, the central controller module of the application can also complete sub-tasks including light source control, image acquisition, line of sight angle estimation, user eye parameter estimation, user eye movement state estimation, multi-sensor data synchronization, three-dimensional line of sight landing point estimation on the image sensor system with the lowest delay according to the requirements of the eye movement tracking task.

[0078] Figure 3 The application scenario diagram of the image sensor for eye movement tracking provided by the embodiment of the application is shown in the figure. Figure 3 The image sensor for eye movement tracking of the application is located in front of the eyeball, and the distance between the image sensor and the eyeball is not more than 5 cm. The image sensor can be placed at any position in front of the eyeball that can cover the eyeball. A plurality of light sources are used in cooperation with the image sensor of the application. Preferably, the light sources are placed as follows: when the user looks straight ahead with both eyes, all the light sources form reflection points on the user's iris, so that the image sensor can receive all the reflection points.

[0079] The image sensor for eye movement tracking can maintain data connection with the CPU of the VR / AR device to interact with data and control signals. The VR / AR device sends a signal to the image sensor for eye movement tracking to request to obtain the eye movement information of the user. After receiving the request signal, the image sensor returns the eye movement information of the user.

[0080] The application does not limit the position of the CPU, but in general, the CPU is very close to the image sensor system of the application and is connected directly through a data line.

[0081] In summary, the image sensor for eye movement tracking provided by the embodiment of the application includes an image acquisition module, an artificial intelligence model reasoning module, an external interface module, and a central controller module. The image acquisition module acquires a first image of the user's eyeball. The artificial intelligence model reasoning module performs reasoning based on the first image according to a preset reasoning algorithm to obtain a reasoning result. The external interface module outputs the reasoning result. The central controller module schedules and controls other modules. The image sensor for eye movement tracking of the application has high integration degree, low power consumption, and small space occupation, can track the eyeball state of the user in real time and perform reasoning, greatly improving the efficiency of tracking the user's eyeball; the application can reduce the delay of the eyeball tracking system to below 3 ms while maintaining the same accuracy as the existing system; the application does not store the eye image information of the user, but only provides the current eyeball state of the user to the upper processor, thereby effectively protecting the privacy of the user; in addition to the image acquisition and eye movement tracking integrated manner, the application also supports information interaction between different sensors, so that the eyeball state of the user can be tracked in real time through multiple image sensors, and the three-dimensional line of sight landing point of the user can be accurately positioned.

[0082] Second embodiment

[0083] Figure 4 The structure diagram of the eye tracking device provided by the embodiment of the present application is shown in the figure. Please refer to Figure 4 The embodiment of the present application also provides an eye tracking device, which comprises the image sensor for eye tracking, the illumination module and the CPU as described above. Specifically, the eye tracking device is composed of the illumination module composed of a plurality of light sources, the image sensor integrated with the artificial intelligence model processing capability and the light source control capability, and a plurality of algorithms carried on the sensor.

[0084] It should be noted that the illumination module comprises a plurality of light sources, and when the user looks straight ahead with both eyes, the light sources form reflection points on the iris of the eyeball. Optionally, the plurality of light sources are infrared LED light sources arranged in a specific manner, for example, the light sources are symmetrically distributed at the positions close to the corners of the eyeball, and the number can be two or four, as shown in the figure. In this way, the recognition efficiency can be improved, the interference on the user's eyeball can be reduced, and good user experience is achieved. Figure 3

[0085] The image sensor is installed with a program for recognizing the behavior of the user's eyeball, for collecting image information including the eyeball and the reflection points, and outputting control instructions and / or eyeball recognition data according to the image information. The image sensor can be combined with an infrared filter, a lens and the like to form a perception device of the eye tracking device to receive light information.

[0086] The CPU is installed with a control program of each device, which can control the data interaction between the image sensor and the illumination module and other image sensors. In the embodiment, the CPU can be a central processor of a virtual reality / augmented reality device, or a specialized chip customized for the eye tracking task.

[0087] In an embodiment, the eye tracking device is applied to a virtual reality device and / or an augmented reality device. Specifically, the eye tracking device can exist in the form of a separate device similar to glasses or a part of a virtual reality / augmented reality device, and has the ability to track the eyeball movement of the user.

[0088] The eye tracking device provided by the embodiment of the present application comprises all the technical features of the above-mentioned image sensor, and the description and explanation contents are basically the same as those of the above-mentioned embodiment, which will not be repeated here.

[0089] ​In summary, the eye tracking device provided by the embodiment of the present application comprises an image sensor for eye tracking, which is used to collect image information comprising an eyeball and a reflection point, and output control instructions and / or eyeball recognition data according to the image information. Compared with the prior art, the eye tracking device of the present application has the following beneficial effects: the control of the illumination device and the recognition of the user's eyeball behavior are all integrated on the image sensor to complete the functions, which reduces the power consumption and the occupied space. The present application can reduce the reaction output time to below 3ms while maintaining the same accuracy as the existing eye tracking device; at the same time, the overall power consumption is low, and the occupied space is significantly reduced. The present application does not store the user's eye image information, but only provides the current user's eyeball state to the upper processor, thereby effectively protecting the user's privacy. In addition to the integrated image collection and eye tracking method, the present application also supports information interaction between different sensors, so that the user's eyeball state can be tracked in real time through multiple image sensors.

[0090] Third embodiment

[0091] Figure 5 The flowchart of the eye tracking method provided by the embodiment of the present application is shown in FIG. 1. Figure 5 The eye tracking method of the embodiment of the present application is applied to the image sensor provided by the first embodiment, the image sensor comprising an image collection module, an artificial intelligence model reasoning module and an external interface module, and the method comprising:

[0092] Step 201: The image collection module collects the user's eye image.

[0093] Step 202: The artificial intelligence model reasoning module performs reasoning based on the eye image according to a preset algorithm to obtain a reasoning result.

[0094] Step 203: The external interface module outputs corresponding eyeball recognition data according to the reasoning result.

[0095] In the embodiment of the present application, the eyeball recognition data comprises a line of sight angle, an eyeball parameter and an eye movement state. Thus, the eye tracking method can realize the functions of line of sight angle recognition, user eyeball parameter, user eye movement state recognition, etc. It should be noted that the above functions can be performed simultaneously or separately, and the specific implementation situation is determined according to the demand, and the present application does not limit it.

[0096] In an embodiment, the image sensor further comprises an image processing module, and after the image collection module collects the user's eye image, it comprises:

[0097] The image processing module processes the eye image into an input format corresponding to the preset algorithm.

[0098] Specifically, the eye image of the user is collected by an image collection module in the image sensor, and the eye image is transmitted to an image processing module. The image processing module is used to process the eye image of the user into an input format required by different algorithms according to the requirements of different algorithms, so as to prepare for the inference of the eye image in the next step. Wherein, the eye image described herein can be understood as the first image of the embodiment, and the image processed by the image processing module can be understood as the second image. In addition, those skilled in the art can understand that the eye images described in the subsequent embodiments can be images in an input format adapted to the preset algorithm after being processed by the image processing module.

[0099] In an embodiment, the eye recognition data includes a gaze angle, and the gaze angle recognition includes:

[0100] extracting key points in the eye image;

[0101] determining the gaze angle of the eye image according to the eye parameters and the key points of the user.

[0102] Figure 6 A flowchart of the gaze angle recognition provided by the embodiment of the present application is shown.

[0103] Please refer to Figure 6 The gaze angle recognition algorithm runs on the artificial intelligence model inference module of the image sensor, and is used to estimate the gaze direction of the single eye of the user. After the image collection module collects the eye image, the eye image is sent to the image processing module, and the image processing module processes the eye image into an input format suitable for artificial intelligence model inference. After obtaining the processed eye image from the image processing module, the eye feature extraction method in the gaze angle recognition algorithm is used to obtain key point information such as pupil, iris, eyeball and eyelid of the user, so as to obtain the eye feature in the eye image. The central controller calculates the current gaze angle of the user in the eye image by combining the eye feature inferred by the artificial intelligence model inference module with the pre-stored personalized eye parameters of the user. Wherein, other key point information and eye parameters required to obtain the gaze angle in the prior art can be obtained based on the image sensor of the present application, so as to obtain the gaze angle of the user; then the gaze angle data can be output to the external CPU through the external interface module, without directly outputting the original eye image, so as to protect the privacy information of the user.

[0104] In an embodiment, the eye recognition data includes eye parameters, and the eye parameter recognition includes:

[0105] guiding the user's eye to look at the direction of the preset landmark point;

[0106] collecting the eye image of the user when detecting that the user is gazing at the landmark point;

[0107] extracting feature points in the eye image for calculating eyeball parameters;

[0108] calculating eyeball parameters according to the feature points.

[0109] In an embodiment, the method of guiding the user's eyeball to look at the direction of the landmark point includes at least one of the following:

[0110] controlling the preset landmark point in the light source to flash;

[0111] controlling the preset landmark point displayed on the screen in front of the user's eyes to flash.

[0112] In an embodiment, the method of detecting that the user is gazing at the landmark point includes: when the angle between the user's line of sight and the line from the pupil to the landmark point is within a preset range.

[0113] Figure 7 A flowchart of the eyeball parameter identification provided by the embodiment of the present application.

[0114] Reference Figure 7 The embodiment of the present application provides a user eyeball parameter calibration method. When it is necessary to calibrate the user's eyeball parameters, an external interface module is used to control the light source to flash or make the screen flash. Specifically, the visible light landmark point light source can be controlled to flash by using an image sensor, or the landmark point displayed on the screen in front of the user's eyes can be used to guide the user's eyeball to look in the direction of the landmark point. The central controller of the image sensor can be used to control the visible light source to flash, or the controller of an AR device can be used to control the landmark point on the screen to flash, or the central controller of the image sensor can be used to control the landmark point on the screen to flash. When the system detects that the user is gazing at the landmark point, the image acquisition module starts to acquire the eye image of the user, an artificial intelligence model reasoning module extracts feature points for calculating eyeball parameters, and a central controller module calculates the eyeball parameters according to the extracted feature points. After the parameter calculation is completed, the eyeball parameters of the user are output, and can be saved in the image sensor. Other feature point information required for obtaining eyeball parameters in the prior art can be obtained based on the image sensor of the present application to obtain the eyeball parameters of the user.

[0115] It should be noted that when it is determined whether the user is gazing, it can be determined whether the angle between the user's line of sight and the line from the pupil to the landmark point is within a preset range. The preset range reference value can be less than or equal to 5°. If it is within the preset range, it is determined that the user is gazing at the landmark point. This method ensures the reliability of the user's eyeball parameters.

[0116] In an embodiment, the eyeball recognition data includes an eye movement state, and the eye movement state recognition includes:

[0117] extracting eye features from the eye image;

[0118] determining the current eye movement state of the user according to the eye features, the eyeball parameters and the historical eye movement state of the user.

[0119] Figure 8 A flowchart of the eye movement state recognition provided by the embodiment of the present application.

[0120] Please refer to Figure 8 Since the eye movement state of the user, such as closing eyes and rapid eye movement, often occurs, the confidence of the line-of-sight angle recognition and the like in these states is low, it is better to accurately recognize the current eye movement state of the user and then calculate the line-of-sight angle and the like, and the current eye movement state of the user needs to be informed to the upper CPU. Of course, the image sensor of the present application can also be used to obtain the eye movement state when other eye movement states are needed. The user eye movement state recognition algorithm collects the eye image through the image acquisition module, uses the eye image as the input, extracts the features through the artificial intelligence model reasoning module to obtain the position information of the pupil, the iris, the eyeball and the eyelid of the user, and then estimates the current eye movement state of the user in combination with the pre-stored eye movement state of the user and the eyeball parameters of the user. Then the eye movement state can be output to the external CPU through the external interface module. Among them, the other eye feature information needed to obtain the eye movement state in the prior art can be obtained based on the image sensor of the present application to obtain the eye movement state of the user.

[0121] In an embodiment, the image sensor further comprises a central controller, and the method further comprises:

[0122] The central controller outputs a light source control instruction according to the eye image, and the light source control instruction can be used to adjust at least one of the brightness, the working frequency and the switch of the light source.

[0123] Among them, the brightness of the light source can be adjusted by changing the duty cycle of the light source.

[0124] Optionally, when adjusting the brightness of the light source, if the brightness of the eye image is lower than the lower limit of the target brightness interval, at least one of increasing the exposure time, the gain of the collected image and increasing the brightness of the light source is used until the brightness of the re-collected eye image is within the target brightness interval.

[0125] If the brightness of the eye image is higher than the upper limit of the target brightness interval, at least one of reducing the brightness of the light source, reducing the gain and the exposure time of the collected image is used until the brightness of the re-collected eye image is within the target brightness interval.

[0126] Preferably, when adjusting the brightness of the light source, if the brightness of the eye image is lower than the lower limit of the target brightness interval, the brightness of the eye image is adjusted by sequentially increasing the exposure time, the gain of the collected image, and the brightness of the light source until the brightness of the newly collected eye image is within the target brightness interval. If the brightness of the eye image is higher than the upper limit of the target brightness interval, the brightness of the eye image is adjusted by sequentially decreasing the brightness of the light source, the gain, and the exposure time of the collected image until the brightness of the newly collected eye image is within the target brightness interval. In this way, the brightness of the image and / or the light source is adjusted in real time according to the collected eye image, so that the quality of the collected image is higher, the process of collecting the image is more intelligent, and the user experience is better.

[0127] Figure 9 One of the flowcharts of the light source control provided by the embodiments of the present application.

[0128] Please refer to Figure 9 If the brightness of the eye image collected in the last frame is too low, such as lower than the lower limit of the target brightness interval, the exposure time parameter of the collected image is first increased, and it is determined whether the brightness of the newly collected image meets the brightness requirement, i.e., whether it is within the target brightness interval. The target brightness interval can be set according to the requirements of the eye tracking task. If yes, the brightness adjustment is ended. If the brightness of the newly collected image is still lower than the lower limit of the target brightness interval, and the exposure time of the collected image has not reached the upper limit, the exposure time is continuously increased until the collected image meets the brightness requirement. If the brightness of the newly collected image is lower than the lower limit of the target brightness interval, and the exposure time of the collected image has been adjusted to the upper limit, the gain parameter of the image is tried to be increased until the image meets the brightness requirement. If the gain parameter has reached the upper limit, and the brightness of the collected image is still lower than the lower limit of the target brightness interval, the brightness of the light source is increased, and the eye image of the user is newly collected until the collected eye image meets the brightness requirement.

[0129] Figure 10 The second flowchart of the light source control provided by the embodiments of the present application.

[0130] Please refer to Figure 10 If the brightness of the eye image collected in the last frame is too high, such as higher than the upper limit of the target brightness interval, the brightness of the light source is first tried to be decreased, and the brightness of the light source can be adjusted by adjusting the duty cycle of the light source. If the brightness of the newly collected eye image of the user meets the brightness requirement after the brightness of the light source is decreased, the brightness adjustment is ended. If the brightness of the newly collected image is still higher than the upper limit of the target brightness interval, and the adjustment of the duty cycle of the light source has reached the lower limit, the gain parameter is tried to be decreased until the collected image meets the brightness requirement. If the gain parameter has reached the lower limit of the adjustment, and the brightness of the newly collected image is still higher than the upper limit of the target brightness interval, the exposure time of the collected image is decreased until the brightness of the collected eye image meets the brightness requirement.

[0131] The eye tracking method provided by the embodiment of the application is applied to an image sensor, the image sensor comprises an image acquisition module, an artificial intelligence model reasoning module and an external interface module, and the method comprises the following steps: the image acquisition module acquires an eye image of a user; the artificial intelligence model reasoning module performs reasoning based on the eye image according to a preset algorithm, and obtains a reasoning result; and the external interface module outputs corresponding eye recognition data according to the reasoning result. The eye tracking method can track the eye state of the user in real time, and through the acquisition and reasoning of the eye image, the control of the light source and the output of information are realized only according to the eye image, thereby improving the use experience of the user.

[0132] The application also provides a terminal comprising at least one processor and at least one memory, the at least one memory being coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, causing the terminal to perform the eye tracking method as described above.

[0133] The application also provides a computer storage medium, the computer storage medium storing computer program instructions; the computer program instructions are executed by a processor to implement the eye tracking method as described above.

[0134] Fourth embodiment

[0135] Figure 11 A flowchart of the line-of-sight landing point estimation method provided by the embodiment of the application is shown.

[0136] For reference Figure 11 The line-of-sight landing point estimation method provided by the embodiment of the application is applied to a first image sensor, the first image sensor comprising an artificial intelligence model reasoning module, wherein the first image sensor can adopt the image sensor provided by the first embodiment of the application, and the line-of-sight landing point estimation method comprises the following steps:

[0137] Step 301: acquiring first line-of-sight information of a first eye according to the first image sensor, the first line-of-sight information comprising a line-of-sight angle of the first eye;

[0138] Step 302: acquiring second line-of-sight information of a second eye according to a second image sensor;

[0139] Step 303: acquiring relative position information of the first image sensor and the second image sensor;

[0140] Step 304: the artificial intelligence model reasoning module calculates a three-dimensional line-of-sight landing point according to the first line-of-sight information, the second line-of-sight information and the relative position information.

[0141] In view of the problems existing in the prior system, the application integrates the function of three-dimensional visual line landing point estimation using multiple sensors into the image sensor, and the image sensor directly outputs the three-dimensional visual line landing point information of the user, thereby saving the bandwidth required for image transmission. Meanwhile, the image sensor system provided by the application can work with other image sensor systems of the same type, or can work with the existing ordinary image sensor. In addition, multiple image sensor systems provided by the application can be expanded, thereby exceeding the quantity limitation of the existing system, meeting the demand of high-precision real-time eye movement tracking task, and can also be used for other applications such as face tracking and gesture tracking. It should be noted that the above functions can be performed simultaneously or separately, and the specific implementation situation is determined according to the demand, and the application does not limit the implementation situation.

[0142] In an embodiment, the first image sensor further comprises an image acquisition module, and the first visual line information is obtained, comprising:

[0143] The image acquisition module acquires the first image information of the first eyeball;

[0144] The artificial intelligence model inference module obtains the first visual line information according to the first image information.

[0145] In an embodiment, the first image sensor further comprises an external interface module, and the second visual line information is obtained according to the second image sensor, comprising:

[0146] The external interface module receives the second visual line information sent by the second image sensor; or,

[0147] The external interface module receives the second image information of the second eyeball sent by the second image sensor, and the artificial intelligence model inference module of the first image sensor obtains the second visual line information according to the second image information.

[0148] In an embodiment, the second image sensor comprises an image acquisition module, an artificial intelligence model inference module and an external interface module, so as to obtain the second visual line information based on the second image sensor. That is, the second image sensor and the first image sensor can be the same type of sensor, and the second visual line information can be directly obtained based on the artificial intelligence model, without the need for processing in the first image sensor.

[0149] It should be noted that the first visual line information and the second visual line information can be obtained by the above-mentioned embodiments of the application.

[0150] Figure 12 The application scenario diagram of the visual line landing point estimation method provided by the fourth embodiment of the application.

[0151] For reference Figure 12The first image sensor 40 is arranged corresponding to the first eyeball of the user to acquire first image information of the first eyeball through an image acquisition module of the first image sensor 40, and first line-of-sight information is obtained by an artificial intelligence model reasoning module of the first image sensor 40 according to the first image information, for example, arranged below the first eyeball of the user. Correspondingly, the second image sensor 41 is arranged corresponding to the second eyeball of the user to acquire second image information of the second eyeball through an image acquisition module of the second image sensor 41, and second line-of-sight information is obtained by an artificial intelligence model reasoning module of the second image sensor 41 according to the second image information, for example, arranged below the second eyeball of the user. In other embodiments, if the second image sensor 41 does not have an artificial intelligence model reasoning module, i.e., a conventional image sensor, the second image information is sent to the first image sensor 40, and then the second line-of-sight information is obtained by the artificial intelligence model reasoning module of the first image sensor 40 according to the second image information. In this way, at least two image sensors are used to capture images of the eyes of the user to obtain the line-of-sight direction information of each eye, and after the spatial positional relationship of the image sensors is known, the three-dimensional line-of-sight landing point information of the user is obtained.

[0152] In addition, it should be noted that the number of second image sensors can be one, two or more. When there are two or more second image sensors, at least one of the second image sensors is used to obtain information of the second eyeball, and the remaining second image sensors can be used to obtain information of the first eyeball or the second eyeball. It should be noted that if there are at least two image sensors for obtaining information of the same eyeball, such as the second eyeball, in an embodiment, the information obtained by all image sensors for obtaining information of the second eyeball is averaged to represent the information of the eyeball. Of course, it can also be designed according to other requirements such as weight in the art.

[0153] In an embodiment, the artificial intelligence model reasoning module calculates the three-dimensional line-of-sight landing point according to the first line-of-sight information, the second line-of-sight information and the relative position information, including:

[0154] The center of the pupil of the first eyeball is taken as the origin O, the line connecting the center of the pupil of the first eyeball and the center of the pupil of the second eyeball is taken as the X axis, the positive direction of the X axis is that the first eyeball points to the second eyeball, the vertical line of the X axis in the plane of the eyeball section passing through the center of the pupil of the first eyeball and taking the origin O is taken as the Y axis, the positive direction of the Y axis is upward (when standing, the direction of the foot pointing to the head), and the straight line passing through the origin O and perpendicular to the eyeball section passing through the center of the pupil of the first eyeball is taken as the Z axis, the positive direction of the Z axis points outward on the paper, and the OXYZ coordinate system is established;

[0155] respectively, according to the relative position information of the first image sensor and the second image sensor;

[0156] According to the angle information and the coordinate information, the coordinate information of the three-dimensional visual line landing point is obtained.

[0157] In an embodiment, the angle information includes a horizontal included angle and a vertical included angle, and the corresponding angle information of the first visual line information and the second visual line information in the OXYZ coordinate system is obtained, including:

[0158] The projection of the first visual line information and the second visual line information in the OXYZ coordinate system in the X-Z plane and the included angle with the Z axis are obtained as the corresponding horizontal included angles and respectively.

[0159] At least the projection of the first visual line information in the OXYZ coordinate system in the Y-Z plane and the included angle with the Z axis are obtained as the corresponding vertical included angle.

[0160] In an embodiment, the calculation formula of obtaining the coordinate information of the three-dimensional visual line landing point according to the angle information and the coordinate information is:

[0161]

[0162] D x =D z tanα1

[0163] D y =D z tanθ1

[0164] In the formula, (D x , D y , D z ) is the coordinate of the three-dimensional visual line landing point, (x, y, z) is the coordinate of the visual line starting point of the second eyeball, α1 is the horizontal included angle of the first visual line information, θ1 is the vertical included angle of the first visual line information, and α2 is the horizontal included angle of the second visual line information.

[0165] It should be noted that, assuming the value of the line of sight angle obtained by the first image sensor 40 in the horizontal direction and the vertical direction is α1 and θ1 respectively, and the value of the line of sight angle obtained by the second image sensor 41 in the horizontal direction and the vertical direction is α2 and θ2 respectively. Among them, the projection of the line of sight in the X-Z plane is the same direction as the X axis, and the projection of the line of sight in the Y-Z plane is the same direction as the Y axis. The origin O of the OXYZ coordinate system is defined as the pupil center of the first eye when the user looks straight, and the function of the coordinate system is to describe the specific position of the three-dimensional line of sight landing point of the user. The X direction is the connecting line of the two eye pupils, the Z direction is perpendicular to the paper, and the Y direction is perpendicular to the plane formed by the X axis and the Z axis.

[0166] In an embodiment, the method further comprises:

[0167] determining whether there is an abnormal eye movement state according to the first line of sight information and the second line of sight information;

[0168] if there is no abnormal eye movement state (such as fixation state), outputting the information of the three-dimensional line of sight landing point;

[0169] if there is an abnormal eye movement state, outputting the information of the abnormal eye movement state.

[0170] In an embodiment, the way of determining whether there is an abnormal eye movement state comprises:

[0171] guiding the user's eyeball to look in the direction of the preset landmark point;

[0172] detecting whether the angle between the user's line of sight and the connecting line of the pupil to the preset landmark point is within a preset range, and if it is within the preset range, there is no abnormal eye movement state.

[0173] Among them, in this embodiment, the user's fixation state is considered to be a state without abnormal eye movement, and the method for judging the user's fixation state can refer to the specific ways in the above-mentioned embodiments of the application. For example, the preset range is set to be less than 5°.

[0174] Figure 13 The specific flowchart of the line of sight landing point estimation method provided by the fourth embodiment of the application.

[0175] Please refer to Figure 13, first, the first image sensor is used to obtain the line-of-sight angle of the first eyeball, and in a further example, the eye movement state information can also be obtained, and the second image sensor is used to obtain the line-of-sight angle and eye movement state information of the second eyeball. Then, the three-dimensional line-of-sight landing point of the user is calculated according to the position information of the second image sensor relative to the first image sensor and the line-of-sight of the first image sensor and the second image sensor. In an example, the normal eye movement state means that the user is in a state of staring, and the abnormal eye movement state means that the user is in a state of closing eyes, rapid eye movement, etc. that cannot obtain the accurate three-dimensional line-of-sight landing point of the user. According to the eye movement state information, it is determined whether the first eyeball or the second eyeball has an abnormal eye movement state. If there is an abnormal eye movement, the related information of the abnormal eye movement state is output; if there is no abnormal eye movement, the three-dimensional line-of-sight landing point information is output.

[0176] In an embodiment, before obtaining the first line-of-sight information, the method further comprises:

[0177] The first image sensor sends an instruction of synchronously collecting images to the second image sensor. Thus, the information can be synchronously obtained based on multiple sensors.

[0178] Figure 14 A flowchart of the multi-sensor cooperative working provided by the fourth embodiment of the present application is shown.

[0179] Please refer to Figure 14 Before collecting the eye images, the first image sensor is determined as the master sensor, and the second image sensor is determined as the slave sensor. The master sensor is responsible for sending a synchronization signal to other slave sensors and receiving data from the slave sensors. After the master sensor sends an instruction of synchronously collecting images, the master sensor and the slave sensor collect the eye images respectively, so that the multiple image sensors are kept synchronous when estimating the three-dimensional line-of-sight landing point, that is, the collection of the eye image data is completed at the same time, and the data processing and data transmission are completed within a limited time. The slave sensor is responsible for collecting data according to the synchronization signal, and if the slave sensor has eye movement tracking capability, the line-of-sight angle and the eye movement state information are obtained by processing the collected eye images. If the slave sensor does not have eye movement tracking capability, the collected eye image data is sent to the master sensor for processing. After the master sensor processes the collected eye images to obtain the line-of-sight angle and the eye movement state information, the data sent by the slave sensor is received. If the data sent by the slave sensor is image data, the corresponding line-of-sight angle and eye movement state are obtained by processing the image data. If the data sent by the slave sensor is not image data, the three-dimensional line-of-sight landing point is calculated and the calculation result is output according to the line-of-sight angle corresponding to the two eyeballs and the position relationship.

[0180] In an embodiment, the first image sensor further comprises an image processing module, the image processing module processes the first image collected by the image collection module into an input format corresponding to a preset inference algorithm, the artificial intelligence model inference module performs inference based on the obtained processed image according to the preset inference algorithm, and obtains an inference result; and / or, the artificial intelligence model inference module comprises an on-chip storage unit and a neural network processor; the on-chip storage unit is used at least for storing a preset inference algorithm program; and the neural network processor is used for executing the preset inference algorithm program pre-stored in the on-chip storage unit to perform inference.

[0181] The line-of-sight landing point estimation method of the embodiment of the application is applied to a first image sensor, the first image sensor comprises an image collection module, an artificial intelligence model inference module and an external interface module, and the line-of-sight landing point estimation method comprises the following steps: obtaining first line-of-sight information, the first line-of-sight information comprising a line-of-sight angle of a first eyeball; obtaining second line-of-sight information of a second eyeball according to a second image sensor; obtaining relative position information of the first image sensor and the second image sensor; and the artificial intelligence model inference module calculating a three-dimensional line-of-sight landing point according to the first line-of-sight information, the second line-of-sight information and the relative position information. The three-dimensional line-of-sight landing point estimation function of the multi-sensor is integrated into the image sensor, the image sensor directly outputs the three-dimensional line-of-sight landing point information of the user, and the precision of calculating the three-dimensional line-of-sight landing point of the user is improved.

[0182] The application further provides a terminal comprising at least one processor and at least one memory, the at least one memory being coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions, when executed by the at least one processor, causing the terminal to perform the line-of-sight landing point estimation method as described above.

[0183] The application further provides a computer storage medium, the computer storage medium storing computer program instructions; the computer program instructions, when executed by a processor, implement the line-of-sight landing point estimation method as described above.

[0184] Those skilled in the art can appreciate that the example methods and steps described in combination with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0185] The above merely describes the preferred embodiments of the present application, and is not intended to limit the present application in any form. Although the present application has been disclosed with the preferred embodiments as above, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical contents to make equivalent embodiments with equivalent changes, without departing from the technical solution of the present application. Any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the technical solution of the present application, still belongs to the scope of the technical solution of the present application.

Claims

1. An eye-tracking method, characterized in that, The method, applied to an image sensor chip, wherein the image sensor chip includes an image acquisition module, an image processing module, an artificial intelligence model inference module, an external interface module, and a central controller module, comprises: The image acquisition module acquires the first image of the user's eyeball; The image processing module is used to process the first image into the input format corresponding to the preset inference algorithm to obtain the second image; The artificial intelligence model reasoning module performs reasoning based on the second image according to a preset algorithm to obtain the reasoning result; The artificial intelligence model inference module includes an on-chip storage unit and a neural network processor. The neural network processor includes an instruction scheduling unit, a matrix calculation unit, a vector calculation unit, and a memory management unit. The instruction scheduling unit sends data transfer instructions and calculation instructions to the corresponding units and manages the execution order of each calculation unit. The matrix calculation unit performs matrix multiplication. The vector calculation unit performs vector addition, subtraction, and dot product calculations. The memory management unit handles data transfer between units. The on-chip storage unit stores the weights, inputs, intermediate calculation results, and outputs of the artificial intelligence model inference module. The external interface module outputs corresponding eye recognition data based on the inference result. The eye recognition data includes at least one of the following: gaze angle, eye parameters, and eye movement state. The central controller module is used to schedule and control other modules to achieve at least one of the following functions: light source control, image acquisition, gaze angle recognition, eye parameter recognition, and eye movement state recognition. The gaze angle recognition includes: extracting key points from the second image; and determining the gaze angle of the second image based on pre-stored user-personalized eye parameters and the key points. The eye parameter recognition includes: controlling the flashing of a preset marker in the light source, or controlling the flashing of a preset marker displayed on the screen in front of the user, to guide the user's eyes to look in the direction of the preset marker; when the user is detected looking at the marker, acquiring a first image of the user's eye through the image acquisition module, and processing it through the image processing module to obtain a second image; extracting feature points in the second image for calculating eye parameters; and calculating eye parameters based on the feature points. The eye movement state recognition includes: extracting eye features from the second image; and determining the user's current eye movement state based on the eye features, the user's eye parameters, and historical eye movement states.

2. The eye-tracking method according to claim 1, characterized in that, The key points include at least one of the pupil, iris, eyeball, and eyelid.

3. The eye-tracking method according to claim 1, characterized in that, The methods for detecting that a user is looking at a marker include: when the angle between the user's line of sight and the line connecting the pupil to the marker is within a preset range.

4. The eye-tracking method according to claim 1, characterized in that, Also includes: The central controller module outputs a light source control command based on the first image. The light source control command is used to adjust at least one of the following: the brightness of the light source, the operating frequency, and the switching on / off of the light source.

5. The eye-tracking method according to claim 4, characterized in that, Adjusting the brightness of the light source includes: adjusting the brightness of the light source by changing the duty cycle of the light source.

6. The eye-tracking method according to claim 4, characterized in that, The method of adjusting the brightness of the light source also includes: If the brightness of the first image is lower than the lower limit of the target brightness range, then at least one of the following is used: increasing the exposure time of the acquired image, increasing the gain, and increasing the brightness of the light source, until the brightness of the re-acquired first image is within the target brightness range. If the brightness of the first image is higher than the upper limit of the target brightness range, then at least one of the following is used: reducing the brightness of the light source, reducing the gain of the acquired image, and reducing the exposure time, until the brightness of the re-acquired first image is within the target brightness range.

7. The eye-tracking method according to claim 4, characterized in that, The method of adjusting the brightness of the light source also includes: If the brightness of the first image is lower than the lower limit of the target brightness range, the exposure time and gain of the acquired image are increased sequentially, and the brightness of the light source is increased until the brightness of the re-acquired first image is within the target brightness range. If the brightness of the first image is higher than the upper limit of the target brightness range, then the brightness of the light source, the gain of the acquired image, and the exposure time are reduced sequentially until the brightness of the re-acquired first image is within the target brightness range.

8. An eye-tracking method, characterized in that, The method, applied to a first image sensor chip and a second image sensor chip, wherein the first image sensor chip and / or the second image sensor chip includes an image acquisition module, an image processing module, an artificial intelligence model inference module, an external interface module, and a central controller module, comprises: The image acquisition module acquires the first image of the user's eyeball; The image processing module is used to process the first image into the input format corresponding to the preset inference algorithm to obtain the second image; The artificial intelligence model reasoning module performs reasoning based on the second image according to a preset algorithm to obtain the reasoning result; The external interface module outputs the corresponding eye recognition data based on the inference result, and receives data sent by other image sensor chips; The central controller module is used to schedule and control other modules to achieve multi-sensor data synchronization and three-dimensional line-of-sight recognition. The three-dimensional gaze point recognition includes: obtaining first gaze information based on the first image sensor chip, the first gaze information including the gaze angle of the first eyeball; obtaining second gaze information of the second eyeball based on the second image sensor chip; obtaining relative position information between the first image sensor chip and the second image sensor chip; and the artificial intelligence model inference module calculating the three-dimensional gaze point based on the first gaze information, the second gaze information and the relative position information.

9. A terminal, characterized in that, The terminal includes at least one processor and at least one memory, the memory being coupled to the processor and storing instructions for execution by the processor, the instructions, when executed by the processor, causing the terminal to perform the eye-tracking method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, they implement the eye-tracking method as described in any one of claims 1 to 8.

Images