Eyeball tracking method and device and electronic equipment

Abstract

The invention relates to the technical field of deep learning, and discloses an eyeball tracking method and device and electronic equipment, and the method comprises the steps: obtaining an eye image data set of a user, inputting the eye image data set into a preset deep learning model, determining the eyeball position information and the eyeball movement mode of the user, based on the eyeball position information and the eyeball movement mode, the intelligent robot is controlled to perform eye interaction with the user, and eye recognition and dynamic position real-time tracking can be accurately performed through a preset deep learning model.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to an eye tracking method, device and electronic device. Background Art

[0002] Eye tracking algorithm is a computer vision technology used to track the position and movement of human eyes in real time. It is usually used in eye tracking, human-computer interaction, user research and other fields.

[0003] Traditional eye tracking algorithms include feature matching, infrared reflection, model-driven, etc. Each eye tracking algorithm has its own advantages and disadvantages. The feature matching or infrared reflection methods have problems such as unstable feature points and being greatly affected by ambient light, and cannot accurately perform eye recognition and real-time dynamic position tracking. Summary of the invention

[0004] Based on this, it is necessary to provide an eye tracking method, device and electronic device to address the above technical problems, which can accurately perform eye recognition and real-time tracking of dynamic positions.

[0005] In a first aspect, an embodiment of the present application provides an eye tracking method, the method comprising:

[0006] Obtain a user's eye image dataset;

[0007] Inputting the eye image data set into a preset deep learning model to determine the user's eye position information and eye movement pattern;

[0008] Based on the eyeball position information and the eyeball movement pattern, the intelligent robot is controlled to perform eye contact interaction with the user.

[0009] In some embodiments, the preset deep learning model includes a convolutional neural network and a recurrent neural network.

[0010] In some embodiments, the convolutional neural network includes several convolutional layers, pooling layers, and fully connected layers.

[0011] In some embodiments, the recurrent neural network comprises a long short-term memory network or a gated recurrent unit.

[0012] In some embodiments, the method further comprises:

[0013] Obtain a user's eye image and a dataset of corresponding eye position annotations;

[0014] Obtaining a dataset of a user's continuous eye position sequence and corresponding eye images;

[0015] Inputting the user's eye image and the corresponding eye position labeled data set into a deep learning model for training; and

[0016] The user's continuous eye position sequence and the corresponding eye image data set are input into a preset deep learning model for training to obtain the preset deep learning model.

[0017] In some embodiments, after acquiring the user's eye image dataset, the method further includes:

[0018] The eye image data set is preprocessed, wherein the preprocessing includes image scaling, grayscale conversion, and contrast enhancement processing.

[0019] In a second aspect, an embodiment of the present application further provides an eye tracking device, the device comprising:

[0020] A first acquisition module is used to acquire a user's eye image dataset;

[0021] A prediction module, used to input the eye image data set into a preset deep learning model to determine the user's eye position information and eye movement pattern;

[0022] A control module is used to control the intelligent machine to interact with the user through eye contact based on the eye position information and eye movement pattern.

[0023] In some embodiments, the apparatus further comprises:

[0024] The second acquisition module is used to acquire the user's eye image and the corresponding eye position annotation data set;

[0025] A third acquisition module is used to acquire a data set of a user's continuous eye position sequence and corresponding eye images;

[0026] A training module, used for inputting the user's eye image and the corresponding eye position annotated data into a deep learning model for training; and

[0027] The user's continuous eye position sequence and the corresponding eye image data set are input into a deep learning model for training to obtain the preset deep learning model.

[0028] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0029] at least one processor; and,

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect above.

[0032] In a fourth aspect, an embodiment of the present application further provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method described in the first aspect above.

[0033] Compared with the prior art, the beneficial effects of the present application are: different from the prior art, the embodiment of the present application provides an eye tracking method, which obtains a user's eye image data set and then inputs the eye image data set into a preset deep learning model to determine the user's eye position information and eye movement pattern, and controls the intelligent robot to interact with the user through eye contact based on the eye position information and eye movement pattern, and can accurately perform eye recognition and dynamic position real-time tracking through the preset deep learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0035] Figure 1 is a flowchart of an eye tracking method provided by an embodiment of the present application;

[0036] Figure 2 is a schematic diagram of the structure of an eye tracking device provided by an embodiment of the present application;

[0037] Figure 3 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a sequence different from the module division in the device or the flow chart. Furthermore, the words "first", "second", "third", etc. used in this application do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0040] like Figure 1 As shown, an embodiment of the present application provides an eye tracking method, the method comprising:

[0041] Step 110: Acquire a user's eye image dataset.

[0042] In an embodiment of the present application, an eye image data set of a user is collected by an eye image acquisition device, and then the collected eye image data set of the user is sent to an electronic device, that is, the electronic device obtains the eye image data set of the user. In some other embodiments, the eye image data set of the user can also be directly obtained from a database. Among them, the eye image data set of the user includes the eye image of the user and a continuous sequence of eyeball positions of the user.

[0043] Step 120: input the eye image data set into a preset deep learning model to determine the user's eye position information and eye movement pattern.

[0044] The preset deep learning model is a model obtained by pre-training. The preset deep learning model in the embodiment of the present application adopts both convolutional neural network and recurrent neural network architectures. Among them, the convolutional neural network includes several convolutional layers, pooling layers and fully connected layers. The recurrent neural network includes a long short-term memory network or a gated recurrent unit, etc.

[0045] Specifically, by obtaining the user's eye image and the corresponding eye position annotated data set; then inputting the user's eye image and the corresponding eye position annotated data set into the deep learning model for training; then obtaining the user's continuous eye position sequence and the corresponding eye image data set; and inputting the user's continuous eye position sequence and the corresponding eye image data set into the preset deep learning model for training, the preset deep learning model is obtained. The model training process requires a large number of data sets. The more data sets there are, the more situations they cover, and thus the higher the recognition ability of the deep learning model.

[0046] It is understandable that in some other embodiments, the acquired data set can be divided into a training sample set and a test sample set, wherein the training sample set is used to train the deep learning model, and the test sample set is used to test the trained deep learning model. Specifically, each image in the training sample set is input into the deep learning model, and the deep learning model is used to automatically train the images in the training sample set to obtain a trained deep learning model.

[0047] Then, each image of the test sample set is input into the trained deep learning model, and each input image is recognized by the model to obtain the corresponding recognition result. The recognition result corresponding to each image is integrated to obtain a recognition result set. The test recognition rate can be determined according to the number of target objects in the recognition result set and the number of target objects in the test sample set. The test recognition rate is used to measure the recognition ability of the trained deep learning model. If the test recognition rate reaches the preset threshold, it means that the recognition ability of the trained deep learning model meets expectations, and the trained deep learning model can be directly used as a trained deep learning model for image recognition. Otherwise, the various parameters of the deep learning model are continuously adjusted, and the deep learning model is trained again until the recognition rate of the model reaches the preset threshold.

[0048] Before inputting the acquired eye image dataset into the preset deep learning model, the eye image dataset needs to be preprocessed, wherein the preprocessing includes but is not limited to image scaling, grayscale, and contrast enhancement processing. After the eye image dataset is preprocessed, the eye image dataset is input into the preset deep learning model to determine the user's eye position information and eye movement pattern.

[0049] Step 130: Control the intelligent robot to perform eye contact interaction with the user based on the eye position information and the eye movement pattern.

[0050] The human-computer interaction process is a process of dynamic communication with users. During the communication process, the user's expression, eyes, and even emotions will change in real time with the scene. Therefore, in the interaction process, real-time tracking and prediction of the user's eye movements and immediate response can greatly improve the user's communication experience. Considering that the user's eye changes are a short-term variable, this task can be completed by using a neural network model with a recurrent architecture, taking the eye position sequence as the input sequence, and transmitting and memorizing information through hidden states. While achieving the task objectives, it can also reduce computational overhead. Specifically, after the user's eye position and eye movement model are determined, the intelligent robot can be controlled to interact with the user through eye contact.

[0051] In an embodiment of the present application, a user's eye image data set is obtained, and then the eye image data set is input into a preset deep learning model to determine the user's eye position information and eye movement pattern, and based on the eye position information and eye movement pattern, an intelligent robot is controlled to perform eye interaction, and the preset deep learning model can accurately perform eye recognition and dynamic position real-time tracking.

[0052] Accordingly, the embodiment of the present application further provides an eye tracking device 200, such as Figure 2 As shown, the device 200 includes:

[0053] A first acquisition module 210 is used to acquire a user's eye image dataset;

[0054] Prediction module 220, used to input the eye image data set into a preset deep learning model to determine the user's eye position information and eye movement pattern;

[0055] The control module 230 is used to control the intelligent robot to perform eye contact interaction with the user based on the eye position information and eye movement pattern.

[0056] The eye tracking device provided in the embodiment of the present application obtains a user's eye image data set through a first acquisition module, and then inputs the eye image data set into a preset deep learning model through a prediction module to determine the user's eye position information and eye movement pattern, and controls the intelligent robot to interact with the user through eye contact based on the eye position information and eye movement pattern through a control module. The preset deep learning model can accurately perform eye recognition and real-time tracking of dynamic position.

[0057] Optionally, in other embodiments of the device, such as Figure 2 As shown, the device 200 also includes:

[0058] The second acquisition module 240 is used to acquire the user's eye image and the corresponding eye position annotation data set;

[0059] The third acquisition module 250 is used to acquire a continuous sequence of eyeball positions of a user and a data set of corresponding eye images;

[0060] A training module 260 is used to input the user's eye image and the corresponding eye position labeled data set into a deep learning model for training; and

[0061] The eye position sequence trained by the user and the corresponding eye image data set are input into the deep learning model for training to obtain the preset deep learning model.

[0062] Optionally, in other embodiments of the device, such as Figure 2 As shown, the device 200 also includes:

[0063] The processing module 270 is used to pre-process the eye image data set, wherein the pre-processing includes image scaling, grayscale conversion and contrast enhancement processing.

[0064] Optionally, in other embodiments of the device, the preset deep learning model includes a convolutional neural network and a recurrent neural network.

[0065] Optionally, in other embodiments of the device, the convolutional neural network includes several convolutional layers, pooling layers and fully connected layers.

[0066] Optionally, in other embodiments of the device, the recurrent neural network includes a long short-term memory network or a gated recurrent unit.

[0067] It should be noted that the above-mentioned eye tracking device can execute the eye tracking method provided in the embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details that are not fully described in the embodiment of the eye tracking device, please refer to the eye tracking method provided in the embodiment of the present application.

[0068] Figure 3 is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application, such as Figure 3 As shown, the electronic device 300 includes:

[0069] One or more processors 310 and memory 320, Figure 3 A processor is taken as an example.

[0070] The processor 310 and the memory 320 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0071] The memory 320 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the eye tracking method in the embodiment of the present application. The processor 310 executes various functional applications and data processing of the electronic device by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, the eye tracking method of the above method embodiment is implemented.

[0072] The memory 320 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the eye tracking device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 320 may optionally include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the eye tracking device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0073] An embodiment of the present application also provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the eye tracking method in any of the above method embodiments.

[0074] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0075] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An eye tracking method, characterized in that: The method comprises: Obtain a user's eye image dataset; Inputting the eye image data set into a preset deep learning model to determine the user's eye position information and eye movement pattern; Based on the eyeball position information and the eyeball movement pattern, the intelligent robot is controlled to perform eye contact interaction with the user.

2. The method according to claim 1, characterized in that The preset deep learning model includes a convolutional neural network and a recurrent neural network.

3. The method according to claim 2, characterized in that The convolutional neural network includes several convolutional layers, pooling layers and fully connected layers.

4. The method according to claim 2, characterized in that: The recurrent neural network includes a long short-term memory network or a gated recurrent unit.

5. The method according to claim 1, characterized in that The method further comprises: Obtain a dataset of the user's eye images and corresponding eye position annotations; Obtaining a dataset of a user's continuous eye position sequence and corresponding eye images; Inputting the user's eye image and the corresponding eye position labeled data set into a deep learning model for training; and The user's continuous eye position sequence and the corresponding eye image data set are input into a preset deep learning model for training to obtain the preset deep learning model.

6. The method according to claim 1, characterized in that After obtaining the user's eye image dataset, the method further includes: The eye image data set is preprocessed, wherein the preprocessing includes image scaling, grayscale conversion, and contrast enhancement processing.

7. An eye tracking device, characterized in that: The device comprises: A first acquisition module is used to acquire a user's eye image dataset; A prediction module, used to input the eye image data set into a preset deep learning model to determine the user's eye position information and eye movement pattern; A control module is used to control the intelligent robot to interact with the user through eye contact based on the eye position information and eye movement pattern.

8. The device according to claim 7, characterized in that The device also includes: The second acquisition module is used to acquire the user's eye image and the corresponding eye position annotation data set; A third acquisition module is used to acquire a data set of a user's continuous eye position sequence and corresponding eye images; A training module, used to input the user's eye image and the corresponding eye position labeled data set into a deep learning model for training; and The user's continuous eye position sequence and the corresponding eye image data set are input into a deep learning model for training to obtain the preset deep learning model.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A non-volatile computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

Images