Eye movement data acquisition method and device, electronic equipment and storage medium

Abstract

The invention discloses an eye movement data acquisition method and device, electronic equipment and a storage medium, and relates to the technical field of eye movement tracking, the method is applied to an eye movement data acquisition system, the eye movement data acquisition system comprises a near-infrared binocular acquisition camera and a display device, and the method comprises the following steps: controlling the display device to display a series of preset test images; in the process of displaying the test image by the display device, controlling the near-infrared binocular acquisition camera to acquire an eye movement image when the target user watches the test image; and determining eye movement data according to each eye movement image and each test image. According to the invention, the accuracy of eye movement data acquisition can be improved.

Description

Technical Field

[0001] The present application relates to the field of eye tracking technology, and in particular to an eye movement data collection method, device, electronic device and storage medium. Background Art

[0002] Eye tracking technology refers to the tracking of eye movements by measuring the position of the eye's gaze point or the movement of the eye relative to the head. The purpose is to monitor the user's eye movement and gaze direction when looking at a specific target.

[0003] As the application demand for eye tracking technology grows, the demand for obtaining high-precision eye movement data also increases. Currently, eye movement data set collection technology mostly collects data by manually clicking discrete points with a mouse, which cannot guarantee the collection of reliable and accurate eye movement data.

[0004] In summary, how to improve the accuracy of eye movement data collection has become a technical problem that needs to be urgently solved in this field. Summary of the invention

[0005] The main purpose of this application is to provide an eye movement data collection method, device, electronic device and storage medium, aiming to improve the accuracy of eye movement data collection.

[0006] To achieve the above purpose, the present application proposes an eye movement data collection method, which is applied to an eye movement data collection system. The eye movement data collection system includes a near-infrared binocular collection camera and a display device. The eye movement data collection method includes:

[0007] Controlling the display device to display a series of preset test images;

[0008] In the process of displaying the test image on the display device, controlling the near-infrared binocular acquisition camera to acquire an eye movement image of a target user when he or she is gazing at the test image;

[0009] Eye movement data is determined according to each of the eye movement images and each of the test images.

[0010] In one embodiment, the test images include first test images and second test images, and the step of controlling the display device to display a series of preset test images includes:

[0011] Controlling the display device to display the first test images in sequence, wherein the first test images respectively include a first fixed point, and the first fixed point is respectively located at a center position and a vertex position of the display device;

[0012] The display device is controlled to display the second test images in sequence, wherein the second test images respectively include second fixed points, the second fixed points are respectively located at different pixel positions of the display device, and the first fixed points do not overlap with the second fixed points.

[0013] In one embodiment, during the process of displaying the test image on the display device, the step of controlling the near-infrared binocular acquisition camera to acquire an eye movement image when the target user gazes at the test image comprises:

[0014] During the process of displaying the first test image on the display device, controlling the near-infrared binocular acquisition camera to acquire a first eye movement image when the target user gazes at the first test image;

[0015] During the process of the display device displaying the second test image, the near-infrared binocular acquisition camera is controlled to acquire a second eye movement image when the target user gazes at the second test image.

[0016] In one embodiment, the step of determining eye movement data according to each of the eye movement images and each of the test images comprises:

[0017] Calculating the initial eye position of the target user according to each of the first eye movement images and each of the first test images;

[0018] Eye movement data is determined according to each of the second eye movement images, each of the second test images and the initial position of the eyeball.

[0019] In one embodiment, the near-infrared binocular acquisition camera includes a first near-infrared binocular acquisition camera and a second near-infrared binocular acquisition camera, and the step of controlling the near-infrared binocular acquisition camera to acquire an eye movement image when a target user gazes at the test image during the process of displaying the test image on the display device includes:

[0020] When the display device displays the test image, controlling the first near-infrared binocular acquisition camera to acquire an eye movement image of the left eye of a target user when the target user is gazing at the test image;

[0021] When the display device displays the test image, the second near-infrared binocular acquisition camera is controlled to acquire the right eye movement image of the target user when the target user gazes at the test image.

[0022] In one embodiment, the eye movement data collection system further includes a calibration camera, and before the step of controlling the display device to display a series of preset test images, the system further includes:

[0023] Control the near-infrared binocular acquisition camera to acquire the first picture of the calibration camera;

[0024] Controlling the calibration camera to capture the second image of the near-infrared binocular acquisition camera and the display device;

[0025] The positional relationship between the near-infrared binocular acquisition camera and the display device is determined according to the first picture and the second picture.

[0026] In one embodiment, the eye movement data acquisition system further includes a face limiting bracket, and the method further includes:

[0027] When it is detected that the target user places the face on the face limiting bracket, the step of controlling the display device to display a preset series of test images is performed.

[0028] In addition, to achieve the above-mentioned purpose, the present application also proposes an eye movement data acquisition device, which is applied to an eye movement data acquisition system. The eye movement data acquisition system includes a near-infrared binocular acquisition camera and a display device. The eye movement data acquisition device includes:

[0029] An image display module, used to control the display device to display a series of preset test images;

[0030] An image acquisition module, used for controlling the near-infrared binocular acquisition camera to acquire an eye movement image of a target user when he or she is gazing at the test image during the process of the display device displaying the test image;

[0031] A data determination module is used to determine eye movement data according to each of the eye movement images and each of the test images.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the eye movement data collection method as described above.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the eye movement data collection method as described above are implemented.

[0034] The present application proposes an eye movement data collection method, which is applied to an eye movement data collection system. The eye movement data collection system includes a near-infrared binocular collection camera and a display device. The display device is controlled to display a series of preset test images in sequence. Each test image is used to guide a target user to gaze at a target area of ​​the display device. At the same time, during the process of the display device displaying the test image, the near-infrared binocular collection camera is controlled to continuously capture images of the target user's eyes to obtain an eye movement image when the target user gazes at the test image. The eye movement image clearly captures the target user's eye movements at different times. Finally, the eye movement data is determined based on each eye movement image and each test image.

[0035] This application realizes continuous and high-precision acquisition of the target user's eye movements by designing an automated collaborative work of a near-infrared binocular acquisition camera and a display device. It can quickly obtain eye images of different eyes with different angles, eye movements, and lighting changes, thereby ensuring the accuracy and reliability of eye movement data acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A flowchart of the eye movement data collection method embodiment 1 of the present application is provided;

[0039] Figure 2 A schematic diagram of the eye movement data collection system structure provided for the eye movement data collection method embodiment 2 of the present application;

[0040] Figure 3 A schematic diagram of a calibration scene provided in Embodiment 2 of the eye movement data collection method of the present application;

[0041] Figure 4 A schematic diagram of a calibration end scene provided in Embodiment 2 of the eye movement data collection method of the present application;

[0042] Figure 5 A schematic diagram of an eye movement data collection scenario provided in Embodiment 2 of the eye movement data collection method of the present application;

[0043] Figure 6 A schematic diagram of a test image provided in Embodiment 2 of the eye movement data collection method of the present application;

[0044] Figure 7 This is a schematic diagram of the module structure of the eye movement data acquisition device according to an embodiment of the present application;

[0045] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the eye movement data collection method in the embodiment of the present application.

[0046] Figure 2 Description of Figure Numbers:

[0047]

[0048]

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: to provide an eye movement data acquisition system, the eye movement data acquisition system includes a near-infrared binocular acquisition camera and a display device; control the display device to display a preset series of test images; in the process of the display device displaying the test images, control the near-infrared binocular acquisition camera to capture the eye movement images of the target user when he looks at the test images; determine the eye movement data according to each eye movement image and each test image.

[0053] As the application demand for eye tracking technology grows, the demand for obtaining high-precision eye movement data also increases. Currently, eye movement data set collection technology mostly collects data by manually clicking discrete points with a mouse, which cannot guarantee the collection of reliable and accurate eye movement data.

[0054] The embodiment of the present application provides a solution. By designing the automated collaborative work of a near-infrared binocular acquisition camera and a display device, continuous and high-precision acquisition of the target user's eye movements is achieved. Eye images of different eyes with different angles, eye movements, and lighting changes can be quickly acquired, thereby ensuring the accuracy and reliability of eye movement data acquisition.

[0055] It should be noted that the execution subject of this embodiment can be an electronic device with data processing, network communication and program running functions, such as a computer, a server, etc., or an electronic device capable of realizing the above functions, etc. The following takes the eye movement data acquisition system as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides an eye movement data collection method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the eye movement data collection method of the present application.

[0057] In this embodiment, the method is applied to an eye movement data acquisition system, the eye movement data acquisition system includes a near-infrared binocular acquisition camera and a display device, and the eye movement data acquisition method includes steps S10 to S30:

[0058] Step S10, controlling the display device to display a series of preset test images;

[0059] It should be noted that the display device in this embodiment can be a display device with high definition and good color reproduction capability, such as a liquid crystal display screen and an LED screen, for displaying the test image.

[0060] The display device is controlled to display a series of preset test images in sequence, and each test image is used to guide the target user to look at the target area of ​​the display device.

[0061] For example, the test image may be a fixed point of a fixed size set in a white background image, and the position of the fixed point is different in different test images to observe the target user's eye movement reaction when looking at the fixed point. The test image may also be other images with simple design and no additional visual interference. In this embodiment, the image content of the test image is not specifically limited.

[0062] Step S20, when the display device displays the test image, controlling the near-infrared binocular acquisition camera to acquire the eye movement image of the target user when he or she is gazing at the test image;

[0063] It should be noted that the near-infrared binocular acquisition camera in this embodiment is a dual-camera device that uses near-infrared spectral technology. It can work stably in different lighting environments, accurately capture subtle movements of the human eye, realize eye tracking in three-dimensional space, improve the accuracy of eye movement image acquisition, and at the same time broaden the application scenarios of eye movement data acquisition.

[0064] When the display device displays the test image, the near-infrared binocular camera is controlled to continuously capture images of the target user's eyes to obtain eye movement images when the target user looks at the test image. The eye movement images clearly capture the target user's eye movements at different times.

[0065] Step S30, determining eye movement data according to each eye movement image and each test image.

[0066] Eye movement data is determined based on each eye movement image and each test image.

[0067] In a feasible implementation, the step of determining eye movement data includes: for each eye movement image and the corresponding test image, determining the three-dimensional spatial coordinates of the human eye in the eye movement image; mapping the three-dimensional spatial coordinates to the display device to determine the position of the gaze point of the target user on the display device; obtaining the fixed point position in the test image, and calculating the eye movement data of the target user based on the gaze point position and the fixed point position, wherein the eye movement data includes the movement trajectory and dwell time of the gaze point, etc.

[0068] During the eye movement data collection process, the target users may include a large number of users with different eye conditions, such as whether they are wearing makeup, glasses, whether the glasses are reflective, etc. The data can also be collected under different lighting environments to ensure the diversity of the collected eye movement data. Finally, the collected eye movement data is integrated into an eye movement dataset for training eye movement tracking models in different fields such as psychology, human-computer interaction, or advertising design.

[0069] In summary, in the embodiments of the present application, by designing the automated collaborative work of the near-infrared binocular acquisition camera and the display device, continuous and high-precision acquisition of the target user's eye movements is achieved, and eye images of different eyes with different angles, eye movements, and lighting changes can be quickly acquired, thereby ensuring the accuracy and reliability of eye movement data acquisition.

[0070] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, the test image includes each first test image and each second test image, and step S10 may include steps S101 to S102:

[0071] Step S101, controlling the display device to display first test images in sequence, wherein each first test image includes a first fixed point, and each first fixed point is located at a center position and a vertex position of the display device respectively;

[0072] The display device is controlled to display each first test image one by one in a certain order and frequency, wherein each first test image includes a clearly identifiable first fixed point, and the first fixed point is located at the center position and four vertex positions of the display device in each first test image, or more specific vertex positions are selected according to actual needs, so as to preliminarily determine the initial positions of the eyeballs of different target users in different gaze directions by collecting eye images of the target users when they are looking at the first test images, so as to provide a calibration benchmark for subsequent eye movement data collection.

[0073] Step S102, controlling the display device to display each second test image in sequence, wherein each second test image includes a second fixed point, each second fixed point is located at a different pixel point position of the display device, and each first fixed point does not overlap with each second fixed point.

[0074] The display device is controlled to display each second test image one by one in a certain order and frequency, wherein each second test image contains a clearly visible second fixed point, and the second fixed point is located at a different pixel point position in the display device in each second test image, and the position of each second fixed point does not overlap with that of each first fixed point, so as to collect the eye movement data of the target user at different gaze points through diversified visual guidance, so as to deeply analyze the visual behavior pattern of the target user.

[0075] Exemplarily, the eye movement data collection process may include a calibration phase of displaying a first test image and an acquisition phase of displaying a second test image. In the calibration phase, the eyes of participating target users are calibrated using a first test image including a center point and a vertex position. By having the target users gaze at the fixed points in the first test image in turn, the initial positions of the target users' eyes in different gaze directions are collected, thereby establishing a targeted model of the target users' eye movements. In the acquisition phase, a second test image in which a large number of fixed points are distributed at different pixel points in a display device is used. For example, 600 fixed points are set to be displayed one by one on the display device in turn, and the display position of each fixed point on the display device is different. The target users' gaze points are guided to move with the positions of the vertices, thereby collecting the eye images of the target users in the acquisition phase and determining the eye movement data.

[0076] In a feasible embodiment, step S20 may include steps S201 to S202:

[0077] Step S201, when the display device displays the first test image, controlling the near-infrared binocular acquisition camera to acquire a first eye movement image when the target user gazes at the first test image;

[0078] When the display device displays the first test image, the near-infrared binocular acquisition camera is synchronously activated to start collecting the first eye movement image of the target user when he or she is looking at the first test image during this process. The gaze point of the target user in the first eye movement image moves following the position change of the first fixed point in the first test image. The first fixed point is the fixed point of the visual stimulus in the first test image. The first fixed point is set at the center and surrounding vertices of the display device, etc., in order to guide the sight of the target user to make a specific movement. By continuously collecting the first eye movement image of the target user, the initial position of the target user's eyeball is determined, thereby helping the terminal to preliminarily establish a personalized model of the target user's eyeball movement.

[0079] Step S202 , when the display device displays the second test image, the near-infrared binocular acquisition camera is controlled to acquire a second eye movement image when the target user gazes at the second test image.

[0080] After displaying the first test image, the display device continues to display the second test image. At this time, the near-infrared binocular acquisition camera is controlled to continue to acquire the second eye movement image when the target user gazes at the second test image, wherein the second fixed point in the second test image is distributed at different pixel positions on the display device, and in the second eye movement image, the target user's eye gaze point moves with the change of the position of the second vertex.

[0081] In this way, the eye movement data acquisition terminal first calibrates the initial position of the target user's eyeball through the first test image, and then acquires the eye movement image through the second test image, so as to comprehensively and accurately record the eye movement behaviors of different target users under different test images.

[0082] In a feasible embodiment, step S30 may include steps S301 to S302:

[0083] Step S301, calculating the initial eye position of the target user according to each first eye movement image and each first test image;

[0084] After collecting the first eye movement images of the target user when facing the first test image, the initial position of the target user is calculated according to each first eye movement image and each first test image.

[0085] Exemplarily, each first eye movement image corresponds to each first test image one by one, and the first test image includes image A with a fixed point located at a center position, image B with a fixed point located at a vertex position of an upper left corner, image C with a fixed point located at a vertex position of a lower left corner, image D with a fixed point located at a vertex position of an upper right corner, and image E with a fixed point located at a vertex position of a lower right corner. First, an eye position (x1, y1) in the first eye movement image collected when the target user views image A is determined, an eye position (x2, y2) in the first eye movement image collected when the target user views image B is determined, an eye position (x3, y3) in the first eye movement image collected when the target user views image C is determined, and an eye position (x4, y5) in the first eye movement image collected when the target user views image D is determined. The eye position (x4, y4) is used to determine the eye position (x5, y5) in the first eye movement image collected when the target user watches image E. Then, based on the image processing algorithm and coordinate conversion technology, each eye position is mapped to the corresponding fixed point position. After the mapping relationship between the eye position and the fixed point position is established, the initial eye position of the target user when collecting eye movement data is calculated using these mapping relationships. The initial eye position represents the natural state or reference state of the target user's eye without any specific visual stimulation. Due to individual differences and the complexity of eye movement, the initial eye position will vary from person to person. By pre-calibrating the initial eye position of the target user, more accurate data can be obtained when the eye movement data is determined later.

[0086] Step S302 , determining eye movement data according to each second eye movement image, each second test image and the initial position of the eyeball.

[0087] The eye movement data of the target user is determined based on the second eye movement images, the second test images and the determined initial position of the eyeball when the target user faces the second test image. The eye movement data reflects the fine movement trajectory of the eyeball of the target user when observing different second test images, including data such as the movement speed, direction, dwell time, and pupil diameter change of the gaze point.

[0088] In a feasible embodiment, the near-infrared binocular acquisition camera includes a first near-infrared binocular acquisition camera and a second near-infrared binocular acquisition camera, and step S20 may include steps S203 to S204:

[0089] Step S203, when the display device displays the test image, controlling the first near-infrared binocular acquisition camera to acquire the left eye movement image of the target user when he or she is gazing at the test image;

[0090] During the process of displaying the test image on the display device, the first near-infrared binocular acquisition camera is controlled to perform continuous image acquisition on the left eye of the target user to obtain the left eye movement image of the target user when looking at the test image. It should be noted that the first near-infrared binocular acquisition camera includes two acquisition lenses, that is, it can simultaneously acquire two left eye movement images of the target user at different angles, avoiding the situation where image acquisition is unavailable due to environmental factors such as angle or reflection. The left eye movement image clearly captures the left eye movement of the target user at different times.

[0091] Step S204 , when the display device displays the test image, the second near-infrared binocular acquisition camera is controlled to acquire the right eye movement image of the target user when he or she is gazing at the test image.

[0092] During the process of displaying the test image on the display device, the second near-infrared binocular acquisition camera is controlled to continuously acquire images of the right eye of the target user to obtain the right eye movement image of the target user when he or she looks at the test image. It should be noted that the second near-infrared binocular acquisition camera also includes two acquisition lenses, and the right eye movement image clearly captures the right eye movement of the target user at different times.

[0093] In this way, the eye movement data collection terminal can simultaneously and independently obtain two eye movement images of the left eye and the right eye of the target user, thereby improving the accuracy and comprehensiveness of eye movement data collection.

[0094] In a feasible embodiment, the eye movement data acquisition system further includes a calibration camera, and steps S40 to S60 may be included before step S10:

[0095] Step S40, controlling the near-infrared binocular acquisition camera to acquire the first picture of the calibration camera;

[0096] It should be noted that the calibration camera is a professional image acquisition device. Before officially starting the eye movement data acquisition, the calibration camera is used to collect images of other devices to determine the relative position and spatial relationship between different devices based on the images, thereby ensuring the accuracy and stability of the entire eye movement data acquisition system.

[0097] Before officially starting the eye movement data collection, the near-infrared binocular acquisition camera is controlled to capture the first picture of the calibration camera to obtain the image information of the calibration camera at the current viewing angle.

[0098] Step S50, controlling the calibration camera to capture the second image of the near-infrared binocular camera and the display device;

[0099] The calibration camera is controlled to collect a second picture of the near-infrared binocular collection camera and the display device, that is, the second picture includes the near-infrared binocular collection camera and the display device.

[0100] Step S60: determining the positional relationship between the near-infrared binocular acquisition camera and the display device according to the first picture and the second picture.

[0101] Based on the acquired first and second images, the positional relationship between the near-infrared binocular acquisition camera and the display device is determined through image processing and algorithm analysis. In the subsequent eye movement data acquisition process, the positional relationship between the near-infrared binocular acquisition camera and the display device can be used as a basis for coordinate mapping between the eye movement image and the test image, thereby ensuring the accuracy and reliability of the data.

[0102] In a feasible embodiment, the eye movement data collection system further includes a face limiting bracket, and the eye movement data collection method may further include step S70:

[0103] Step S70 is to control the display device to display a series of preset test images when it is detected that the target user places the face on the face limiting bracket.

[0104] When the eye movement data acquisition system successfully identifies and confirms that the target user has correctly and stably placed the face on the face limit bracket through its built-in sensor or detection mechanism, it automatically triggers and executes the step of controlling the display device to display a series of preset test images in sequence and the subsequent steps, that is, when it is detected that the target user has placed the face on the face limit bracket, the display device is controlled to display a series of preset test images, and when the display device displays the test images, the near-infrared binocular acquisition camera is controlled to collect eye movement images of the target user when he looks at the test images; and the eye movement data is determined according to each eye movement image and each test image.

[0105] It should be noted that the face limit bracket is used to fix the head position of the target user to reduce the eye movement data deviation caused by head movement or rotation. When it is detected that the target user places his face on the face limit bracket, the eye movement data collection will automatically start. When it is detected that the face moves away from the face limit bracket, the collection process will be automatically paused to ensure that the collected eye movement data is more accurate.

[0106] For example, to help understand the implementation process of the eye movement data collection method, please refer to Figure 2 , Figure 2 An eye movement data collection system is provided to explain the eye movement data collection process, specifically:

[0107] The eye movement data acquisition system includes: a calibration camera 1, a face limit bracket 2, a near-infrared binocular acquisition camera 3, a platform 4, a display device 5 and a processing device. The face limit bracket 2 and the near-infrared binocular acquisition camera 3 are fixedly installed on the platform 4, the calibration camera 1 is detachably installed on the platform 4, the display device 5 is placed at a certain distance from the face limit bracket 2, and the processing device can be an electronic device with data processing, network communication and program running functions, such as a computer, server, etc., or an electronic device that can realize the above functions, etc.

[0108] The calibration camera 1 is used to capture images of the near-infrared binocular acquisition camera 3 and the display device 5 to determine the positional relationship between the infrared binocular acquisition camera 3 and the display device 5. After the positional relationship is determined, the calibration camera 1 can be removed to start collecting eye movement data.

[0109] The processing device is used to generate a collection request and send it to the display device 5 when it is detected that the target user places the face on the face limiting bracket 2.

[0110] The display device 5 is used to display a series of preset test images in response to the acquisition request of the processing device, and the test images are used to guide the target user to look at the target area of ​​the display device 5;

[0111] A near-infrared binocular acquisition camera 3 is used to collect an eye movement image of a target user when the target user is gazing at the test image during the process of displaying the test image on the display device 5, and to send the eye movement image to a processing device;

[0112] A processing device is used to determine, for each test image displayed on the display device 5, from each received eye movement image, the eye movement image corresponding to each test image; determine the gaze point of the target user on the display device 5 at each moment according to the eye movement image corresponding to each test image, and determine the eye movement data of the target user corresponding to each test image according to the determined gaze point at each moment; after obtaining the eye movement data, the eye movement features of the target user corresponding to each test image can also be determined according to the eye movement data corresponding to each test image; and train the eye movement tracking model to be trained based on the eye movement features of each test image.

[0113] In the actual eye movement data collection process, the calibration scene is as follows Figure 3 As shown, first, the calibration camera 1 shoots the screen of the near-infrared binocular acquisition camera 3 and the display device 5, and the near-infrared binocular acquisition camera 3 shoots the calibration camera 1, and the position relationship among the calibration camera 1, the near-infrared binocular acquisition camera 3 and the display device 5 is determined, and the position calibration between the devices is completed; then, as shown in Figure 4 As shown in FIG. 1 , after the calibration is completed, the calibration camera 1 is removed and the tester (i.e., the target user) is waited for to start collecting eye movement data; then, as shown in FIG. Figure 5As shown, the tester places the face at the position of the face limit bracket 2, and the tester's eyes move following the changes of the test image displayed on the screen of the display device 5. At the same time, the control device controls the binocular acquisition camera 3 to capture the eye image of the tester to complete the eye movement image acquisition work.

[0114] It should be noted that the test image is Figure 6 As shown, a fixed point of a fixed size is set in a white background image. The number of test images is preferably 600. The position of the fixed point in each test image is different. By increasing the position of the fixed point to subdivide the eye rotation angle, more precise eye movement data can be obtained.

[0115] It should also be noted that in this embodiment, two near-infrared binocular acquisition cameras are provided, one of which is used to collect the eye movement image of the left eye of the test person, and the other is used to collect the eye movement image of the right eye of the test person. One near-infrared binocular acquisition camera includes two acquisition lenses, that is, it can simultaneously collect four eye movement images of the test person at different angles, avoiding the situation where image acquisition is unavailable due to environmental factors such as angle or reflection.

[0116] In this way, by designing the automated collaborative work of the near-infrared binocular acquisition camera and the display device, continuous and high-precision acquisition of the target user's eye movements is achieved, and eye images of different eyes with different angles, eye movements, and lighting changes can be quickly acquired, thereby ensuring the accuracy and reliability of eye movement data acquisition.

[0117] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the eye movement data collection method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0118] The present application also provides an eye movement data acquisition device, which is applied to an eye movement data acquisition system. The eye movement data acquisition system includes a near-infrared binocular acquisition camera and a display device. Please refer to Figure 7 , the eye movement data acquisition device includes:

[0119] An image display module 10, used to control the display device to display a series of preset test images;

[0120] The image acquisition module 20 is used to control the near-infrared binocular acquisition camera to acquire the eye movement image of the target user when he or she is gazing at the test image during the process of displaying the test image on the display device;

[0121] The data determination module 30 is used to determine the eye movement data according to each eye movement image and each test image.

[0122] Optionally, the test image includes each first test image and each second test image, and the image display module 10 is further used to:

[0123] Controlling the display device to display first test images in sequence, wherein each first test image includes a first fixed point, and each first fixed point is located at a center position and a vertex position of the display device respectively;

[0124] The display device is controlled to display each second test image in sequence, wherein each second test image includes a second fixed point, each second fixed point is located at a different pixel point position of the display device, and each first fixed point does not overlap with each second fixed point.

[0125] Optionally, the image acquisition module 20 is further used for:

[0126] During the process of displaying the first test image on the display device, controlling the near-infrared binocular acquisition camera to acquire a first eye movement image when the target user gazes at the first test image;

[0127] When the display device displays the second test image, the near-infrared binocular acquisition camera is controlled to acquire a second eye movement image when the target user gazes at the second test image.

[0128] Optionally, the data determination module 30 is further configured to:

[0129] Calculating the initial eye position of the target user according to each first eye movement image and each first test image;

[0130] Eye movement data is determined according to each second eye movement image, each second test image and an initial position of the eyeball.

[0131] Optionally, the near-infrared binocular acquisition camera includes a first near-infrared binocular acquisition camera and a second near-infrared binocular acquisition camera, and the image acquisition module 20 is further used for:

[0132] When the display device displays the test image, controlling the first near-infrared binocular acquisition camera to acquire the left eye movement image of the target user when the target user gazes at the test image;

[0133] When the display device displays the test image, the second near-infrared binocular acquisition camera is controlled to acquire the right eye movement image of the target user when the target user gazes at the test image.

[0134] Optionally, the eye movement data acquisition system further includes a calibration camera, and the eye movement data acquisition device further includes a calibration module (not shown), the calibration module being used to:

[0135] Control the near-infrared binocular acquisition camera to acquire the first picture of the calibration camera;

[0136] Control the calibration camera to collect the second picture of the near-infrared binocular collection camera and the display device;

[0137] The positional relationship between the near-infrared binocular acquisition camera and the display device is determined according to the first picture and the second picture.

[0138] Optionally, the eye movement data acquisition system further includes a face limiting bracket, an image display module 10, and is further used for:

[0139] When it is detected that the target user places the face on the face limiting bracket, the step of controlling the display device to display a series of preset test images is executed.

[0140] The eye movement data collection device provided in the embodiment of the present application adopts the eye movement data collection method in the above embodiment, which can improve the accuracy of eye movement data collection. Compared with the prior art, the beneficial effects of the eye movement data collection device provided in the embodiment of the present application are the same as the beneficial effects of the eye movement data collection method provided in the above embodiment, and other technical features in the eye movement data collection device are the same as the features disclosed in the eye movement data collection method in the above embodiment, which will not be repeated here.

[0141] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the eye movement data collection method in the above-mentioned embodiment 1.

[0142] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0143] like Figure 8As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0144] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0145] The electronic device provided by the present application adopts the eye movement data collection method in the above embodiment to improve the accuracy of eye movement data collection. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the eye movement data collection method provided by the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0146] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0147] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0148] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the eye movement data collection method in the above-mentioned embodiment.

[0149] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0150] The computer-readable storage medium may be included in the head-mounted display device; or may exist independently without being assembled into the head-mounted display device.

[0151] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device: controls a display device to display a preset series of test images; during the display device displaying the test images, controls a near-infrared binocular acquisition camera to capture eye movement images of a target user when the user is gazing at the test images; and determines eye movement data based on each eye movement image and each test image.

[0152] The computer program code for performing the operation of the embodiment of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0153] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0154] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0155] The readable storage medium provided in the embodiment of the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned eye movement data collection method, and can improve the accuracy of eye movement data collection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiment of the present application are the same as the beneficial effects of the eye movement data collection method provided in the above-mentioned embodiment, and will not be repeated here.

[0156] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for collecting eye movement data, characterized in that: Applied to an eye movement data acquisition system, the eye movement data acquisition system includes a near-infrared binocular acquisition camera and a display device; the method includes: Controlling the display device to display a series of preset test images; In the process of displaying the test image on the display device, controlling the near-infrared binocular acquisition camera to acquire an eye movement image of a target user when he or she is gazing at the test image; Eye movement data is determined according to each of the eye movement images and each of the test images.

2. The eye movement data collection method according to claim 1, characterized in that: The test images include first test images and second test images, and the step of controlling the display device to display a series of preset test images includes: Controlling the display device to display the first test images in sequence, wherein the first test images respectively include a first fixed point, and the first fixed point is respectively located at a center position and a vertex position of the display device; The display device is controlled to display the second test images in sequence, wherein the second test images respectively include second fixed points, the second fixed points are respectively located at different pixel positions of the display device, and the first fixed points do not overlap with the second fixed points.

3. The eye movement data collection method according to claim 2, characterized in that: The step of controlling the near-infrared binocular acquisition camera to acquire an eye movement image of a target user when gazing at the test image during the process of displaying the test image on the display device comprises: During the process of displaying the first test image on the display device, controlling the near-infrared binocular acquisition camera to acquire a first eye movement image when the target user gazes at the first test image; During the process of the display device displaying the second test image, the near-infrared binocular acquisition camera is controlled to acquire a second eye movement image when the target user gazes at the second test image.

4. The eye movement data collection method according to claim 3, characterized in that: The step of determining eye movement data according to each of the eye movement images and each of the test images comprises: Calculating the initial eye position of the target user according to each of the first eye movement images and each of the first test images; Eye movement data is determined according to each of the second eye movement images, each of the second test images and the initial position of the eyeball.

5. The eye movement data collection method according to claim 1, characterized in that: The near-infrared binocular acquisition camera includes a first near-infrared binocular acquisition camera and a second near-infrared binocular acquisition camera. The step of controlling the near-infrared binocular acquisition camera to acquire an eye movement image of a target user when gazing at the test image during the process of displaying the test image on the display device includes: When the display device displays the test image, controlling the first near-infrared binocular acquisition camera to acquire an eye movement image of the left eye of a target user when the target user is gazing at the test image; When the display device displays the test image, the second near-infrared binocular acquisition camera is controlled to acquire the right eye movement image of the target user when the target user gazes at the test image.

6. The eye movement data collection method according to claim 1, characterized in that: The eye movement data acquisition system further includes a calibration camera, and before the step of controlling the display device to display a series of preset test images, it also includes: Control the near-infrared binocular acquisition camera to acquire the first picture of the calibration camera; Controlling the calibration camera to capture the second image of the near-infrared binocular acquisition camera and the display device; The positional relationship between the near-infrared binocular acquisition camera and the display device is determined according to the first picture and the second picture.

7. The eye movement data collection method according to claim 1, characterized in that: The eye movement data acquisition system further includes a face limiting bracket, and the method further includes: When it is detected that the target user places the face on the face limiting bracket, the step of controlling the display device to display a preset series of test images is performed.

8. An eye movement data collection device, characterized in that: Applied to an eye movement data acquisition system, the eye movement data acquisition system includes a near-infrared binocular acquisition camera and a display device; the device includes: An image display module, used to control the display device to display a series of preset test images; An image acquisition module, used for controlling the near-infrared binocular acquisition camera to acquire an eye movement image of a target user when he or she is gazing at the test image during the process of the display device displaying the test image; A data determination module is used to determine eye movement data according to each of the eye movement images and each of the test images.

9. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the eye movement data collection method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the eye movement data collection method according to any one of claims 1 to 7 are implemented.

Images