Eye movement test track generation method, eye movement control method, device and system

Abstract

The invention discloses an eye movement test track generation method, device and system, an eye movement control method, device and system and a computer readable storage medium. The eye movement test track generation method provided by the embodiment of the invention comprises the following steps: collecting real eye movement data; training an adversarial neural network model according to the real eye movement data, and generating virtual eye movement data based on the adversarial neural network model according to the virtual eye movement starting point and the virtual eye movement ending point; and determining dynamic available data of the human eye simulation equipment according to the virtual eye movement data. Thus, only a small amount of real eye movement data needs to be collected for training of the adversarial neural network model, and a large amount of virtual eye movement data can be generated based on the trained adversarial neural network model so as to perform motion control on the human eye simulation equipment, thereby effectively improving the test efficiency of the optical instrument.

Description

Technical Field

[0001] The present application relates to the field of eye movement technology, and in particular to an eye movement test trajectory generation method, an eye movement control method, a device, a system and a computer-readable storage medium. Background Art

[0002] With the rapid development of optical instruments, intelligent robots and other fields, the demand for human eye simulation equipment is increasing. Such equipment can be used for quality inspection of optical instruments such as eye trackers and ophthalmic optical biometers, and can also be used as components of intelligent robots. At present, when performing quality inspection on the above optical instruments, it is necessary to collect a large number of real eye movement trajectories to control the motion of human eye simulation equipment, and the test efficiency is low. Summary of the invention

[0003] The embodiments of the present application provide an eye movement test trajectory generation method, an eye movement control method, an apparatus, a system and a computer-readable storage medium to solve at least one of the above-mentioned technical problems.

[0004] The eye movement test trajectory generation method of the embodiment of the present application includes:

[0005] Collect real eye movement data;

[0006] Training an adversarial neural network model according to the real eye movement data;

[0007] Generate virtual eye movement data based on the adversarial neural network model according to the virtual eye movement starting point and the virtual eye movement end point;

[0008] Dynamically available data of a human eye simulation device is determined according to the virtual eye movement data.

[0009] In some embodiments, the adversarial neural network model includes an eye movement data generator and an eye movement data discriminator, and training the adversarial neural network model according to the real eye movement data includes:

[0010] Generate virtual data samples based on noise data by the eye movement data generator, wherein the noise data includes an eye movement starting point sample and an eye movement end point sample;

[0011] Distinguishing the real eye movement data from the virtual data sample by the eye movement data discriminator;

[0012] The weights of the eye movement data generator and the eye movement data discriminator are adjusted according to the distinction result until the adversarial neural network model reaches a Nash equilibrium.

[0013] In some embodiments, adjusting the weights of the eye movement data generator and the eye movement data discriminator according to the distinction result until the adversarial neural network model reaches a Nash equilibrium includes:

[0014] Determine whether the adversarial neural network model reaches Nash equilibrium according to the distinction result;

[0015] When the adversarial neural network model does not reach Nash equilibrium, determining whether the distinction result is correct;

[0016] When the distinction result is incorrect, adjusting the weight of the eye movement data discriminator, and returning to the step of distinguishing the real eye movement data from the virtual data sample by the eye movement data discriminator;

[0017] When the distinction result is correct, adjusting the weight of the eye movement data generator, and returning to the step of generating virtual data samples based on noise data by the eye movement data generator;

[0018] When the adversarial neural network model reaches a Nash equilibrium, it is determined that the training of the adversarial neural network model is completed.

[0019] In certain embodiments, the incorrect distinction result includes:

[0020] The eye movement data discriminator judges the real eye movement data as virtual data;

[0021] The eye movement data discriminator judges the virtual data sample as real data;

[0022] The correct distinction results include:

[0023] The eye movement data discriminator determines the virtual data sample as virtual data.

[0024] In some embodiments, the virtual eye movement data includes gaze point coordinates and a timestamp corresponding to the gaze point coordinates, and determining the dynamic available data of the human eye simulation device according to the virtual eye movement data includes:

[0025] Based on the particle swarm optimization algorithm and the Hooke-Jeff algorithm, the virtual eye movement data is subjected to fitting optimization processing so that the curve motion corresponding to the gaze point coordinates and the timestamp satisfies the process of acceleration-uniform speed-deceleration.

[0026] In some embodiments, the virtual eye movement data is subjected to fitting optimization processing based on a particle swarm optimization algorithm and a Hooke-Jeff algorithm so that the curved motion corresponding to the gaze point coordinates and the timestamp satisfies the process of acceleration-uniform speed-deceleration, including:

[0027] Initializing particle swarm information, wherein the particle swarm information includes positions and velocities of particles;

[0028] Calculating a population fitness function value according to the particle swarm information and the virtual eye movement data;

[0029] updating the position and velocity of the particle according to the population fitness function value, and recalculating the population fitness function value according to the updated position and velocity of the particle and the virtual eye movement data;

[0030] When the population fitness function value decreases by more than a predetermined proportion, the updated position and speed of the particle are optimized based on the Hooke-Jeff algorithm.

[0031] In some embodiments, the fitting optimization processing of the virtual eye movement data based on the particle swarm optimization algorithm and the Hooke-Jeff algorithm so that the curved motion corresponding to the gaze point coordinates and the timestamp satisfies the acceleration-uniform speed-deceleration process also includes:

[0032] When the population fitness function value decreases by no more than the predetermined ratio, determining whether the population fitness function value decreases in the most recent predetermined number of iterations;

[0033] When the population fitness function value decreases in the most recent predetermined number of iterations, returning to the step of calculating the population fitness function value according to the particle population information and the virtual eye movement data;

[0034] When the population fitness function value does not decrease in the most recent predetermined number of iterations, the updated position and speed of the particle are optimized based on the Hooke-Jeff algorithm.

[0035] The eye movement test trajectory generation device of the embodiment of the present application includes:

[0036] A collection module, used to collect real eye movement data;

[0037] A training module, used for training an adversarial neural network model according to the real eye movement data;

[0038] A generating module, used for generating virtual eye movement data based on the adversarial neural network model according to the virtual eye movement starting point and the virtual eye movement end point;

[0039] A determination module is used to determine the dynamic available data of the human eye simulation device according to the virtual eye movement data.

[0040] The eye movement test trajectory generation system of the embodiment of the present application includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the eye movement test trajectory generation method of any of the above embodiments is implemented.

[0041] The computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the program is executed by a processor, the eye movement test trajectory generation method of any of the above-mentioned embodiments is implemented.

[0042] The eye movement control method of the embodiment of the present application is used to control a human eye simulation device, and the eye movement control method includes:

[0043] Acquire dynamic available data of the human eye simulation device, wherein the dynamic available data is derived from: collecting real eye movement data; training an adversarial neural network model according to the real eye movement data; generating virtual eye movement data based on the adversarial neural network model according to a virtual eye movement starting point and a virtual eye movement end point; and determining the dynamic available data according to the virtual eye movement data;

[0044] The motion control data of the human eye simulation device is calculated according to the dynamically available data.

[0045] In some embodiments, the motion control data includes one or more of a rotation angle, a running speed, a running acceleration, and a running deceleration.

[0046] The eye movement control device of the embodiment of the present application is used to control a human eye simulation device, and the eye movement control device includes:

[0047] An acquisition module is used to acquire dynamic available data of the human eye simulation device, wherein the dynamic available data is derived from: collecting real eye movement data; training an adversarial neural network model according to the real eye movement data; generating virtual eye movement data based on the adversarial neural network model according to a virtual eye movement starting point and a virtual eye movement end point; and determining the dynamic available data according to the virtual eye movement data;

[0048] A calculation module is used to calculate the motion control data of the human eye simulation device according to the dynamically available data.

[0049] The eye movement control system of the embodiment of the present application includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the eye movement control method of any of the above embodiments is implemented.

[0050] The computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the program is executed by a processor, the eye movement control method of any of the above-mentioned embodiments is implemented.

[0051] In the eye movement test trajectory generation method, eye movement control method, device, system and computer-readable storage medium of the embodiment of the present application, real eye movement data is collected to train the adversarial neural network model, and then virtual eye movement data is generated based on the trained adversarial neural network model, and the dynamic available data of the human eye simulation device is determined according to the virtual eye movement data to perform motion control on the human eye simulation device. In this way, only a small amount of real eye movement data needs to be collected for training the adversarial neural network model, and a large amount of virtual eye movement data can be generated based on the trained adversarial neural network model to perform motion control on the human eye simulation device, which effectively improves the test efficiency of optical instruments.

[0052] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0054] Figure 1 It is a flowchart of a method for generating an eye movement test trajectory in certain embodiments of the present application;

[0055] Figure 2 It is a flowchart of a method for generating an eye movement test trajectory in certain embodiments of the present application;

[0056] Figure 3 It is a flowchart of a method for generating an eye movement test trajectory in certain embodiments of the present application;

[0057] Figure 4 It is a flowchart of training an adversarial neural network model in certain embodiments of the present application;

[0058] Figure 5 It is a flowchart of a method for generating an eye movement test trajectory in certain embodiments of the present application;

[0059] Figure 6 It is a flowchart of a method for generating an eye movement test trajectory in certain embodiments of the present application;

[0060] Figure 7 It is a schematic diagram of a process of performing fitting optimization processing on virtual eye movement data based on a particle swarm optimization algorithm and a Hooke-Jeff algorithm in certain embodiments of the present application;

[0061] Figure 8 It is a flowchart of a method for generating an eye movement test trajectory in certain embodiments of the present application;

[0062] Fig. 9It is a flowchart of a method for generating an eye movement test trajectory in certain embodiments of the present application;

[0063] Fig.10 is a flowchart of an eye movement control method according to certain embodiments of the present application;

[0064] Fig.11 is a schematic diagram of the structure of an eye tracking device according to some embodiments of the present application;

[0065] Fig.12 is a top view of an eye tracking device according to certain embodiments of the present application;

[0066] Fig.13 is a partial schematic diagram of an eye tracking device according to certain embodiments of the present application;

[0067] Fig.14 is a schematic diagram of the structure of an eye tracking device according to some embodiments of the present application;

[0068] Fig.15 is a schematic diagram of a simulated eye gaze target point of an eye tracking device according to certain embodiments of the present application;

[0069] Fig.16 is a schematic diagram of a simulated eye gaze target point of an eye tracking device according to certain embodiments of the present application;

[0070] Fig.17 It is a flowchart of a method for generating an eye tracking eye movement test trajectory according to certain embodiments of the present application;

[0071] Fig.18 is a schematic diagram of simulating eyeball rotation to generate deviation in certain embodiments of the present application;

[0072] Fig.19 is a schematic diagram of simulating eyeball rotation to generate deviation in certain embodiments of the present application;

[0073] Fig. 20 It is a module schematic diagram of an eye movement test trajectory generating device in certain embodiments of the present application;

[0074] Fig.21 is a module schematic diagram of an eye movement test trajectory generation system of certain embodiments of the present application;

[0075] Fig. 22 is a schematic diagram of the connection state between a computer-readable storage medium and a processor in certain embodiments of the present application;

[0076] Fig.23 is a schematic diagram of a module of an eye movement control device according to some embodiments of the present application;

[0077] Fig.24is a schematic diagram of a module of an eye movement control system according to certain embodiments of the present application;

[0078] Fig.25 It is a schematic diagram of the connection status between a computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0079] The following is a further description of the embodiments of the present application in conjunction with the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements or elements with the same or similar functions from beginning to end. In addition, the embodiments of the present application described below in conjunction with the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the present application.

[0080] See also Figure 1 and Figure 2 , the embodiment of the present application provides a method for generating an eye movement test trajectory. The method for generating an eye movement test trajectory comprises:

[0081] 010: Collect real eye movement data;

[0082] 020: Training an adversarial neural network model based on real eye movement data;

[0083] 030: Generate virtual eye movement data based on the adversarial neural network model according to the virtual eye movement starting point and the virtual eye movement end point;

[0084] 040: Determine the dynamic available data of the human eye simulation device based on the virtual eye movement data.

[0085] In the eye movement test trajectory generation method of the embodiment of the present application, real eye movement data is collected to train the adversarial neural network model, and then virtual eye movement data is generated based on the trained adversarial neural network model according to the virtual eye movement starting point and the virtual eye movement end point, and the dynamic available data of the human eye simulation device is determined according to the virtual eye movement data to control the motion of the human eye simulation device. In this way, only a small amount of real eye movement data needs to be collected for training the adversarial neural network model, and a large amount of virtual eye movement data can be generated based on the trained adversarial neural network model to control the motion of the human eye simulation device, which effectively improves the test efficiency of optical instruments.

[0086] Specifically, an eye tracker can be used to collect real eye movement data of real users, and the real eye movement data includes multiple gaze point coordinates of real users on the screen, and timestamps corresponding to the coordinates of each gaze point. The adversarial neural network model is trained based on the real eye movement data. Afterwards, the virtual eye movement starting point and the virtual eye movement end point are input into the trained adversarial neural network model to generate virtual eye movement data. In order to enable the human eye simulation eye movement device to operate, it is also necessary to determine the dynamic available data of the human eye simulation device based on the virtual eye movement data.

[0087] Furthermore, the dynamically available data can be applied to the human eye simulation device to test the tracking accuracy of the device under test. Taking the device under test as an eye tracking device as an example, the specific process is as follows: According to the dynamically available data, the motion control parameters of the human eye simulation device can be calculated to control the motion of the human eye simulation device. When testing the eye tracking device, the eye tracking device is mounted on the human eye simulation device. The human eye simulation device simulates the movement of the human eye according to the dynamically available data to obtain an eye movement test trajectory. At the same time, the eye tracking device performs eye tracking on the human eye simulation device to obtain an eye tracking trajectory corresponding to the eye movement test trajectory. The eye movement test trajectory is compared with the eye tracking trajectory, and the difference is calculated to determine the tracking accuracy of the eye tracking device. Among them, a threshold value can be set in advance. When the difference is greater than or equal to the threshold value, the tracking accuracy of the eye tracking device is judged to be unqualified; when the difference is less than the threshold value, the tracking accuracy of the eye tracking device is judged to be qualified.

[0088] The device to be tested can also be an ophthalmic optical instrument. Some ophthalmic optical instruments will emit light of various spectra to illuminate the human eye during use. In order to ensure the safety of light of different spectra irradiating the human eye, a human eye simulation device can be used for safety testing. The specific process is as follows: the human eye simulation device is controlled according to the dynamic available data to simulate the rotation of the human eye. The ophthalmic optical instrument sequentially emits light of different spectra to illuminate the human eye simulation device. The human eye simulation device is monitored to determine whether the light emitted by the ophthalmic optical instrument meets the safe range acceptable to the human eye.

[0089] In addition, some ophthalmic optical instruments can be equipped with different lenses or lens combinations to correct human vision. To ensure the accuracy of the correction, a human eye simulation device can be used for precision testing. The specific process is as follows: The ophthalmic optical instrument is mounted on the human eye simulation device, and different lenses or lens combinations are set in the ophthalmic optical instrument. The human eye simulation device is controlled according to the dynamic available data to simulate the rotation of the human eye to be closer to the real human eye. Monitor the effect achieved by light passing through the lens or lens combination to reach the human eye simulation device to determine whether the corrected vision meets the requirements. In the above process, the human eye simulation device simulates the rotation of the human eye according to the dynamic available data to be closer to the real human eye, making the test results more reliable.

[0090] In the related art, the motion control of the human eye simulation device is performed based on the real eye movement data to complete the tracking accuracy test of the eye tracking device. It is necessary to collect a large amount of real eye movement data in advance. The process is cumbersome and the test efficiency is low. In the implementation of the present application, virtual eye movement data is generated based on the trained adversarial neural network model to control the motion of the human eye simulation device, so as to complete the tracking accuracy test of the eye tracking device. In this way, only a small amount of real eye movement data needs to be collected for the training of the adversarial neural network model, and a large amount of virtual eye movement data can be generated based on the trained adversarial neural network model, which effectively improves the test efficiency of the optical instrument.

[0091] See also Figures 2 to 4 In some embodiments, the adversarial neural network model includes an eye movement data generator and an eye movement data discriminator, and training the adversarial neural network model (i.e., 020) according to real eye movement data includes:

[0092] 021: Generate virtual data samples based on the noise data by an eye movement data generator, wherein the noise data includes an eye movement starting point sample and an eye movement end point sample;

[0093] 022: Distinguish real eye movement data from virtual data samples through eye movement data discriminator;

[0094] 023: Adjust the weights of the eye movement data generator and the eye movement data discriminator according to the distinction results until the adversarial neural network model reaches Nash equilibrium.

[0095] Specifically, the adversarial neural network model includes an eye movement data generator and an eye movement data discriminator. When the model training starts, the noise data is input into the eye movement data generator, and the noise data includes an eye movement starting point sample and an eye movement end point sample. The eye movement data generator can automatically generate a virtual data sample based on the noise data. The generated virtual data sample and the real eye movement data are input into the eye movement data discriminator together, and the eye movement data discriminator can distinguish between the real eye movement data and the virtual data sample. Afterwards, the weights of the eye movement data generator and the eye movement data discriminator are adjusted according to the distinction result of the eye movement data discriminator until the adversarial neural network model reaches a Nash equilibrium.

[0096] Afterwards, the user can input the virtual eye movement starting point and the virtual eye movement end point into the trained adversarial neural network model according to actual needs. The adversarial neural network model can generate high-quality virtual eye movement data based on the input virtual eye movement starting point and the virtual eye movement end point.

[0097] See also Figure 4 and Figure 5 In some embodiments, the weights of the eye movement data generator and the eye movement data discriminator are adjusted according to the distinction results until the adversarial neural network model reaches a Nash equilibrium (i.e., 023), including:

[0098] 0231: Determine whether the adversarial neural network model reaches Nash equilibrium based on the differentiation results;

[0099] 0232: When the adversarial neural network model does not reach Nash equilibrium, determine whether the distinction result is correct;

[0100] 0233: When the distinction result is incorrect, adjust the weight of the eye movement data discriminator, and return to the step of distinguishing the real eye movement data from the virtual data sample through the eye movement data discriminator;

[0101] 0234: When the distinction result is correct, adjust the weight of the eye movement data generator, and return to the step of generating virtual data samples based on the noise data through the eye movement data generator;

[0102] 0235: When the adversarial neural network model reaches Nash equilibrium, it is determined that the training of the adversarial neural network model is completed.

[0103] Specifically, to determine whether the adversarial neural network model has reached Nash equilibrium, the feedback formula of the eye movement data generator is as follows:

[0104]

[0105] Among them, z (i) is the noise data. G(z (i) ) is the eye movement data generator based on the noise data z (i) Generated virtual data samples. D(G(z (i) )) is the judgment accuracy of the eye movement data discriminator on the virtual data sample, which is between 0-100%.

[0106] The feedback formula of the eye movement data discriminator is as follows:

[0107]

[0108] Among them, x (i) is the real eye movement data. (i) ) is the accuracy of the eye movement data discriminator on the real eye movement data, which is between 0-100%.

[0109] When any one of the feedback formula of the eye movement data generator and the feedback formula of the eye movement data discriminator is not at the peak value, it is determined that the adversarial neural network model has not reached the Nash equilibrium. When the feedback formula of the eye movement data generator and the feedback formula of the eye movement data discriminator are both at the peak value, it is determined that the adversarial neural network model has reached the Nash equilibrium.

[0110] When the adversarial neural network model does not reach Nash equilibrium, it is determined whether the distinction result of the eye movement data discriminator is correct, and the weight of the eye movement data generator or the eye movement data discriminator is adjusted according to whether the distinction result is correct or not. When the distinction result is incorrect, it means that the eye movement data discriminator cannot accurately distinguish between real eye movement data and virtual data samples. Therefore, it is necessary to adjust the weight of the eye movement data discriminator and return to the step of distinguishing real eye movement data from virtual data samples through the eye movement data discriminator. Repeat the above steps until the adversarial neural network model reaches Nash equilibrium.

[0111] When the distinction result is correct, it means that the eye movement data discriminator can accurately distinguish between real eye movement data and virtual data samples, and there is no need to adjust the weights of the eye movement data discriminator. At this time, the adversarial neural network model has not reached Nash equilibrium, which means that the quality of the virtual data samples generated by the eye movement data generator is poor. Therefore, it is necessary to adjust the weights of the eye movement data generator and return to the step of generating virtual data samples based on noise data through the eye movement data generator. Repeat the above steps until the adversarial neural network model reaches Nash equilibrium.

[0112] When the adversarial neural network model reaches Nash equilibrium, it is determined that the training of the adversarial neural network model is completed, and the training of the adversarial neural network model ends. Nash equilibrium represents a stable state of the strategies of the eye movement data generator and the eye movement data discriminator in the adversarial neural network model. When Nash equilibrium is reached, neither the eye movement data generator nor the eye movement data discriminator can obtain better results by unilaterally changing the strategy. In this way, it is ensured that the adversarial neural network model will not experience drastic fluctuations during the training process, and the robustness of the adversarial neural network model is improved. For the adversarial neural network model that reaches Nash equilibrium through training, the virtual data samples generated by the eye movement data generator will be more realistic, and the distinction results of the eye movement data discriminator will be more accurate, so that the authenticity of the virtual eye movement data generated based on the adversarial neural network model will also be higher.

[0113] In some embodiments, the incorrect distinction result includes: the eye movement data discriminator determines the real eye movement data as virtual data; the eye movement data discriminator determines the virtual data sample as real data. The correct distinction result includes: the eye movement data discriminator determines the virtual data sample as virtual data.

[0114] Specifically, when the eye movement data discriminator judges real eye movement data as virtual data, it means that the distinction result is incorrect; when the eye movement data discriminator judges virtual data samples as real data, it means that the distinction result is incorrect; in both cases, the weight of the eye movement data discriminator needs to be adjusted.

[0115] When the eye movement data discriminator judges the virtual data sample as virtual data, it means that the distinction result is correct. At this time, there is no need to adjust the eye movement data discriminator. In order to make the adversarial neural network model reach Nash equilibrium, the weight of the eye movement data generator can be adjusted. When the eye movement data discriminator judges the real eye movement data as real data, the weights of the eye movement data discriminator and the eye movement data generator do not need to be adjusted.

[0116] See also Figure 6 and Figure 7 In some embodiments, the virtual eye movement data includes a gaze point coordinate and a timestamp corresponding to the gaze point coordinate, and determining the dynamic available data (i.e., 040) of the human eye simulation device according to the virtual eye movement data includes:

[0117] 041: Based on the particle swarm optimization algorithm and the Hooke-Jeff algorithm, the virtual eye movement data is fitted and optimized so that the curve motion corresponding to the gaze point coordinates and timestamps satisfies the acceleration-uniform speed-deceleration process.

[0118] Specifically, the virtual eye movement data includes the gaze point coordinates and the timestamp corresponding to the gaze point coordinates. The virtual eye movement data is in the format of one timestamp corresponding to one gaze point coordinate. The speed of a segment of movement between two adjacent timestamps is different, and due to the drift of the gaze point, there will be errors in the gaze point coordinates. When the human eye simulation device performs subdivision curve movement, it needs to strictly follow the process of acceleration-uniform speed-deceleration.

[0119] Therefore, in order to control the motion of the human eye simulation device, it is also necessary to determine the dynamic available data based on the virtual eye movement data. A hybrid optimization algorithm combining the particle swarm optimization algorithm and the Hooke-Jeff algorithm can be used to perform fitting optimization processing on the virtual eye movement data, so that the curve motion corresponding to the gaze point coordinates and timestamps satisfies the acceleration-uniform speed-deceleration process.

[0120] The particle swarm optimization algorithm is based on the principle of swarm intelligence. It realizes global search capabilities through information sharing and collaboration between particles, and can effectively find the global optimal solution in the search space. The Hooke-Jeff algorithm is a local search algorithm that is good at performing fine searches in a given area to find local optimal solutions. The hybrid optimization algorithm that combines the two can use the local search capabilities of the Hooke-Jeff algorithm to fine-tune the search results based on the global search of the particle swarm optimization algorithm. In this way, the search efficiency and global optimization capabilities can be improved, thereby improving the accuracy and stability of the optimization results.

[0121] See also Figure 7 and Figure 8In some embodiments, based on the particle swarm optimization algorithm and the Hooke-Jeff algorithm, the virtual eye movement data is subjected to fitting optimization processing so that the curve motion corresponding to the gaze point coordinates and the timestamp satisfies the process of acceleration-uniform speed-deceleration (i.e., 041), including:

[0122] 0411: Initialize the particle swarm information, which includes the position and velocity of the particles;

[0123] 0412: Calculate the population fitness function value based on particle swarm information and virtual eye movement data;

[0124] 0413: Update the position and velocity of the particles according to the population fitness function value, and recalculate the population fitness function value according to the updated particle position and velocity and virtual eye movement data;

[0125] 0414: When the value of the population fitness function drops by more than a predetermined proportion, the position and velocity of the updated particles are optimized based on the Hooke-Jeff algorithm.

[0126] Specifically, the virtual eye movement data is optimized based on a particle swarm optimization algorithm, and the particle swarm optimization algorithm can adopt a particle swarm optimization algorithm with inertia weight. First, the particle swarm group information is initialized, the particle swarm includes multiple particles, each particle represents a fixation point in the virtual eye movement data, and the particle swarm group information includes the position and speed of the particle.

[0127] In the implementation of the present application, it is necessary to optimize the velocity, acceleration, deceleration and curve equation of the fixation point. The curve equation is determined according to the virtual eye movement data. If the virtual eye movement data includes n fixation points, the curve equation is n-order and includes n+1 optimization variables. The position of the particle swarm in the solution space is represented by X i =(x i1 , x i2 , ..., x in ), the speed is expressed as V i =(v i1 , v i2 , ..., v in ).

[0128] The population fitness function value is calculated based on the particle swarm information and virtual eye movement data. The population fitness function is as follows:

[0129]

[0130] Among them, weight 1 and weight 2 is the weight. n is the number of fixation points in the virtual eye movement data. is the coordinate of fixation point i in the virtual eye movement data, is the coordinate of the fixation point i obtained by the particle swarm optimization algorithm. is the absolute distance difference between the fixation point i in the virtual eye movement data and the fixation point i fitted by the particle swarm optimization algorithm. is the timestamp of fixation point i in the virtual eye movement data, is the timestamp of the fixation point i obtained by the particle swarm optimization algorithm. is the difference between the timestamp corresponding to the fixation point i in the virtual eye movement data and the fixation point i obtained by the particle swarm optimization algorithm. The curve where the fixation point i is located obtained by the particle swarm optimization algorithm is the curve represented by the above curve equation. The purpose of optimization is to make the distance from each fixation point in the virtual eye movement data to the above curve as short as possible.

[0131] In the particle swarm optimization algorithm, the particle swarm is optimized for multiple iterations, and the position and speed of the particles are continuously updated. In each iteration, the population fitness function value of the particle swarm is calculated, and the optimal solution of each particle itself can be determined based on the population fitness function value, and the overall optimal solution of the particle swarm can also be determined. The position and speed of the particle can be updated based on the optimal solution of each particle itself and the overall optimal solution of the particle swarm. It should be noted that the particle swarm optimization algorithm with inertia weight used in the embodiment of the present application has an inertia weight added compared to the particle swarm optimization algorithm in the related art, and the convergence speed of the algorithm can be adjusted by adjusting the inertia weight.

[0132] After the position and velocity of the particles are updated, the population fitness function value of the particle swarm is calculated again, and the decrease ratio of the population fitness function value is calculated. When the population fitness function value decreases by more than a predetermined ratio, it indicates that the optimization of the virtual eye movement data based on the particle swarm optimization algorithm is completed, and the iterative optimization of the particle swarm is ended, and then the position and velocity of the updated particles are optimized based on the Hooke-Jeff algorithm. The predetermined ratio is set according to the actual situation. The smaller the predetermined ratio is set, the slower the convergence speed of the particle swarm optimization algorithm, but the more accurate the optimization result. In the implementation mode of the present application, the predetermined ratio can be set between 5% and 10%. For example, the predetermined ratio can be set to 5%, 6%, 7%, 8%, 9%, 10%. In one example, the predetermined ratio is set to 8%.

[0133] The optimization principle based on the Hooke-Jeff algorithm is as follows: search for a direction along a base direction that makes the objective function value smaller. The objective function can use the above-mentioned population fitness function. If the objective function value in this direction does not decrease, search in the opposite direction of the base direction. If none is found, switch to the next base direction and repeat this cycle. If all base directions cannot be found, return to the origin, shorten the step size, and restart the search until the algorithm converges. After the algorithm converges, the final position and velocity of the particle are obtained, and the dynamic available data can be determined. The dynamic available data may include the optimized gaze point coordinates, the timestamp corresponding to the gaze point coordinates, the velocity, and the acceleration or deceleration.

[0134] See also Figure 7 and Fig. 9 In some embodiments, based on the particle swarm optimization algorithm and the Hooke-Jeff algorithm, the virtual eye movement data is subjected to fitting optimization processing so that the curve motion corresponding to the gaze point coordinates and the timestamp satisfies the process of acceleration-uniform speed-deceleration (i.e., 041), including:

[0135] 0415: When the population fitness function value decreases by less than a predetermined ratio, determine whether the population fitness function value decreases in the most recent predetermined number of iterations;

[0136] 0416: When the population fitness function value decreases in the most recent predetermined number of iterations, return to the step of calculating the population fitness function value according to the particle swarm information;

[0137] 0417: When the value of the population fitness function does not decrease in the most recent predetermined number of iterations, the position and velocity of the updated particles are optimized based on the Hooke-Jeff algorithm.

[0138] Specifically, when the population fitness function value decreases by no more than a predetermined proportion, it is necessary to further determine whether the population fitness function value decreases in the most recent predetermined number of iterations of the particle swarm. The predetermined number of times is set according to the actual situation. For example, the predetermined number of times can be set to 50 times. When the population fitness function value decreases in the most recent predetermined number of iterations, it indicates that the optimization can continue based on the particle swarm optimization algorithm, and the step of calculating the population fitness function value based on the particle swarm information is returned to continue the iteration. When the population fitness function value does not decrease in the most recent predetermined number of iterations, it indicates that the optimization of the virtual eye movement data based on the particle swarm optimization algorithm is completed, the iterative optimization of the particle swarm is ended, and the position and velocity of the updated particles are optimized based on the Hooke-Jeff algorithm.

[0139] See also Fig.10 The present application also provides an eye movement control method for controlling a human eye simulation device. The eye movement control method includes:

[0140] 0110: Obtaining dynamic available data of a human eye simulation device, wherein the dynamic available data is derived from: collecting real eye movement data; training an adversarial neural network model based on the real eye movement data; generating virtual eye movement data based on the adversarial neural network model according to a virtual eye movement starting point and a virtual eye movement end point; determining dynamic available data based on the virtual eye movement data;

[0141] 0120: Compute motion control data for human eye simulation devices based on dynamically available data.

[0142] In some embodiments, the motion control data includes one or more of a rotation angle, a running speed, a running acceleration, and a running deceleration.

[0143] Specifically, the motion control data of the human eye simulation device can be calculated based on the dynamically available data, so as to be applied to the human eye simulation device and perform motion control on the human eye simulation device. The dynamically available data may include optimized gaze point coordinates and timestamps corresponding to the gaze point coordinates. The rotation angles of each rotation axis in the human eye simulation device can be calculated based on the dynamically available data. The optimized gaze point speed and acceleration or deceleration can also be calculated based on the optimized gaze point coordinates and timestamps corresponding to the gaze point coordinates. The running speed and running acceleration or running deceleration of the human eye simulation device when tracking the corresponding gaze point can be calculated based on the dynamically available data, the optimized gaze point speed and acceleration or deceleration combined with the motion parameters of the human eye simulation device. The human eye simulation device can be motion controlled in combination with the rotation angle, so that the human eye simulation device moves according to the trajectory formed by the dynamically available data.

[0144] It should be noted that the dynamically available data can be applied to any human eye simulation device. For example, the dynamically available data can be applied to a human eye simulation device having two rotation axes, three rotation axes, four rotation axes or more rotation axes, without limitation herein.

[0145] Taking a human eye simulation device with two rotation axes as an example, the human eye simulation device and an eye tracking method for controlling the motion of the human eye simulation device are described in detail below.

[0146] The human eye simulation device is a mechanical device that simulates the movement of the human eye. It can be used in many fields such as eye movement equipment testing, eye disease diagnosis and treatment, visual function assessment, human-computer interaction, etc.

[0147] like Fig.11 As shown, the human eye simulation device 200 of the embodiment of the present application includes: a simulated eyeball 201 , a first rotating component 230 and a second rotating component 240 .

[0148] The first rotating component 230 is connected to the simulated eyeball 201. The line connecting the pupil and the center of the simulated eyeball 201 forms a first angle with the axial direction of the first rotating component 230. The first rotating component 230 is used to drive the simulated eyeball 201 to move axially around the first rotating component 230 to adjust the pitch angle of the simulated eyeball 201.

[0149] The pitch angle refers to the angular change in the gaze direction of the simulated eyeball 201 in the vertical direction, corresponding to the upward or downward eye movement of the human eye. The second rotating component 240 connects the simulated eyeball 201 and the first rotating component 230, and the second rotating component 240 is used to drive the simulated eyeball 201 and the first rotating component 230 to move axially around the second rotating component 240 to adjust the horizontal azimuth of the simulated eyeball 201.

[0150] In this embodiment, the first rotating component 230 is used to adjust the pitch angle of the simulated eyeball 201, and the second rotating component 240 is used to adjust the horizontal azimuth angle of the simulated eyeball 201. The rotation axis of the first rotating component 230 and the rotation axis of the second rotating component 240 are in two non-parallel planes. The first rotating component 230 and the second rotating component 240 can realize the all-round rotation of the simulated eyeball 201.

[0151] It should be noted that the line of sight of the simulated eyeball 201 is the direction of the line connecting the pupil and the center of the eyeball of the simulated eyeball 201. The line connecting the pupil and the center of the eyeball forms a first angle with the axial direction of the first rotating component 230 (the first angle is not 0° or 180°). When the first rotating component 230 drives the simulated eyeball 201 to move around the axial direction of the first rotating component 230, the vertical position of the pupil on the simulated eyeball 201 changes, and the angle between the line of sight and the horizontal plane changes accordingly, thereby realizing the pitch angle adjustment.

[0152] It can be understood that if the line connecting the pupil and the center of the eyeball is in the same straight line as the axis of the first rotating component 230, when the first rotating component 230 drives the simulated eyeball 201 to move around the axis of the first rotating component 230, the pupil of the simulated eyeball 201 is on the rotation axis, and the position of the pupil in the vertical direction will not change, and the pitch angle adjustment cannot be achieved.

[0153] In actual implementation, the axial setting angle of the first rotating component 230 can be adjusted according to the angle range of the pitch angle required to adjust the simulated eyeball 201. The first rotating component 230 can be set in a horizontal direction or in an inclined direction with a certain angle to the horizontal direction.

[0154] For example, the axis of the first rotating component 230 is horizontal, and the line connecting the pupil and the center of the eyeball forms a first angle with the axis of the first rotating component 230. When the first rotating component 230 drives the simulated eyeball 201 to rotate, the movement trajectory of the pupil on the simulated eyeball 201 is on a vertical plane, thereby realizing the pitch angle adjustment of the line of sight of the simulated eyeball 201.

[0155] For another example, the axial direction of the first rotating component 230 is inclined at a certain angle to the horizontal direction, and the axial direction of the first rotating component 230 is not completely vertical. The line connecting the pupil and the center of the eyeball forms a first angle with the axial direction of the first rotating component 230. When the first rotating component 230 drives the simulated eyeball 201 to rotate, the movement trajectory of the pupil on the simulated eyeball 201 is on a non-horizontal plane, and the pitch angle adjustment can also be achieved.

[0156] In this embodiment, the pitch angle adjustment of the simulated eyeball 201 can be achieved through the first driving component, and the horizontal azimuth angle adjustment of the simulated eyeball 201 can be achieved through the second driving component.

[0157] In actual implementation, the first driving assembly may include a first driving device and a first rotating component 230, wherein the first driving device is used to drive the first rotating component 230 to drive the simulated eyeball 201 to rotate around the axial direction of the first rotating component 230, so as to adjust the pitch angle of the simulated eyeball 201. The second driving assembly may include a second driving device and a second rotating component 240, wherein the second driving device is used to drive the second rotating component 240 to drive the simulated eyeball 201 and the first rotating component 230 to rotate around the axial direction of the second rotating component 240 synchronously.

[0158] In this embodiment, the first driving device is a power source for driving the first rotating component 230 to rotate, and the second driving device is a power source for driving the second rotating component 240 to rotate. The first driving device and the second driving device can be stepping motors, DC motors and other devices.

[0159] It can be understood that the rotation of the second rotating component 240 alone will not affect the pitch angle of the simulated eyeball 201; the pitch angle and horizontal azimuth angle of the simulated eyeball 201 are adjusted respectively by the first rotating component 230 and the second rotating component 240, so that the line of sight of the simulated eyeball 201 can focus on any target point to be tracked.

[0160] The rotation axis of the first rotating component 230 is not parallel to the rotation axis of the second rotating component 240 and the two rotation axes are in two non-parallel planes to ensure that, when the human eye simulation device 200 tracks the target point, according to the position information of the simulated eyeball 201 and the target point and the first angle, the first rotating component 230 drives the simulated eyeball 201 to rotate to adjust the pitch angle of the simulated eyeball 201, and the second rotating component 240 drives the simulated eyeball 201 and the first rotating component 230 to rotate synchronously to adjust the horizontal azimuth angle of the simulated eyeball 201.

[0161] In actual implementation, the axial direction of the first rotating component 230 and the axial direction of the second rotating component 240 can be perpendicular to each other, which is suitable for looking at a target located in front of the simulated eyeball 201; the axial direction of the first rotating component 230 can be tilted upward or downward relative to the horizontal direction, so that the simulated eyeball 201 looks at a target point with a larger pitch angle.

[0162] The human eye simulation device 200 is used to drive the simulated eyeball 201 to rotate to a first target angle for pitch adjustment through the first rotating component 230 according to the eyeball coordinate information of the simulated eyeball 201 and the position information of the target point to be tracked, and to drive the simulated eyeball 201 to rotate to a second target angle for horizontal azimuth adjustment through the second rotating component 240, so that the simulated eyeball focuses on the target point. The first target angle corresponds to the angle that the first rotating component 230 needs to drive the simulated eyeball 201 to rotate, and the second target angle corresponds to the angle that the second rotating component 240 needs to drive the simulated eyeball 201 and the first rotating component 230 to rotate simultaneously.

[0163] When the first rotating component 230 drives the simulated eyeball 201 to rotate the first target angle, and the second rotating component 240 drives the simulated eyeball 201 and the first rotating component 230 to rotate the second target angle, the pupil of the simulated eyeball 201 and the target point can be adjusted to be in the same straight line, that is, the line of sight is aligned with the target point.

[0164] It can be understood that the eye coordinate information of the two simulated eyeballs 201 is different, and the first target angle and the second target angle of the two simulated eyeballs 201 are also different.

[0165] In this embodiment, the first target angle and the second target angle can be calculated by subtracting the current eye coordinate information of the simulated eye 201 and the corresponding eye coordinate information when the simulated eye 201 rotates to the fixation target point.

[0166] Based on the first target angle and the second target angle, control parameters for controlling the first rotating component 230 to rotate to the first target angle and the second rotating component 240 to rotate to the second target angle are calculated and output as corresponding target control instructions. The first rotating component 230 is controlled to rotate to the first target angle by the first driving device, and the second rotating component 240 is controlled to rotate to the second target angle by the second driving device, so as to simulate the eyeball 201 gazing at the target point.

[0167] In actual implementation, the motion controller of the human eye simulation device 200 can control the rotation direction, speed and other parameters of the first drive device and the second drive device to drive the first rotating component 230 and the second rotating component 240 to rotate, and the first drive device and the second drive device move simultaneously to simulate the pitch angle rotation and horizontal rotation of the eyeball 201 and simulate the multi-dimensional complex movement of the human eye.

[0168] The pitch angle rotation and horizontal rotation of the simulated eyeball 201 can be controlled separately.

[0169] According to the human eye simulation device 200 provided in the embodiment of the present application, by setting the first rotating component 230 and the second rotating component 240, the all-round angle adjustment of the simulated eyeball 201 is realized, and the action of the simulated eyeball 201 looking at the target point is decomposed into the rotation of the first rotating component 230 and the second rotating component 240. Through the pitch angle rotation and the horizontal azimuth angle rotation, the simulated eyeball 201 looks at the target point, accurately simulating the complex multi-dimensional movements of the human eye, and effectively improving the movement control accuracy of the simulated eyeball 201.

[0170] In some embodiments, the human eye simulation device 200 further includes an eyeball base 260 , and the simulated eyeball 201 is located on the eyeball base 260 .

[0171] The first rotating component 230 is connected to the simulated eyeball 201 through the eyeball base 260 , and the second rotating component 240 is connected to the first rotating component 230 and the simulated eyeball 201 through the eyeball base 260 and can drive the two to rotate synchronously.

[0172] The third driving component is connected to the two eyeball bases 100 respectively, and drives the two eyeball bases 260 to move towards or away from each other, so as to adjust the pupil distance between the two simulated eyeballs 201 .

[0173] In this embodiment, two eyeball bases 260 may be arranged side by side along the first direction, and each eyeball base 260 may be provided with a simulated eyeball 201 .

[0174] It is understandable that the orientation relationship indicated by the first direction is only for the convenience of describing the arrangement of the two eyeball bases 260, rather than indicating or implying that the eyeball bases 260 or simulated eyeballs 201 must have a specific orientation relationship.

[0175] For example, the first direction may be a horizontal left-right direction, a horizontal front-back direction, or other directions.

[0176] The eyeball base 260 is a base for placing the simulated eyeball 201 , and mechanisms such as the bracket 210 , the first rotating component 230 and the second rotating component 240 may also be arranged on the eyeball base 260 .

[0177] In actual implementation, the simulated eyeball 201, the bracket 210 and the first rotating component 230 can be placed on the upper cover of the eyeball base 260; the second rotating component 240 can be placed in the accommodating cavity of the eyeball base 260, and the part extending from the upper cover is connected to the bracket 210, driving the simulated eyeball 201, the bracket 210 and the first rotating component 230 to rotate in the horizontal direction together.

[0178] In this embodiment, the second driving device can also be arranged in the accommodating cavity of the eyeball base 260, wherein one end of the second rotating component 240 is connected to the output end of the second driving device, and the other end of the second rotating component 240 is connected to the bracket 210.

[0179] In some embodiments, when the two simulated eyeballs 201 are at the starting position, the axes of the two first rotating components 230 form a second angle.

[0180] It should be noted that the simulated eyeball 201, the bracket 210 of the simulated eyeball 201 and the first rotating component 230 are relatively protruding structures in the human eye simulation device 200 (the second rotating component 240 is placed vertically and can be set in the eyeball base 260 of the simulated eyeball 201, and is relatively non-protruding). If the two first rotating components 230 are placed in the same direction, the rotation of the second rotating component 240 may cause mechanical interference to structures such as the simulated eyeball 201, the bracket 210 and the first rotating component 230. The protruding simulated eyeball 201 may also mechanically interfere with the eye tracker waiting to be tested, thereby affecting the equipment test results.

[0181] like Fig.12As shown, when the two simulated eyeballs 201 are in the starting position, the axes of the two first rotating components 230 form a second angle, that is, the relatively protruding simulated eyeballs 201 are offset and placed, leaving a rotation space without interference for structures such as the simulated eyeballs 201, the bracket 210 and the first rotating component 230. When the second rotating component 240 rotates, mechanical interference between the simulated eyeballs 201, the bracket 210 and the first rotating component 230 can be avoided, and the rotation of the two first rotating components 230 does not affect each other, thereby ensuring the normal operation of the simulated eyeball 201 in two directions.

[0182] In this embodiment, the axes of the two first rotating components 230 form a second angle, and the simulated eyeball 201 is placed in an offset manner, which can effectively prevent mechanical interference of the human eye simulation device 200.

[0183] In some embodiments, the human eye simulation device 200 may further include a third driving component for adjusting the pupil distance between the two simulated eyeballs 201 .

[0184] In this embodiment, the third driving component is connected to the two eyeball bases 260 , and the third driving component is used to drive the two eyeball bases 260 to move toward or away from each other along a first direction to adjust the pupil distance between the two simulated eyeballs 201 .

[0185] It can be understood that the simulated eyeball 201 is placed on the eyeball base 260. When the third driving component drives the two eyeball bases 260 to move toward or away from each other, the distance between the two eyeball bases 260 changes, and the pupil distance between the two simulated eyeballs 201 changes accordingly. The third driving component can be controlled by a motion controller to adjust the pupil distance between the two simulated eyeballs 201, thereby simulating human eye conditions with different pupil distances.

[0186] In some embodiments, the third driving assembly includes a third driving device and a linear slide 270 , the third driving device and the linear slide 270 are connected, and two eyeball bases 260 are disposed on the linear slide 270 .

[0187] In this embodiment, the motion controller outputs control instructions to the third drive device. Driven by the third drive device, the linear slide 270 can drive the two eyeball bases 260 to achieve linear motion, thereby adjusting the pupil distance between the two simulated eyeballs 201.

[0188] It can be understood that the third driving device is a power source for driving the linear slide 270 , and the third driving device can be a stepping motor, a DC motor or the like.

[0189] In actual implementation, the human eye simulation device 200 may include a device base, in which a storage space is defined for accommodating a third driving device. A linear slide 270 is installed on the device base, and two eyeball bases 260 are arranged on the linear slide 270. Under the drive of the third driving device, the pupil distance between the two simulated eyeballs 201 is adjusted.

[0190] In some embodiments, the maximum rotation angle of the second rotating component 240 is determined based on the pupil distance of the two simulated eyeballs 201 .

[0191] It can be understood that by analyzing the gaze directions of the two simulated eyeballs 201 based on the pupil distance of the two simulated eyeballs 201, the rotation angle range of the two simulated eyeballs 201 that mechanically interferes can be determined, and then the maximum rotation angle of the simulated eyeball 201 that can be driven by the second rotating component 240 to achieve horizontal rotation can be determined.

[0192] In some embodiments, the maximum rotation angle of the first rotating component 230 is 360 degrees, and the maximum rotation angle of the second rotating component 240 is less than 360 degrees.

[0193] In this embodiment, the first rotating component 230 can drive the simulated eyeball 201 to rotate 360 ​​degrees in all directions and adjust the pitch angle of the simulated eyeball 201; the maximum rotation angle of the simulated eyeball 201 driven by the second rotating component 240 is less than 360 degrees, which can avoid mechanical interference between the simulated eyeball 201, the bracket 210 and the first rotating component 230 and other structures. The rotation angle less than 360 degrees is also more in line with the rotation law of the real human eye.

[0194] In some embodiments, the human eye simulation device 200 further includes a support 210 .

[0195] like Fig.13 As shown, the bracket 210 includes a first fixing plate 211 and a second fixing plate 212 that are perpendicular to each other. The other end of the first rotating component 230 passes through the mounting hole of the first fixing plate 211 and is connected to the simulated eyeball 201; the other end of the second rotating component 240 is connected to the second fixing plate 212.

[0196] The bracket 210 is a frame for supporting and installing the simulated eyeball 201 , and each simulated eyeball 201 corresponds to one bracket 210 .

[0197] In this embodiment, the simulated eyeball 201 is installed on the bracket 210, the first rotating component 230 is installed on the first fixed plate 211, and the second rotating component 240 is installed on the second fixed plate 212. The first rotating component 230 drives the simulated eyeball 201 to rotate, and the second rotating component 240 drives the bracket 210, the simulated eyeball 201 and the first rotating component 230 to rotate together.

[0198] It should be noted that when the two simulated eyeballs 201 are in the starting position, the axes of the two first rotating parts 230 form an angle. The bracket 210, as a relatively protruding structure in the human eye simulation device 200, simulates the offset placement of the eyeball 201, which can avoid mechanical interference between the bracket 210 and the first rotating part 230 on the bracket 210.

[0199] In actual implementation, the first driving device can also be arranged on the first fixed plate 211, wherein the first driving device is installed on one side of the first fixed plate 211, one end of the first rotating component 230 is connected to the output end of the first driving device, and the other end of the first rotating component 230 passes through the mounting hole of the first fixed plate 211 and is connected to the simulated eyeball 201 on the other side of the first fixed plate 211.

[0200] In some embodiments, the simulated eyeball 201 may include an eyeball structure 110 and a fake eye patch 120 . The sphere of the eyeball structure 110 is provided with a section, and the fake eye patch 120 covers the section. The radius of the fake eye patch 120 is smaller than the radius of the eyeball structure 110 .

[0201] Among them, the eyeball structure 110 is a spherical structure that simulates the human eye. The eyeball structure 110 is provided with a cut surface, and the cut surface is covered with a fake eye patch 120 that simulates the human eye iris. The size of the iris is smaller than the size of the entire eyeball. The radius of the fake eye patch 120 can be set according to the radius of the iris of different human eyes.

[0202] In this embodiment, by designing the eyeball structure 110 and the artificial eye patch 120, different sizes, curvatures, and arcs of human eyes can be simulated, so that the simulated eyeball 201 is closer to the real human eye.

[0203] In actual implementation, the eyeball structure 110 can be manufactured using 3D printing technology or CNC machine tool technology. When the human eye simulation device 200 is used to test the human eye simulation device 200, its eyeball structure 110 and the artificial eye patch 120 can be identified by camera image recognition technology to track eye movement trajectories.

[0204] In some embodiments, the cross-section of the eyeball structure 110 is perpendicular to the axial direction of the first rotating component 230 .

[0205] like Fig.14 As shown, the cross-section of the eyeball structure 110 is perpendicular to the axis of the first rotating component 230, and the center position of the artificial eye patch 120 covering the cross-section is the pupil. When the first rotating component 230 drives the simulated eyeball 201 to rotate, the movement trajectory of the pupil of the artificial eye patch 120 is not the maximum circular cross-section of the eyeball structure 110, and there is a certain angle difference between the trajectory of the pupil and the vertical plane in front of the human eye simulation device 200.

[0206] For example, Fig.15 As shown, when the first rotating component 230 of the simulated eyeball 201 corresponding to the left eye rotates clockwise, the pupil initially facing the display screen 300 in front will gradually move away from the display screen 300, and the movement trajectory of the pupil projected on the horizontal plane will also form a certain angle with the display screen 300.

[0207] In this embodiment, the first rotating component 230 of the simulated eyeball 201 rotates to adjust the pitch angle of the simulated eyeball 201, causing a certain position deviation of the pupil, and controlling the second rotating component 240 of the simulated eyeball 201 to rotate for angle compensation.

[0208] For example, when the pupil of the simulated eyeball 201 corresponding to the left eye is away from the display screen 300, the second rotating component 240 of the simulated eyeball 201 is controlled to rotate clockwise to perform angle compensation, thereby shortening the distance between the pupil and the display screen 300.

[0209] In some embodiments, the human eye simulation device 200 may further include a fourth driving component connected to the eyeball base 260 .

[0210] Each fourth driving assembly is connected to an eyeball base 260 , and the fourth driving assembly is used to drive the eyeball base 260 to move along the axial direction of the second rotating component 240 to adjust the height of the simulated eyeball 201 .

[0211] In this embodiment, the eyeball base 260 is a base that can be raised and lowered, and the motion controller outputs a control instruction to the fourth drive component. Driven by the fourth drive component, the eyeball base 260 moves along the axial direction of the second rotating component 240 to achieve height adjustment of the simulated eyeball 201.

[0212] The human eye simulation device 200 of the embodiment of the present application can realize pitch angle rotation, horizontal rotation, pupil distance and height adjustment, and can accurately simulate the multi-dimensional complex movements of different human eyes.

[0213] The embodiment of the present application also provides an eye tracking control method, which, by fixing the pupil distance, decomposes the action of simulating the eyeball 201 looking at the target point into the rotation of the first rotating component 230 and the second rotating component 240, and accurately simulates complex eye movements through pitch angle rotation and horizontal rotation.

[0214] The eye tracking control method of the embodiment of the present application acts on two simulated eyeballs 201 having a first rotating component 230 and a second rotating component 240 to track a target point. The rotation axis of the first rotating component 230 and the rotation axis of the second rotating component 240 are not parallel and the two rotation axes are in two non-parallel planes. The line connecting the pupil and the center of the simulated eyeball 201 forms a first angle with the axial direction of the first rotating component 230.

[0215] The eye tracking control method of the embodiment of the present application can be used to control the movement of the simulated eyeball 201 of the above-mentioned human eye simulation device 200, and accurately simulate complex continuous eyeball movements.

[0216] The executor of the eye tracking control method provided in the embodiment of the present application may be an electronic device or a functional module or functional entity in the electronic device that can implement the eye tracking control method. The eye tracking control method provided in the embodiment of the present application is described below using an electronic device as an example of the executor.

[0217] like Fig.17 As shown, the eye tracking control method includes steps S10 to S30.

[0218] Step S10: Obtain eyeball coordinate information of two simulated eyeballs 201 and coordinate information of a target point to be tracked, and determine the pupil distance between the two simulated eyeballs 201 according to the eyeball coordinate information.

[0219] In this step, a spatial rectangular coordinate system may be established to obtain the coordinate information of the target point and the eye coordinate information of the two simulated eyeballs 201 according to the relative position relationship between the simulated eyeball 201 and the target point.

[0220] For example, Fig.15 As shown, the simulated eyeball 201 is gazing at a target point on the display screen 300. When the human eye simulation device 200 is located at the original position, the eye centers of the two simulated eyeballs 201 are d and e respectively, and the vertical mapping points of d and e on the display screen 300 are a and b respectively. Ad=be=h, where h is the visual depth, i.e., the distance between the eye center of the simulated eyeball 201 and the display screen 300, and de=IPD, where IPD is the pupil distance.

[0221] The midpoint of the line connecting points a and b is the screen center point O of the display screen 300, and point c is the target point on the display screen 300. The target point on the display screen 300 may be determined based on the above-mentioned dynamically available data, and the gaze point included in the dynamically available data may be used as the target point.

[0222] A spatial rectangular coordinate system can be established with the midpoint of the line connecting the two simulated eyeballs 201 (the midpoint of the line connecting points d and e) as the coordinate origin, the line connecting the two simulated eyeballs 201 as the x-axis, the line connecting the midpoint of the line connecting the two simulated eyeballs 201 to the center point of the display screen 300 as the y-axis, and the vertical direction as the z-axis.

[0223] According to the relative position relationship between point c and the center point O of the screen, the coordinate information (x0, h, z0) of point c is obtained, and the eye coordinate information of the two simulated eyeballs 201 are respectively (1 / 2IPD, 0, 0) and (-1 / 2IPD, 0, 0).

[0224] In this embodiment, a spatial rectangular coordinate system can be established at a certain position of the human eye simulation device 200, or at a certain position of the display screen 300, to obtain the coordinate information of the target point and the eye coordinate information of the two simulated eyeballs 201.

[0225] It should be noted that the eyeball coordinate information of the simulated eyeball 201 includes coordinate information of various positions of the simulated eyeball 201 , for example, the eyeball coordinate information includes coordinate information of structural positions such as the eyeball structure 110 , the artificial eye patch 120 , and the pupil.

[0226] In this step, the pupil distance, ie, the distance between the eyeball centers of the two simulated eyeballs 201 , is calculated based on the eyeball coordinate information of the two simulated eyeballs 201 .

[0227] For example, the eyeball coordinate information of the two simulated eyeballs 201 are respectively (1 / 2IPD, 0, 0) and (-1 / 2IPD, 0, 0), and the pupil distance between the two simulated eyeballs 201 is IPD.

[0228] Step S20, when the pupil distance and the target pupil distance meet the preset conditions, based on the eye coordinate information of the two simulated eyeballs 201 and the coordinate information of the target point, determine the first target angle and the second target angle that each simulated eyeball 201 needs to rotate to track the target point.

[0229] The target pupil distance is a preset pupil distance value. The first target angle and the second target angle are also the rotation angles of the human eye simulation device 200 .

[0230] It should be noted that the pupil distance and the target pupil distance meet a preset condition, which may be that the pupil distance is equal to the target pupil distance, or that the pupil distance and the target pupil distance are in a multiple relationship.

[0231] For example, when the pupil distance between the two simulated eyeballs 201 is equal to the preset target pupil distance, the position deviation between the line of sight direction of the simulated eyeball 201 and the target point is determined based on the eye coordinate information and coordinate information of the two simulated eyeballs 201, and the position deviation is adjusted by rotating the first rotating component 230 and the second rotating component 240 so that the line of sight direction of the simulated eyeball 201 is facing the target point, that is, the simulated eyeball 201 is looking at the target point.

[0232] Among them, the first target angle corresponds to the angle that the first rotating component 230 should drive the simulated eyeball 201 to rotate when the simulated eyeball 201 tracks the target point, and the second target angle corresponds to the angle that the second rotating component 240 should drive the first rotating component 230 and the simulated eyeball 201 to rotate synchronously when the simulated eyeball 201 tracks the target point.

[0233] In this step, the position deviation between the line of sight direction of the simulated eyeball 201 and the target point is determined based on the coordinate information of the target point and the eye coordinate information of the simulated eyeball 201, and the line of sight direction of the simulated eyeball 201 is adjusted to face the target point by rotating the first rotating component 230 and the second rotating component 240.

[0234] In actual implementation, the first target angle and the second target angle can be calculated based on the difference between the current eye coordinate information of the simulated eye 201 and the corresponding eye coordinate information when the simulated eye 201 rotates to the fixation target point.

[0235] When the first rotating component 230 drives the simulated eyeball 201 to rotate to the first target angle, and the second rotating component 240 drives the simulated eyeball 201 and the first rotating component 230 to rotate synchronously to the second target angle, the eyeball center, pupil and target point of the simulated eyeball 201 can be adjusted to the same straight line, that is, the line of sight of the simulated eyeball 201 is facing the target point.

[0236] It can be understood that the eye coordinate information of the two simulated eyeballs 201 is different, and the first target angle and the second target angle of the two simulated eyeballs 201 are also different.

[0237] Step S30, based on the first target angle and the second target angle, output a target control instruction to control the first rotating component 230 to drive the simulated eyeball 201 to rotate the first target angle and control the second rotating component 240 to drive the first rotating component 230 and the simulated eyeball 201 to rotate the second target angle synchronously.

[0238] In this embodiment, control parameters for controlling the first rotating component 230 to rotate to a first target angle and the second rotating component 240 to rotate to a first target angle are calculated to obtain corresponding target control instructions, and the first rotating component 230 is controlled to rotate to a first target angle and the second rotating component 240 is controlled to rotate to a second target angle, so as to simulate the eyeball 201 gazing at the target point.

[0239] Take for example that the first rotating component 230 drives the simulated eyeball 201 to rotate in the vertical direction to achieve pitch angle adjustment; and the second rotating component 240 drives the simulated eyeball 201 to rotate in the horizontal direction.

[0240] In this embodiment, based on the coordinate information and the eyeball coordinate information, the first target angle is calculated to be Pc and the second target angle is Qc, and the corresponding target control instructions are output. After the first rotating component 230 is rotated by Pc and the second rotating component 240 is rotated by Qc, the line of sight of the simulated eyeball 201 can be adjusted to face the target point.

[0241] According to the eye tracking control method provided in the embodiment of the present application, the pupil distance between the two simulated eyeballs 201 is fixed, and the action of the simulated eyeball 201 looking at the target point is decomposed into the rotation of the first rotating component 230 and the second rotating component 240 by acquiring the coordinate information of the target point, and motion control is performed so that the simulated eyeball 201 looks at the target point, thereby improving the motion control accuracy of the human eye simulation device 200, and accurately simulating complex eye movements through pitch angle rotation and horizontal rotation.

[0242] In some embodiments, the eye tracking control method may further include:

[0243] When the pupil distance between the two simulated eyeballs 201 and the target pupil distance do not meet the preset condition, the pupil distance between the two simulated eyeballs 201 is adjusted until the pupil distance meets the preset condition.

[0244] For example, the third driving component of the human eye simulation device 200 can be used to drive the two eyeball bases 260 to move toward or away from each other, thereby changing the pupil distance between the two simulated eyeballs 201 until the pupil distance between the two simulated eyeballs 201 is adjusted to the target pupil distance, thereby facilitating the subsequent angle decomposition of the rotation of the first rotating component 230 and the second rotating component 240 and improving the motion control accuracy of the simulated eyeball 201.

[0245] In some embodiments, step S30, based on the eyeball coordinate information of the two simulated eyeballs 201 and the coordinate information of the target point, determining the first target angle and the second target angle that each simulated eyeball 201 needs to rotate to track the target point includes:

[0246] Determine the target pitch angle of the simulated eyeball 201 gazing at the target point based on the eyeball coordinate information and the target point coordinate information;

[0247] Determine a first target angle of the simulated eyeball 201 based on the target pitch angle and the first included angle;

[0248] Based on the eyeball coordinate information, the coordinate information of the target point and the first target angle, a second target angle is determined.

[0249] In this embodiment, based on the eye coordinate information, the target pitch angle required for each simulated eye 201 to look at the target point can be calculated through the right-angle triangle algorithm of the pitch angle. The target pitch angle is used to characterize the angle between the line of sight of the pupil and the horizontal plane when the simulated eye 201 looks at the target point.

[0250] like Fig.16As shown, by connecting the target point of the display screen 300 and the two simulated eyeballs 201, a triangle can be obtained. Through the right triangle algorithm of the pitch angle, the target pitch angle of each simulated eyeball 201 looking at the target point can be calculated.

[0251] Based on the fact that the angle between the pupil's sight line emission direction and the horizontal plane is equal to the target pitch angle, trigonometric calculations can be performed to obtain the vertical coordinate information of the pupil's target position when the simulated eyeball 201 rotates to the target point.

[0252] It should be noted that there is a first angle between the axial direction of the first rotating component 230 and the line of sight emission direction of the simulated eyeball 201. The movement of the pupil position caused by the rotation of the first rotating component 230 is related to the first angle, that is, when calculating the first target angle, it is necessary to combine the first angle for calculation.

[0253] For example, there is a first angle of 45° between the first rotating component 230 and the line of sight of the simulated eyeball 201. When the target pitch angle range of the simulated eyeball 201 is -45°≤β≤45°, after conversion through the rectangular coordinate system and trigonometric functions, the first target angle of -45°≤β≤45° can be obtained. That is, when the first rotating component 230 rotates 360 degrees, it can drive the simulated eyeball 201 to adjust the pitch angle within the angle range of -45°≤β≤45°.

[0254] In actual implementation, the first rotating component 230 drives the simulated eyeball 201 to move axially around the first rotating component 230, driving the pupil to move from the current position to the target position of the gaze target point, and the first target angle is calculated based on the current position and the target position combined with the first angle.

[0255] It should be noted that the specific calculation and derivation of the target pitch angle by using the eyeball coordinate information and the coordinate information of the target point is derived by those skilled in the art based on the calculation principle of the angle between the line of sight emission direction and the horizontal plane, which will not be repeated here. It can be understood that when the two simulated eyeballs 201 are placed side by side, corresponding to the left eye and the right eye of the human eye respectively, the eyeball coordinate information of the two simulated eyeballs 201 is different, and the target pitch angles corresponding to the two simulated eyeballs 201 are different, namely β-left and β-right, and the first target angle calculated in combination with the first angle is also different.

[0256] For example, Fig.16 As shown, two simulated eyeballs 201 are placed side by side, corresponding to the left eye and the right eye respectively. Driven by the first rotating component 230, the two simulated eyeballs 201 look at the target point c above the display screen 300. The two simulated eyeballs 201 correspond to pitch angles β-left and β-right respectively.

[0257] It can be understood that after the rotation angle of the first rotating component 230 is fixed, the rotation of the second rotating component 240 will not change the pitch angle of the simulated eyeball 201. After the first target angle is calculated, the second target angle is calculated by combining the coordinate information of the target point and the eye coordinate information of the simulated eyeball 201.

[0258] In some embodiments, determining the second target angle based on the eyeball coordinate information, the target point coordinate information, and the first target angle may include:

[0259] Based on the first target angle and the movement trajectory of the pupil, determining the projection displacement deviation of the pupil on the horizontal plane before and after the first rotating component 230 rotates;

[0260] Based on the projection displacement deviation and the distance between the projection and the center of the eyeball, a second compensation angle is determined, and the second compensation angle compensates for the angular deviation of the pupil in the horizontal plane;

[0261] Determine the target azimuth of the simulated eyeball 201 on the horizontal plane based on the coordinate information of the target point and the coordinate information of the eyeball;

[0262] A second target angle is determined based on the second compensation angle and the target azimuth angle.

[0263] It should be noted that when the first rotating component 230 rotates to the first target angle, the vertical projection of the movement trajectory of the pupil of the simulated eyeball 201 to the horizontal plane will produce a certain displacement (i.e., projection displacement deviation), resulting in an angular deviation of the pupil in the horizontal azimuth. The second compensation angle is calculated to compensate for the angular deviation of the pupil in the horizontal plane caused by the first rotating component 230 rotating to the first target angle.

[0264] For example, Fig.18 As shown, circle R1 is a motion trajectory simulating a pupil rotation of the eyeball 201, and r is the radius of the plane where the pupil motion trajectory is located.

[0265] In this embodiment, the first rotating component 230 rotates the first target angle β, and the pupil moves from the K1 position to the K2 position. According to the trigonometric function relationship within the circle, the projection displacement deviation a of the pupil on the horizontal plane caused by the first rotating component 230 rotating the first target angle can be calculated. 1 =r×cosβ.

[0266] like Fig.19 As shown, circle R2 is the vertical projection of the simulated eyeball 201, which can be regarded as the projection of the cross section of the second rotating component 240. The angle offset generated when the pupil moves from position K1 to position K2 is α 1 , that is, the second compensation angle is α 1, the distance between the projection and the center of the eyeball can be the distance between the plane where the actual movement trajectory of the pupil is located and the center of the eyeball.

[0267] In this embodiment, according to the projection displacement deviation a1 and the distance h1 between the projection and the eyeball center, the angle can be calculated by the trigonometric function relationship of the vertical projection. Draw an auxiliary dotted line to calculate the second compensation angle α 1 .

[0268] In the above-mentioned embodiment, the training of the adversarial neural network model is carried out in the host computer, and dynamic available data is generated. The gaze point included in the dynamic available data is displayed on the display screen 300 as a target point to calculate the rotation angle. Of course, other possible methods are also included. For example, in other embodiments, the display screen 300 may not be required, and the human eye simulation device 200 can directly read the dynamic available data, calculate the rotation angle inside the human eye simulation device 200, and control the movement of the human eye simulation device 200; or, the host computer may not be required, and a chip that can be used for model training is carried in the human eye simulation device 200, and the adversarial neural network model is trained in the human eye simulation device 200, and dynamic available data is generated, and the rotation angle is calculated to control the movement of the human eye simulation device 200. As long as it is possible to use the dynamic available data for the human eye simulation device so that it moves according to the trajectory formed by the dynamic available data, it is within the scope of the present application.

[0269] See also Fig. 20 , the embodiment of the present application also provides an eye movement test trajectory generation device 400, which is applied to a human eye simulation device. The eye movement test trajectory generation device 400 includes a collection module 410, a training module 420, a generation module 430 and a determination module 440. The collection module 410 collects real eye movement data. The training module 420 is used to train an adversarial neural network model based on real eye movement data. The generation module 430 is used to generate virtual eye movement data based on the adversarial neural network model according to a virtual eye movement starting point and a virtual eye movement end point. The determination module 440 is used to determine the dynamic available data of the human eye simulation device based on the virtual eye movement data.

[0270] In some embodiments, the adversarial neural network model includes an eye movement data generator and an eye movement data discriminator, and the training module 420 is specifically used to generate virtual data samples based on noise data through the eye movement data generator, wherein the noise data includes eye movement starting point samples and eye movement end point samples; distinguish real eye movement data from virtual data samples through the eye movement data discriminator; and adjust the weights of the eye movement data generator and the eye movement data discriminator according to the distinction results until the adversarial neural network model reaches a Nash equilibrium.

[0271] In certain embodiments, the training module 420 is specifically used to determine whether the adversarial neural network model has reached Nash equilibrium based on the distinction result; when the adversarial neural network model has not reached Nash equilibrium, determine whether the distinction result is correct; when the distinction result is incorrect, adjust the weight of the eye movement data discriminator, and return to the step of distinguishing real eye movement data from virtual data samples through the eye movement data discriminator; when the distinction result is correct, adjust the weight of the eye movement data generator, and return to the step of generating virtual data samples based on noise data through the eye movement data generator; when the adversarial neural network model reaches Nash equilibrium, determine that the training of the adversarial neural network model is completed.

[0272] In certain embodiments, the incorrect differentiation result includes:

[0273] The eye movement data discriminator judges the real eye movement data as virtual data;

[0274] The eye movement data discriminator judges the virtual data samples as real data;

[0275] Correct distinction results include:

[0276] The eye movement data discriminator judges the virtual data samples as virtual data.

[0277] In some embodiments, the virtual eye movement data includes gaze point coordinates and a timestamp corresponding to the gaze point coordinates. The determination module 440 is specifically used to perform fitting optimization processing on the virtual eye movement data based on the particle swarm optimization algorithm and the Hooke-Jeff algorithm, so that the curved motion corresponding to the gaze point coordinates and the timestamp satisfies the acceleration-uniform speed-deceleration process.

[0278] In some embodiments, the determination module 440 is specifically used to initialize particle swarm information, which includes the position and velocity of particles; calculate the population fitness function value based on the particle swarm information and virtual eye movement data; update the position and velocity of particles based on the population fitness function value, and recalculate the population fitness function value based on the updated position and velocity of particles and virtual eye movement data; when the population fitness function value decreases by more than a predetermined proportion, optimize the updated position and velocity of particles based on the Hooke-Jeff algorithm.

[0279] In some embodiments, the determination module 440 is specifically used to determine whether the population fitness function value has decreased in the most recent predetermined number of iterations when the population fitness function value decreases by no more than a predetermined proportion; when the population fitness function value has decreased in the most recent predetermined number of iterations, return to the step of calculating the population fitness function value based on the particle swarm information; when the population fitness function value has not decreased in the most recent predetermined number of iterations, optimize the position and velocity of the updated particles based on the Hooke-Jeff algorithm.

[0280] It should be pointed out that the explanation of the eye movement test trajectory generating method in the aforementioned embodiment is also applicable to the eye movement test trajectory generating device 400 of the embodiment of the present application, and will not be elaborated here.

[0281] See also Fig.21 The embodiment of the present application also provides an eye movement test trajectory generation system 500. The eye movement test trajectory generation system 500 includes one or more processors 510 and a memory 520. The memory 520 stores a computer program. When the computer program is executed by the processor 510, the eye movement test trajectory generation method of any of the above embodiments is implemented.

[0282] For example, when the computer program is executed by the processor 510, the following eye movement test trajectory generation method is implemented:

[0283] 010: Collect real eye movement data;

[0284] 020: Training an adversarial neural network model based on real eye movement data;

[0285] 030: Generate virtual eye movement data based on the adversarial neural network model according to the virtual eye movement starting point and the virtual eye movement end point;

[0286] 040: Determine the dynamic available data of the human eye simulation device based on the virtual eye movement data.

[0287] For another example, when the computer program is executed by the processor 510, the following eye movement test trajectory generation method is implemented:

[0288] 021: Generate virtual data samples based on the noise data by an eye movement data generator, wherein the noise data includes an eye movement starting point sample and an eye movement end point sample;

[0289] 022: Distinguish real eye movement data from virtual data samples through eye movement data discriminator;

[0290] 023: Adjust the weights of the eye movement data generator and the eye movement data discriminator according to the distinction results until the adversarial neural network model reaches Nash equilibrium.

[0291] It should be pointed out that the explanations and descriptions of the eye movement test trajectory generating method and the eye movement test trajectory generating device 400 in the aforementioned embodiments are also applicable to the eye movement test trajectory generating system 500 of the embodiments of the present application, and will not be elaborated here.

[0292] See also Fig. 22The embodiment of the present application also provides a computer-readable storage medium 600 on which a computer program 610 is stored. When the program is executed by a processor 620, the eye movement test trajectory generation method of any of the above embodiments is implemented.

[0293] For example, when the program is executed by the processor 620, the following eye movement test trajectory generation method is implemented:

[0294] 010: Collect real eye movement data;

[0295] 020: Train an adversarial neural network model based on real eye movement data, and generate virtual eye movement data based on the adversarial neural network model;

[0296] 030: Determine the dynamic available data of the human eye simulation device based on the virtual eye movement data.

[0297] For another example, when the program is executed by the processor 620, the following eye movement test trajectory generation method is implemented:

[0298] 021: Generate virtual data samples based on the noise data by an eye movement data generator, wherein the noise data includes an eye movement starting point sample and an eye movement end point sample;

[0299] 022: Distinguish real eye movement data from virtual data samples through eye movement data discriminator;

[0300] 023: Adjust the weights of the eye movement data generator and the eye movement data discriminator according to the distinction results until the adversarial neural network model reaches a Nash equilibrium;

[0301] 024: Generate virtual eye movement data based on the adversarial neural network model according to the input virtual eye movement starting point and virtual eye movement end point.

[0302] It should be pointed out that the explanations and descriptions of the eye movement test trajectory generating method and the eye movement test trajectory generating device 400 in the aforementioned embodiments are also applicable to the computer-readable storage medium 600 of the embodiments of the present application, and will not be elaborated here.

[0303] See also Fig.23 The embodiment of the present application also provides an eye movement control device 700 for controlling a human eye simulation device, and the eye movement control device 700 includes an acquisition module 710 and a calculation module 720. The acquisition module 710 is used to obtain dynamic available data of the human eye simulation device, wherein the dynamic available data comes from: collecting real eye movement data; training an adversarial neural network model based on the real eye movement data; generating virtual eye movement data based on the adversarial neural network model according to the virtual eye movement starting point and the virtual eye movement end point; and determining the dynamic available data according to the virtual eye movement data. The calculation module 720 is used to calculate the motion control data of the human eye simulation device according to the dynamic available data.

[0304] In some embodiments, the motion control data includes one or more of a rotation angle, a running speed, a running acceleration, and a running deceleration.

[0305] It should be pointed out that the explanation of the eye movement control method in the aforementioned embodiment is also applicable to the eye movement control device 700 of the embodiment of the present application, and will not be elaborated here.

[0306] See also Fig.24 The present application also provides an eye movement control system 800. The eye movement control system 800 includes one or more processors 810 and a memory 820. The memory 820 stores a computer program. When the computer program is executed by the processor 810, the eye movement control method of any of the above embodiments is implemented.

[0307] For example, when the computer program is executed by the processor 810, the following eye movement control method is implemented:

[0308] 110: Obtaining dynamic available data of a human eye simulation device, wherein the dynamic available data is derived from: collecting real eye movement data; training an adversarial neural network model according to the real eye movement data; generating virtual eye movement data based on the adversarial neural network model according to a virtual eye movement starting point and a virtual eye movement end point; and determining dynamic available data according to the virtual eye movement data;

[0309] 120: Calculate motion control data of a human eye simulation device based on dynamically available data.

[0310] It should be pointed out that the explanations and descriptions of the eye movement control method and the eye movement control device 700 in the aforementioned embodiments are also applicable to the eye movement control system 800 in the embodiments of the present application, and will not be elaborated here.

[0311] See also Fig.25 The embodiment of the present application also provides a computer-readable storage medium 900, on which a computer program 910 is stored. When the program is executed by a processor 920, the eye movement control method of any of the above embodiments is implemented.

[0312] For example, when the program is executed by the processor 920, the following eye movement control method is implemented:

[0313] 110: Obtaining dynamic available data of a human eye simulation device, wherein the dynamic available data is derived from: collecting real eye movement data; training an adversarial neural network model according to the real eye movement data; generating virtual eye movement data based on the adversarial neural network model according to a virtual eye movement starting point and a virtual eye movement end point; and determining dynamic available data according to the virtual eye movement data;

[0314] 120: Calculate motion control data of a human eye simulation device based on dynamically available data.

[0315] It should be pointed out that the explanations and descriptions of the eye movement control method and the eye movement control device 700 in the aforementioned embodiments are also applicable to the computer-readable storage medium 900 in the embodiments of the present application, and will not be elaborated here.

[0316] In summary, in the eye movement test trajectory generation method, device 400, system 500 and computer-readable storage medium 600, as well as the eye movement control method, device 700, system 800 and computer-readable storage medium 900 of the embodiments of the present application, real eye movement data is collected to train the adversarial neural network model, and then virtual eye movement data is generated based on the trained adversarial neural network model, and the dynamic available data of the human eye simulation device is determined according to the virtual eye movement data to perform motion control on the human eye simulation device. In this way, only a small amount of real eye movement data needs to be collected for training the adversarial neural network model, and a large amount of virtual eye movement data can be generated based on the trained adversarial neural network model to perform motion control on the human eye simulation device, which effectively improves the test efficiency of the eye tracking device.

[0317] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0318] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0319] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0320] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0321] A person of ordinary skill in the art will appreciate that all or part of the steps carried by the method for implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment. In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0322] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application. The scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for generating eye movement test trajectories, It is characterized in that The eye movement test trajectory generation method comprises: Collect real eye movement data; Training an adversarial neural network model according to the real eye movement data; Generate virtual eye movement data based on the adversarial neural network model according to the virtual eye movement starting point and the virtual eye movement end point; Dynamically available data of a human eye simulation device is determined according to the virtual eye movement data.

2. The eye movement test trajectory generation method according to claim 1, It is characterized in that The adversarial neural network model includes an eye movement data generator and an eye movement data discriminator, and the training of the adversarial neural network model according to the real eye movement data includes: Generate virtual data samples based on noise data by the eye movement data generator, wherein the noise data includes an eye movement starting point sample and an eye movement end point sample; Distinguishing the real eye movement data from the virtual data sample by the eye movement data discriminator; The weights of the eye movement data generator and the eye movement data discriminator are adjusted according to the distinction result until the adversarial neural network model reaches a Nash equilibrium.

3. The eye movement test trajectory generation method according to claim 2, It is characterized in that The step of adjusting the weights of the eye movement data generator and the eye movement data discriminator according to the differentiation result until the adversarial neural network model reaches a Nash equilibrium includes: Determine whether the adversarial neural network model reaches Nash equilibrium according to the distinction result; When the adversarial neural network model does not reach Nash equilibrium, determining whether the distinction result is correct; When the distinction result is incorrect, adjusting the weight of the eye movement data discriminator, and returning to the step of distinguishing the real eye movement data from the virtual data sample by the eye movement data discriminator; When the distinction result is correct, adjusting the weight of the eye movement data generator, and returning to the step of generating virtual data samples based on noise data by the eye movement data generator; When the adversarial neural network model reaches a Nash equilibrium, it is determined that the training of the adversarial neural network model is completed.

4. The eye movement test trajectory generation method according to claim 3, It is characterized in that The incorrect distinction results include: The eye movement data discriminator determines the real eye movement data as virtual data; The eye movement data discriminator judges the virtual data sample as real data; The correct distinction results include: The eye movement data discriminator determines the virtual data sample as virtual data.

5. The eye movement test trajectory generation method according to claim 1, It is characterized in that The virtual eye movement data includes a gaze point coordinate and a timestamp corresponding to the gaze point coordinate, and determining the dynamic available data of the human eye simulation device according to the virtual eye movement data includes: Based on the particle swarm optimization algorithm and the Hooke-Jeff algorithm, the virtual eye movement data is subjected to fitting optimization processing so that the curve motion corresponding to the gaze point coordinates and the timestamp satisfies the process of acceleration-uniform speed-deceleration.

6. The eye movement test trajectory generation method according to claim 5, It is characterized in that The fitting optimization processing of the virtual eye movement data based on the particle swarm optimization algorithm and the Hooke-Jeff algorithm so that the curve motion corresponding to the gaze point coordinates and the timestamp satisfies the process of acceleration-uniform speed-deceleration includes: Initializing particle swarm information, wherein the particle swarm information includes positions and velocities of particles; Calculating a population fitness function value according to the particle swarm information and the virtual eye movement data; updating the position and velocity of the particle according to the population fitness function value, and recalculating the population fitness function value according to the updated position and velocity of the particle and the virtual eye movement data; When the population fitness function value decreases by more than a predetermined proportion, the updated position and speed of the particle are optimized based on the Hooke-Jeff algorithm.

7. The eye movement test trajectory generation method according to claim 6, It is characterized in that The method of performing fitting optimization processing on the virtual eye movement data based on the particle swarm optimization algorithm and the Hooke-Jeff algorithm so that the curve motion corresponding to the gaze point coordinates and the timestamp satisfies the process of acceleration-uniform speed-deceleration also includes: When the population fitness function value decreases by no more than the predetermined ratio, determining whether the population fitness function value decreases in the most recent predetermined number of iterations; When the population fitness function value decreases in the most recent predetermined number of iterations, returning to the step of calculating the population fitness function value according to the particle population information and the virtual eye movement data; When the population fitness function value does not decrease in the most recent predetermined number of iterations, the updated position and speed of the particle are optimized based on the Hooke-Jeff algorithm.

8. An eye movement test trajectory generation device, It is characterized in that The eye movement test trajectory generating device comprises: A collection module, used to collect real eye movement data; A training module, used for training an adversarial neural network model according to the real eye movement data; A generating module, used for generating virtual eye movement data based on the adversarial neural network model according to the virtual eye movement starting point and the virtual eye movement end point; A determination module is used to determine the dynamic available data of the human eye simulation device according to the virtual eye movement data.

9. An eye movement test trajectory generation system, It is characterized in that The eye movement test trajectory generation system includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the eye movement test trajectory generation method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the eye movement test trajectory generation method described in any one of claims 1 to 7 is implemented.

11. An eye movement control method, It is characterized in that Used to control a human eye simulation device, the eye movement control method comprises: Acquire dynamic available data of the human eye simulation device, wherein the dynamic available data is derived from: collecting real eye movement data; training an adversarial neural network model according to the real eye movement data; generating virtual eye movement data based on the adversarial neural network model according to a virtual eye movement starting point and a virtual eye movement end point; and determining the dynamic available data according to the virtual eye movement data; The motion control data of the human eye simulation device is calculated according to the dynamically available data.

12. The eye movement control method according to claim 11, It is characterized in that The motion control data includes one or more of a rotation angle, a running speed, a running acceleration, and a running deceleration.

13. An eye control device, It is characterized in that Used to control a human eye simulation device, the eye movement control device comprises: An acquisition module is used to acquire dynamic available data of the human eye simulation device, wherein the dynamic available data is derived from: collecting real eye movement data; training an adversarial neural network model according to the real eye movement data; generating virtual eye movement data based on the adversarial neural network model according to a virtual eye movement starting point and a virtual eye movement end point; and determining the dynamic available data according to the virtual eye movement data; A calculation module is used to calculate the motion control data of the human eye simulation device according to the dynamically available data.

14. An eye movement control system, It is characterized in that The eye movement control system includes one or more processors and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the eye movement control method according to claim 11 or 12 is implemented.

15. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the eye movement control method described in claim 11 or 12 is implemented.

Images