Attention dynamic assessment method and system based on teenager eye movement tracking

Abstract

The invention discloses an attention dynamic assessment method and system based on teenager eye movement tracking, and the method specifically comprises the steps: obtaining eye video data of a target object in response to a starting operation; in response to the analysis operation, generating a classification result of the attention state of the target object according to the eye video data; in response to the visualization operation, generating a visualization attention thermodynamic diagram corresponding to the classification result, and outputting a stage attention attenuation curve report; wherein the step of generating the classification result of the attention state of the target object comprises the steps of calculating relative displacement of a pupil center and a cornea reflection point of the target object by adopting an optical flow algorithm according to eye video data, generating a three-dimensional sight line vector, constructing a space-time attention feature matrix, and finally inputting a pre-trained attention neural network model to obtain the classification result of the attention state of the target object. And outputting a classification result of the attention state. The method can improve the attention assessment accuracy of the teenager object, has the dynamic calibration capability, and can be widely applied to the technical field of computers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a dynamic attention evaluation method and system based on eye movement tracking of teenagers. BACKGROUND

[0002] With the advancement of scientific early education of teenagers, the accuracy and real-time demand of attention evaluation as an important indicator of cognitive development are increasingly prominent. Traditional evaluation methods rely on observer subjective judgment (such as scale score) or single behavior record (such as key response), which has the following significant defects:

[0003] Lack of objectivity: the video behavior analysis method used in the prior art is easily disturbed by the environment and cannot quantify the micro-cognitive activities such as fixation focus shift;

[0004] Dynamic analysis is missing: the existing scheme only counts the cumulative fixation time, and does not build an attention intensity decay model, making it difficult to capture the transient fluctuation characteristics of attention;

[0005] The above technical bottlenecks seriously restrict the intelligent upgrading of education products for teenagers, and an attention evaluation system with non-invasive, high robustness and dynamic calibration capability is urgently needed to meet the urgent needs of precise education intervention. SUMMARY

[0006] The main purpose of the embodiment of the present application is to provide a dynamic attention evaluation method and system based on eye movement tracking of teenagers, which can improve the accuracy of attention evaluation for teenagers and has dynamic calibration capability.

[0007] To achieve the above purpose, one aspect of the embodiment of the present application provides a dynamic attention evaluation method based on eye movement tracking of teenagers, comprising the following steps:

[0008] In response to the start operation, the eye video data of the target object is obtained;

[0009] In response to the analysis operation, the classification result of the attention state of the target object is generated according to the eye video data;

[0010] In response to the visualization operation, the visualization attention heat map corresponding to the classification result is generated and the stage attention decay curve report is outputted;

[0011] The step of generating the classification result of the attention state of the target object includes:

[0012] According to the eye video data, the relative displacement of the pupil center and the corneal reflection point of the target object is calculated by using the optical flow algorithm to generate a three-dimensional visual line vector;

[0013] According to the three-dimensional visual line vector, a space-time attention feature matrix is constructed, wherein the space-time attention feature matrix includes indexes of three dimensions of gaze duration, saccade speed and blink frequency.

[0014] The space-time attention feature matrix is input into a pre-trained attention neural network model, and a classification result of an attention state is output.

[0015] In some embodiments, in response to the start operation, the eye video data of the target object is acquired, including:

[0016] In response to the start operation, the camera of the target device is started, and the eye video data of the target object is collected through the camera;

[0017] Alternatively, in response to the start operation, the eye video data of the target object is loaded from the local storage file of the target device.

[0018] In some embodiments, in response to the start operation, the eye video data of the target object is acquired, further including:

[0019] According to the acquired eye video data, the eye video data is presented in a first display area of a first interface to provide a preview picture or a playable video for the eye video data.

[0020] In some embodiments, according to the eye video data, a light flow algorithm is used to calculate the relative displacement of the pupil center and the corneal reflection point of the target object, and a three-dimensional visual line vector is generated, including the following steps:

[0021] Image preprocessing and feature detection are performed on each frame of image of the acquired eye video data, and a key feature set in the eye video data is obtained;

[0022] According to the key feature set, the pupil center is detected to determine the pupil center of the target object;

[0023] According to the pupil center, the corneal reflection point is located through clustering analysis of infrared reflection points;

[0024] According to the corneal reflection point determined in each frame of image of the eye video data, a motion vector between the pupil and the corneal reflection point in consecutive frames is determined;

[0025] According to the motion vector, the displacement of the pupil center relative to the corneal reflection point is calculated;

[0026] The relative displacement in the two-dimensional image is converted into the visual line direction in the three-dimensional space, the mapping of the three-dimensional visual line vector is completed, and the final three-dimensional visual line vector is obtained.

[0027] In some embodiments, the pupil center detection is performed according to the set of key features, and the pupil center of the target object is determined, specifically:

[0028] The pupil boundary of the target object is fitted by an ellipse formula, and then the pupil center is determined according to the fitted pupil boundary; the expression of this step is:

[0029]

[0030] Where (h, k) represents the coordinates of the center of the ellipse, also representing the pupil center; a and b represent the long axis and the short axis of the ellipse; theta represents the rotation angle; x and y represent the coordinates of the pixel point.

[0031] In some embodiments, the expression of the process of locating the corneal reflection point through clustering analysis of the infrared reflection point according to the pupil center is:

[0032]

[0033] Where x i represents the coordinates of the highlighted pixel in the image; C represents the center of the corneal reflection point.

[0034] Another aspect of the embodiment of the present application also provides an attention dynamic evaluation system based on eye movement tracking of adolescents, comprising:

[0035] The first module is configured to acquire eye video data of a target object in response to a start operation;

[0036] The second module is configured to generate a classification result of an attention state of the target object according to the eye video data in response to an analysis operation;

[0037] The third module is configured to generate a visualized attention heat map corresponding to the classification result and output a stage attention decay curve report in response to a visualization operation;

[0038] The step of generating the classification result of the attention state of the target object includes:

[0039] The relative displacement of the pupil center and the corneal reflection point of the target object is calculated by using an optical flow algorithm according to the eye video data, and a three-dimensional gaze vector is generated;

[0040] A spatio-temporal attention feature matrix is constructed according to the three-dimensional gaze vector, wherein the spatio-temporal attention feature matrix includes indicators in three dimensions of fixation duration, saccade speed and blink frequency;

[0041] The spatio-temporal attention feature matrix is input into a pre-trained attention neural network model, and a classification result of the attention state is output.

[0042] Another aspect of the present application provides an electronic device, comprising a processor and a memory.

[0043] The memory is configured to store a program.

[0044] The processor executes the program to implement the method as described above.

[0045] Another aspect of the present application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the method as described above.

[0046] Another aspect of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the method as described above.

[0047] The embodiments of the present application at least have the following beneficial effects: the present application provides a dynamic attention evaluation method and system based on eye movement tracking of teenagers, which acquires eye video data of a target object in response to a starting operation; generates a classification result of an attention state of the target object according to the eye video data in response to an analysis operation; generates a visualized attention heat map corresponding to the classification result and outputs a stage attention decay curve report in response to a visualization operation; wherein the step of generating the classification result of the attention state of the target object comprises: calculating the relative displacement of the pupil center and the corneal reflection point of the target object by using an optical flow algorithm to generate a three-dimensional gaze vector according to the eye video data; constructing a spatio-temporal attention feature matrix according to the three-dimensional gaze vector, wherein the spatio-temporal attention feature matrix includes indicators of three dimensions of fixation duration, saccade speed and blink frequency; inputting the spatio-temporal attention feature matrix into a pre-trained attention neural network model to output a classification result of the attention state. The present application can improve the accuracy of attention evaluation for teenagers, and has dynamic calibration capability. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is an implementation environment schematic diagram provided by the embodiments of the present application;

[0049] Figure 2 is a flowchart of the overall steps provided by the embodiments of the present application;

[0050] Figure 3 is a product interface schematic diagram provided by the embodiments of the present application;

[0051] Figure 4 is an attention evaluation result display interface diagram provided by the embodiments of the present application;

[0052] Figure 5 is a specific implementation process diagram of step S202 provided by the embodiment of the present application.

[0053] Figure 6 is a hardware structure diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, but is only an example of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0055] It can be understood that the terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0056] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or", used to describe the association between the associated objects, means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application.

[0058] The method and system for dynamically evaluating attention based on eye movement tracking of teenagers provided by the embodiments of the present application relate to the technical field of computer technology. The method for dynamically evaluating attention based on eye movement tracking of teenagers provided by the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; the software can be an application for implementing the method for dynamically evaluating attention based on eye movement tracking of teenagers, and the like, but is not limited to the above forms.

[0059] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0060] As shown in Figure 1 , it is a schematic diagram of an implementation environment provided by the embodiments of the present application. Referring to Figure 1 , the implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected by wireless or wired means to complete data transmission and exchange.

[0061] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0062] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0063] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0064] Exemplary based on Figure 1 The implementation environment shown in this embodiment of the invention provides a dynamic attention assessment method based on adolescent eye tracking. The following description uses the application of this dynamic attention assessment method based on adolescent eye tracking in server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0065] Reference Figure 2 , Figure 2 The flowchart illustrates a method for dynamic attention assessment based on adolescent eye-tracking applied to a server, as provided in this embodiment of the invention. The execution subject of this method can be any of the aforementioned computer devices (including servers or terminals). (Refer to...) Figure 2 The method may include the following steps:

[0066] In response to the startup operation, acquire eye video data of the target object;

[0067] In response to the analysis operation, a classification result of the attention state of the target object is generated based on the eye video data;

[0068] In response to the visualization operation, a visual attention heatmap corresponding to the classification result is generated and a report of the phased attention decay curve is output.

[0069] The step of generating the classification result of the attention state of the target object includes:

[0070] According to the eye video data, a light flow algorithm is used to calculate the relative displacement of the pupil center and the corneal reflection point of the target object, and a three-dimensional visual line vector is generated;

[0071] According to the three-dimensional visual line vector, a space-time attention feature matrix is constructed, wherein the space-time attention feature matrix includes indicators in three dimensions of gaze duration, saccade speed and blink frequency;

[0072] The space-time attention feature matrix is input into a pre-trained attention neural network model, and a classification result of the attention state is output.

[0073] In some embodiments, the eye video data of the target object is acquired in response to the start operation, comprising:

[0074] In response to the start operation, the camera of the target device is started, and the eye video data of the target object is collected through the camera;

[0075] Alternatively, in response to the start operation, the eye video data of the target object is loaded from the local storage file of the target device.

[0076] In some embodiments, the eye video data of the target object is acquired in response to the start operation, further comprising:

[0077] According to the acquired eye video data, the eye video data is presented in a first display area of a first interface to provide a preview picture or a playable video for the eye video data.

[0078] In some embodiments, the three-dimensional visual line vector is generated by calculating the relative displacement of the pupil center and the corneal reflection point of the target object using a light flow algorithm according to the eye video data, comprising the following steps:

[0079] Image preprocessing and feature detection are performed on each frame of image of the acquired eye video data, and a key feature set in the eye video data is obtained;

[0080] According to the key feature set, the pupil center is detected to determine the pupil center of the target object;

[0081] According to the pupil center, the corneal reflection point is located through clustering analysis of infrared reflection points;

[0082] According to the corneal reflection point determined in each frame of image of the eye video data, the motion vector of the pupil and the corneal reflection point between consecutive frames is determined;

[0083] According to the motion vector, the displacement of the pupil center relative to the corneal reflection point is calculated;

[0084] The relative displacement in the two-dimensional image is converted into a line-of-sight direction in a three-dimensional space, mapping of a three-dimensional line-of-sight vector is completed, and a final three-dimensional line-of-sight vector is obtained.

[0085] In some embodiments, the pupil center is detected according to the key feature set, and the pupil center of the target object is determined, specifically:

[0086] The pupil boundary of the target object is fitted through an ellipse formula, and then the pupil center is determined according to the fitted pupil boundary; the expression of this step is:

[0087]

[0088] Where (h, k) represents the coordinates of the center of the ellipse, also representing the pupil center; a and b represent the long axis and the short axis of the ellipse; theta represents the rotation angle; x and y represent the coordinates of the pixel point.

[0089] In some embodiments, the expression of the process of locating the corneal reflection point through clustering analysis of the infrared reflection point according to the pupil center is:

[0090]

[0091] Where x i represents the coordinates of the highlighted pixel in the image; C represents the center of the corneal reflection point.

[0092] Another aspect of the embodiment of the application also provides an attention dynamic evaluation system based on eye movement tracking of teenagers, comprising:

[0093] The first module is configured to acquire eye video data of a target object in response to a starting operation;

[0094] The second module is configured to generate a classification result of an attention state of the target object in response to an analysis operation according to the eye video data;

[0095] The third module is configured to generate a visualized attention heat map corresponding to the classification result and output a stage attention decay curve report in response to a visualization operation;

[0096] The step of generating the classification result of the attention state of the target object includes:

[0097] According to the eye video data, the relative displacement of the pupil center and the corneal reflection point of the target object is calculated by using an optical flow algorithm, and a three-dimensional line-of-sight vector is generated;

[0098] According to the three-dimensional line-of-sight vector, a spatio-temporal attention feature matrix is constructed, wherein the spatio-temporal attention feature matrix includes indexes in three dimensions of fixation duration, saccade speed and blink frequency.

[0099] Input the spatio-temporal attention feature matrix into a pre-trained attention neural network model to output a classification result of an attention state.

[0100] The specific implementation process of the present application will be described in detail below in conjunction with the accompanying drawings of the specification:

[0101] Reference Figure 3 Based on the attention dynamic evaluation method based on eye movement tracking of teenagers provided in the embodiments of the present application, an app product can be obtained, which can include a main interface 201, a video preview interface 201 of a target object, a start analysis button 203, and a video playback and result presentation interface 204 in specific applications, as shown in the figure. Figure 2

[0102] In the product interface shown in Figure 3 , the attention dynamic evaluation method based on eye movement tracking of teenagers provided in the embodiments of the present application can include the following steps:

[0103] S201, in response to a start operation, acquiring eye video data of a target object;

[0104] In step S201, based on the start operation, open the app operation interface as shown in Figure 3 , the eye video data of the target object can be acquired, and after the video data is acquired, the acquired video content can be viewed in the video preview interface 201. Figure 3

[0105] S202, in response to an analysis operation, generating a classification result of an attention state of the target object according to the eye video data;

[0106] In the above step S202, the embodiments can click the button 203 of “start analysis” in the interface shown in Figure 3 , and then perform background data analysis on the acquired eye video data to generate a classification result.

[0107] S203, in response to a visualization operation, generating a visualization attention heat map corresponding to the classification result and outputting a stage attention decay curve report;

[0108] In step S203 of the embodiments of the present application, after the fenxi result is obtained by clicking the analysis button, specifically as shown in Figure 4 , the result presentation interface 301 can be jumped to, in the result presentation interface, the user can view the visualization result interface of the related analysis result, and can also comment and like the result of this analysis in the comment area below the analysis result. ​​

[0109] In the step S201, the eye video data of the target object is acquired in response to the starting operation. The following methods can be used:

[0110] 1. In response to the starting operation, the camera of the target device is started, and the eye video data of the target object is collected by the camera.

[0111] 2. Alternatively, in response to the starting operation, the eye video data of the target object is loaded from the local storage file of the target device.

[0112] It can be understood that if the method of the present application is applied to a smart mobile terminal device such as a smart phone, the first method described above can be used to directly start the camera on site and collect the eye video data of the target object. The target object can be any person who wants to use the method for attention evaluation, such as a teenager.

[0113] In addition, the video file can also be read from the storage file of the smart terminal, and then the video file is loaded and analyzed, so that the user can use the method or software product of the present application to analyze and process for others.

[0114] The specific content of the step of generating the classification result of the attention state of the target object in the step S202 of the embodiment of the present application will be described in detail as follows. Figure 5 As shown in the figure, the step includes the following steps:

[0115] S2021, image pre-processing and feature detection are performed on each frame of image of the acquired eye video data, and a set of key features in the eye video data are detected.

[0116] S2022, the pupil center of the target object is determined by detecting the pupil center according to the set of key features.

[0117] S2023, the corneal reflection point is located by clustering analysis of the infrared reflection point according to the pupil center.

[0118] S2024, the motion vector between the pupil and the corneal reflection point in consecutive frames is determined according to the corneal reflection point determined in each frame of image of the eye video data.

[0119] S2025, the displacement of the pupil center relative to the corneal reflection point is calculated according to the motion vector.

[0120] S2026, the relative displacement in the two-dimensional image is converted into the line-of-sight direction in the three-dimensional space, the mapping of the three-dimensional line-of-sight vector is completed, and the final three-dimensional line-of-sight vector is obtained.

[0121] Specifically, in step S2021, the image preprocessing of the embodiments of the present application can include grayscale, Gaussian filter denoising. Grayscale converts color images to grayscale images, reduces the amount of calculation, while retaining key features. Gaussian filter is used to smooth the image, reduce noise interference, which helps the subsequent feature detection more accurate. The result of step S2021 is the processed grayscale image, which removes noise and provides a clear input for the subsequent steps.

[0122] In step S2021, the following formula 1 can be used for image grayscale processing:

[0123] Formula 1:

[0124] Where I(x, y) represents the brightness value of the grayscale image at coordinates (x, y); R, G, B represent the red, green, and blue channel values of the original image.

[0125] The following formula 2 can be used for Gaussian filter denoising processing:

[0126]

[0127] Where, represents the standard deviation of the Gaussian kernel (typical value σ = 1.5); represents the kernel radius (k = 3σ).

[0128] In the above step S2022, the embodiments of the present application use ellipse fitting to locate the boundary of the pupil. This step depends on the image processed in step S2021, because only in the denoised image, the boundary of the pupil will be clearer, and the ellipse fitting will be more accurate. If step S2021 does not effectively denoise, the edge of the pupil may be blurred, resulting in inaccurate ellipse fitting, which in turn affects the subsequent steps.

[0129] In this step S2022, the following formula 3 can be used to fit the pupil boundary:

[0130]

[0131] Where (h, k) represents the center coordinates of the ellipse, also represents the center of the pupil; a and b represent the major and minor axes of the ellipse; θ represents the rotation angle; x and y represent the coordinates of the pixel point.

[0132] In the above step S2023, the embodiment of the present application finds the center of corneal reflection by clustering analysis of the infrared reflection points. The corneal reflection points are bright spots generated on the corneal surface by the infrared light source, and the positions of these points are fixed relative to the eyeball. Step S2023 needs the result of step S2022, i.e. the position of the pupil center, because the positions of the corneal reflection points are usually related to the position of the pupil, and it can be necessary to adjust the range or parameters of the clustering analysis according to the position of the pupil to improve the accuracy of positioning.

[0133] Specifically, the above step S2023 can be implemented by using the following formula 4:

[0134]

[0135] wherein x i represents the coordinates of the highlighted pixels in the image; C represents the center of the corneal reflection points (which can usually include 4-5 points).

[0136] In the above step S2024, the embodiment of the present application uses improved optical flow calculation, including head motion compensation. The optical flow method is used to calculate the motion vectors of the pupil and the corneal reflection points between consecutive frames. Head motion compensation is introduced here, combined with the data of the IMU sensor. Step S2024 relies on the results of step S2022 and step S2023, i.e. the positions of the pupil center and the corneal reflection points, and also needs the data of the IMU to compensate for the displacement caused by head motion. The result of step S2024 is the motion vector of each feature point (pupil and corneal reflection point), and these vectors are used for the relative displacement calculation in step S2025.

[0137] Specifically, the above step S2024 can be implemented by using the following formula 5:

[0138]

[0139] wherein Vx, Vy represent the pixel motion speed (unit: pixel / frame); represents the weight of the smoothing term (the value can be 0.2-0.5); represents the head motion compensation coefficient (the value can be 0.1-0.3). In the embodiment of the present application, I(x, y) represents the brightness value of the gray-scale image at coordinates (x, y).

[0140] ​In step S2025, the displacement of the pupil center relative to the corneal reflection point is calculated. The displacement of the corneal reflection point can reflect the overall movement of the head, while the displacement of the pupil is the result of the eye movement and the head movement. By subtracting the displacement of the corneal reflection point, the influence of the head movement can be eliminated, and the displacement caused by the pure eye movement can be obtained. In step S2025, the motion vectors of each point obtained in step S2024 are used to perform difference calculation to obtain the relative displacement, which is the key input for generating the three-dimensional gaze vector.

[0141] Specifically, step S2025 can be implemented by using the following formula 6:

[0142] ;

[0143] wherein, represents the cumulative displacement of the pupil center in n frames; represents the average displacement of the corneal reflection point (as a reference for head movement); represents the displacement of the pupil center relative to the corneal reflection point.

[0144] In step S2026, the relative displacement in the two-dimensional image is converted into the gaze direction in the three-dimensional space. The intrinsic matrix and the calibration parameters of the camera are needed to map the two-dimensional displacement to the three-dimensional space. The result of step S2026 is the final three-dimensional gaze vector, which is used for subsequent attention evaluation.

[0145] Specifically, the process of three-dimensional space mapping in step S2026 can be implemented by using the following formula 7:

[0146] ;

[0147] wherein, represents the three-dimensional gaze vector (unit vector); represents the camera intrinsic matrix; represents the focal length (unit: pixel); represents the principal point coordinate; represents the pupil-corneal reflection point reference distance.

[0148] It should be noted that in step S2026, the gaze direction of the identified target object can also be corrected, and the formula for this process is: ;

[0149] wherein, represents the original calculated gaze direction angle; represents the head angular velocity measured by the gyroscope (unit: rad / s); represents the motion compensation coefficient (value: 0.05-0.15). represent the corrected gaze direction.

[0150] Then, the application constructs a spatio-temporal attention feature matrix according to the three-dimensional gaze vector, wherein the spatio-temporal attention feature matrix includes indicators of three dimensions of gaze duration, saccade speed and blink frequency.

[0151] It should be noted that the spatio-temporal attention feature matrix constructed by the embodiment of the application simultaneously focuses on indicators of three dimensions of gaze duration, saccade speed and blink frequency, can fully grasp the key feature information contained in the eye video data of the target object, and is helpful to improve the accuracy of attention evaluation and analysis.

[0152] In the process of constructing the spatio-temporal attention feature matrix, the following steps can be adopted:

[0153] Step 1: data preprocessing and coordinate normalization, normalizing the three-dimensional coordinate system;

[0154] Step 2: matrix construction and standardization, determining the feature weight of each feature by principal component analysis, and then constructing a spatio-temporal feature matrix based on the preset number of time windows and the number of feature channels. In this process, smoothing processing can also be used to smooth the feature data of the time dimension, realize exponential weighted moving average, and suppress transient noise.

[0155] Finally, the embodiment of the application inputs the constructed spatio-temporal attention feature matrix into the pre-trained attention neural network model, and outputs the classification result of the attention state.

[0156] It should be noted that the pre-trained attention neural network model can be, for example, a Transformer-XL model, which can support long-range attention decay analysis; or can be, for example, a PreFLMR model, which can support rapid deployment, etc.

[0157] The embodiment of the application mainly proposes an improved solution for how to generate a three-dimensional gaze vector according to eye video data and further construct a spatio-temporal attention feature matrix containing three dimensions, while the specific pre-trained attention neural network model can be selected according to actual scene requirements, and will not be described here.

[0158] To sum up, in the process of generating the classification result of the attention state of the target object according to the eye video data, the pupil center and the corneal reflection point of the target object can be calculated by using the optical flow algorithm to generate a three-dimensional gaze vector, and then a space-time attention feature matrix including indexes of three dimensions of fixation duration, saccade speed and blink frequency can be constructed according to the three-dimensional gaze vector, so that the accuracy of attention evaluation for the juvenile object can be improved, and the dynamic calibration capability is possessed.

[0159] It can be understood that the contents in the above method embodiments are all applicable to the present system embodiment, the present system embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0160] The present application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned attention dynamic evaluation method based on juvenile eye movement tracking when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0161] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiment, the present device embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0162] Please refer to Figure 6 , Figure 6 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0163] The processor 601 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided by the present application embodiment;

[0164] The memory 602 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 602 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 602 and are called and executed by the processor 601 to implement the attention dynamic evaluation method based on eye movement tracking of teenagers provided by the embodiments of the present application;

[0165] The input / output interface 603 is used to realize information input and output;

[0166] The communication interface 604 is used to realize the communication interaction between the device and other devices, and the communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.);

[0167] The bus 605 transmits information between various components (for example, the processor 601, the memory 602, the input / output interface 603, and the communication interface 604) of the device;

[0168] The processor 601, the memory 602, the input / output interface 603, and the communication interface 604 are connected to each other through the bus 605 to realize the communication connection between the device.

[0169] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the above-mentioned attention dynamic evaluation method based on eye movement tracking of teenagers.

[0170] It can be understood that the contents in the above-mentioned method embodiments are all applicable to the present storage medium embodiment, the function specifically realized by the present storage medium embodiment is the same as that of the above-mentioned method embodiments, and the beneficial effects achieved are also the same as those achieved by the above-mentioned method embodiments.

[0171] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0172] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user history data, and user location information, the user's permission or consent is obtained first, and the collection, use, and processing of such data comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or separate consent is obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the embodiments of the present application to normally operate is obtained.

[0173] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0174] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0175] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0176] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and device can be implemented as software, firmware, hardware and their appropriate combinations.

[0177] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0178] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0179] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0180] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0181] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for dynamic assessment of attention based on eye movement tracking of adolescents, characterized in that, The method comprises the following steps: in response to the starting operation, obtaining eye video data of a target object; in response to the analysis operation, generating a classification result of the attention state of the target object according to the eye video data; in response to the visualization operation, generating a visual attention heat map corresponding to the classification result and outputting a stage attention decay curve report; wherein the step of generating the classification result of the attention state of the target object comprises: according to the eye video data, calculating the relative displacement of the pupil center and the corneal reflection point of the target object by using an optical flow algorithm to generate a three-dimensional visual line vector; according to the three-dimensional visual line vector, constructing a spatio-temporal attention feature matrix, wherein the spatio-temporal attention feature matrix includes indicators in three dimensions of gaze duration, saccade speed and blink frequency; inputting the spatio-temporal attention feature matrix into a pre-trained attention neural network model to output a classification result of the attention state.

2. The attention dynamic evaluation method based on eye movement tracking of teenagers according to claim 1, characterized in that, The step of obtaining the eye video data of the target object in response to the starting operation comprises: in response to the starting operation, starting the camera of the target device, and collecting the eye video data of the target object through the camera; or, in response to the starting operation, loading the eye video data of the target object from the local storage file of the target device.

3. The attention dynamic evaluation method based on eye movement tracking of teenagers according to claim 2, characterized in that, The step of obtaining the eye video data of the target object in response to the starting operation further comprises: according to the obtained eye video data, presenting the eye video data in a first display area of a first interface to provide a preview picture or a playable video for the eye video data.

4. The attention dynamic evaluation method based on eye movement tracking of teenagers according to claim 1, characterized in that, The step of calculating the relative displacement of the pupil center and the corneal reflection point of the target object by using an optical flow algorithm to generate a three-dimensional visual line vector according to the eye video data comprises the following steps: image preprocessing and feature detection are performed on each frame of image of the obtained eye video data, and a key feature set in the eye video data is obtained; pupil center detection is performed according to the key feature set to determine the pupil center of the target object; the corneal reflection point is located by cluster analysis of infrared reflection points according to the pupil center; the motion vector between the pupil and the corneal reflection point in consecutive frames is determined according to the corneal reflection point determined in each frame of image of the eye video data; the displacement of the pupil center relative to the corneal reflection point is calculated; the relative displacement in the two-dimensional image is converted into the visual line direction in the three-dimensional space to complete the mapping of the three-dimensional visual line vector, and the final three-dimensional visual line vector is obtained.

5. The attention dynamic evaluation method based on eye movement tracking of teenagers according to claim 4, characterized in that, The step of performing pupil center detection according to the key feature set to determine the pupil center of the target object specifically comprises: the pupil boundary of the target object is fitted by using an ellipse formula, and then the pupil center is determined according to the fitted pupil boundary; the expression of this step is: ， wherein, (h, k) represents the coordinates of the center of the ellipse, also represents the center of the pupil; a and b represents the major axis and the minor axis of the ellipse; θ represents the rotation angle; x and y represents the coordinates of the pixel point.

6. The attention dynamic evaluation method based on eye movement tracking of teenagers according to claim 4, characterized in that, The expression of the process of locating the corneal reflection point by cluster analysis of infrared reflection points according to the pupil center is: ， wherein, x i representing the coordinates of the highlighted pixel in the image; C representing the center of the corneal reflection point.

7. A dynamic assessment system of attention based on eye movement tracking of adolescents, characterized by, The first module is configured to obtain the eye video data of the target object in response to the starting operation. ​ The second module is configured to generate a classification result of the attention state of the target object according to the eye video data in response to an analysis operation; The third module is configured to generate a visual attention heat map corresponding to the classification result and output a stage attention decay curve report in response to a visualization operation; The step of generating the classification result of the attention state of the target object includes: calculating relative displacement of a pupil center and a corneal reflection point of the target object by using an optical flow algorithm to generate a three-dimensional gaze vector according to the eye video data; constructing a spatiotemporal attention feature matrix according to the three-dimensional gaze vector, wherein the spatiotemporal attention feature matrix includes indexes of three dimensions of gaze duration, saccade velocity and blink frequency; inputting the spatiotemporal attention feature matrix into a pre-trained attention neural network model to output a classification result of the attention state.

8. An electronic device, comprising: The processor and the memory are included. The memory is configured to store a program. The processor executes the program to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images