An eye movement feature map acquisition method and device, a terminal, and a storage medium

Abstract

The application relates to the technical field of eye movement data processing, in particular to an eye movement feature map acquisition method and device, a terminal and a storage medium. A time sequence matrix is constructed through each eye movement data sequence; an attention weight matrix is determined according to the time sequence matrix and an attention mechanism, so as to distinguish the importance of different parts in the time sequence matrix; an eye movement space-time graph is constructed according to the time sequence matrix, and feature extraction is performed on the eye movement space-time graph, so as to obtain a multi-dimensional space-time feature graph; finally, a weighted eye movement feature graph is obtained by combining the attention weight matrix and the space-time feature graph, so that more comprehensive and effective feature representations of eye movement data can be acquired.

Description

Technical Field

[0001] This application relates to the field of eye-tracking data processing technology, specifically to a method, apparatus, terminal, and storage medium for acquiring eye-tracking feature maps. Background Technology

[0002] The core focus of driver assistance and autonomous driving technologies is driving safety, and the driver's mental state is a crucial factor in ensuring driving safety. Currently, eye-tracking data of the driver is collected, and features are extracted and classified to determine whether the driver is fatigued. However, eye-tracking data contains rich temporal and spatial information, and feature extraction using only neural networks is insufficient to obtain a comprehensive and effective feature representation, resulting in low classification accuracy.

[0003] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0004] This application provides a method, apparatus, terminal, and storage medium for acquiring eye-tracking feature maps, in order to solve the problem in related technologies that eye-tracking data contains rich temporal and spatial information, and it is difficult to obtain a comprehensive and effective feature representation by simply extracting features from eye-tracking data through neural networks.

[0005] To achieve the above objectives, this application adopts the following technical solution: A method for obtaining eye-tracking feature maps includes the following steps: Acquire several eye-tracking data sequences, and determine a time series matrix based on each of the eye-tracking data sequences; The time series matrix is ​​processed based on an attention mechanism to obtain an attention weight matrix; An eye-tracking spatiotemporal map is constructed based on the time series matrix, and a spatiotemporal feature map is obtained by feature extraction from the eye-tracking spatiotemporal map. The eye-tracking spatiotemporal map is used to reflect the spatiotemporal relationship of each data point in the time series matrix. The eye-tracking feature map is determined based on the attention weight matrix and the spatiotemporal feature map.

[0006] Based on the above technical means, this application embodiment constructs a time series matrix through multiple eye-tracking data sequences, determines an attention weight matrix through the time series matrix and an attention mechanism to distinguish the importance of different parts in the time series matrix, constructs an eye-tracking spatiotemporal map through the time series matrix and extracts features from it to obtain a multi-dimensional spatiotemporal feature map, and finally combines the attention weight matrix and the spatiotemporal feature map to obtain a weighted eye-tracking feature map, thereby obtaining a more comprehensive and effective feature representation of the eye-tracking data.

[0007] Optionally, in one embodiment of this application, acquiring several eye-tracking data sequences includes: Data is acquired from several sensors, and different feature extraction algorithms are used to extract features from the data acquired from different sensors to obtain the eye-tracking data sequence; wherein the sensors are deployed at different locations.

[0008] Based on the above technical means, in this application embodiment, multiple sensors are deployed in different locations, and the data collected by each sensor are input into different feature extraction algorithms to obtain multiple categories of eye movement features.

[0009] Optionally, in one embodiment of this application, determining the time series matrix based on each of the eye-tracking data sequences includes: Each of the aforementioned eye-tracking data sequences is preprocessed, wherein the preprocessing includes at least one of the following processing methods: data denoising, data alignment, data interpolation, and data downsampling. The time series matrix is ​​determined based on the preprocessed eye-tracking data sequences.

[0010] Based on the above technical means, the embodiments of this application can improve the integrity, reliability and efficiency of data analysis by performing data denoising, data alignment and data interpolation on the original collected eye movement data sequence, so that the constructed time series matrix can accurately reflect the changes of different eye movement features over time.

[0011] Optionally, in one embodiment of this application, the step of processing the time series matrix based on the attention mechanism to obtain the attention weight matrix includes: The query matrix, key matrix, and value matrix of the attention mechanism are determined based on the time series matrix, and the attention weight matrix is ​​obtained through the query matrix, the key matrix, and the value matrix.

[0012] Based on the aforementioned technical means, this application embodiment calculates the query matrix, key matrix, and value matrix of a time series matrix using an attention mechanism, and then calculates the attention distribution of the time series matrix to obtain an attention weight matrix. The attention weight matrix can effectively distinguish the importance of different parts of the time series matrix.

[0013] Optionally, in one embodiment of this application, constructing the eye-tracking spatiotemporal map based on the time series matrix includes: Graph nodes are constructed based on each data point in the time series matrix; Based on the spatiotemporal relationship of each data point in the time series matrix, construct the edges between each graph node; The eye-tracking spatiotemporal graph is obtained based on each of the graph nodes and the edges between them.

[0014] Based on the above technical means, the embodiments of this application construct graph nodes through data points in the time series matrix, and construct edges between graph nodes through the spatiotemporal relationships between data points, thereby obtaining an eye-tracking spatiotemporal graph that can reflect the spatiotemporal relationships of each data point. The eye-tracking spatiotemporal graph can be used to analyze the temporal and / or spatial proximity of each data point in the time series matrix.

[0015] Optionally, in one embodiment of this application, the step of extracting features from the eye-tracking spatiotemporal map to obtain a spatiotemporal feature map includes: The eye-tracking spatiotemporal map is input into a spatiotemporal convolution model to obtain the spatiotemporal feature map.

[0016] Based on the above technical means, the embodiments of this application use spatiotemporal convolution to perform convolution operations on the constructed eye-tracking spatiotemporal map, which can perform feature extraction and learning in both time and space dimensions.

[0017] Optionally, in one embodiment of this application, determining the eye-tracking feature map based on the attention weight matrix and the spatiotemporal feature map includes: The eye-tracking feature map is obtained by performing a dot product operation on the attention weight matrix and the spatiotemporal feature map.

[0018] Based on the above technical means, the embodiments of this application fuse the attention weight matrix and the spatiotemporal feature map by dot product calculation to obtain a weighted eye movement feature map, which can obtain a more comprehensive and effective feature representation of eye movement data.

[0019] A second aspect of this application provides an eye-tracking feature map acquisition device, comprising: The acquisition module is used to acquire several eye-tracking data sequences and determine a time series matrix based on each of the eye-tracking data sequences. The attention module is used to process the time series matrix based on the attention mechanism to obtain the attention weight matrix; The graph analysis module is used to construct an eye-tracking spatiotemporal graph based on the time series matrix, and to extract features from the eye-tracking spatiotemporal graph to obtain a spatiotemporal feature graph, wherein the eye-tracking spatiotemporal graph is used to reflect the spatiotemporal relationship of each data point in the time series matrix; The fusion module is used to determine the eye-tracking feature map based on the attention weight matrix and the spatiotemporal feature map.

[0020] A third aspect of this application provides a terminal device, the terminal device including a memory, a processor, and an eye-tracking feature map acquisition program stored in the memory and executable on the processor. When the processor executes the eye-tracking feature map acquisition program, it implements the steps of the eye-tracking feature map acquisition method as described in any of the preceding claims.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing an eye-tracking feature map acquisition program, which, when executed by a processor, implements the steps of the eye-tracking feature map acquisition method as described in any of the preceding claims.

[0022] The beneficial effects of this application are: This application transforms multiple eye-tracking data sequences into a time-series matrix, determines an attention weight matrix using the time-series matrix and an attention mechanism to distinguish the importance of different parts of the time-series matrix, constructs an eye-tracking spatiotemporal map using the time-series matrix and extracts features from it to obtain a multi-dimensional spatiotemporal feature map, and finally merges the attention weight matrix with the spatiotemporal feature map to obtain a weighted eye-tracking feature map, thereby obtaining a more comprehensive and effective feature representation of the eye-tracking data.

[0023] This application obtains multi-category eye-movement features by deploying multiple sensors at different locations and inputting the data collected by each sensor into different feature extraction algorithms. Furthermore, by performing data denoising, data alignment, and data interpolation on the original acquired eye-movement data sequences, the integrity, reliability, and efficiency of data analysis can be improved, enabling the constructed time-series matrix to accurately reflect the changes of different eye-movement features over time.

[0024] This application uses the nearest neighbor technique to construct an eye-tracking spatiotemporal graph corresponding to the time series matrix. The graph nodes represent each data point in the time series matrix, and the edges represent the proximity of each data point in time and / or space, thereby better analyzing the spatiotemporal relationship of each data point in the time series matrix.

[0025] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be expressed in part through the description itself. This application has been put into practice. Attached Figure Description

[0026] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the method for obtaining eye-tracking feature maps according to an embodiment of this application; Figure 2This is a schematic diagram showing the location of the embedded points of the sensor in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the eye movement feature map acquisition device according to an embodiment of this application; Figure 4 This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0028] The following description, with reference to the accompanying drawings, illustrates a method, apparatus, terminal, and storage medium for acquiring eye-tracking feature maps according to embodiments of this application. Addressing the problem mentioned in the background art that eye-tracking data contains rich temporal and spatial information, and that feature extraction from eye-tracking data using only neural networks is insufficient to obtain a comprehensive and effective feature representation, this application provides a method for acquiring eye-tracking feature maps. In this method, several eye-tracking data sequences are acquired, and a time-series matrix is ​​determined based on each of the eye-tracking data sequences. The time-series matrix is ​​processed based on an attention mechanism to obtain an attention weight matrix. An eye-tracking spatiotemporal map is constructed based on the time-series matrix, and features are extracted from the eye-tracking spatiotemporal map to obtain a spatiotemporal feature map. The eye-tracking spatiotemporal map reflects the spatiotemporal relationship between data points in the time-series matrix. The eye-tracking feature map is determined based on the attention weight matrix and the spatiotemporal feature map. This invention transforms multiple eye-tracking data sequences into a time-series matrix. An attention weight matrix is ​​determined using the time-series matrix and an attention mechanism to distinguish the importance of different parts of the time-series matrix. An eye-tracking spatiotemporal map is constructed using the time-series matrix, and features are extracted from it to obtain a multi-dimensional spatiotemporal feature map. Finally, the attention weight matrix is ​​fused with the spatiotemporal feature map to obtain a weighted eye-tracking feature map, thus achieving a more comprehensive and effective feature representation of the eye-tracking data.

[0029] Specifically, Figure 1 This is a flowchart illustrating a method for obtaining eye-tracking feature maps provided in an embodiment of this application.

[0030] like Figure 1 As shown, the method for obtaining this eye-tracking feature map includes the following steps: Step S100: Obtain several eye-tracking data sequences and determine a time series matrix based on each of the eye-tracking data sequences.

[0031] Specifically, this embodiment first requires acquiring several eye movement data sequences of the subject, who can be the driver or other personnel closely related to driving safety. Each eye movement data sequence reflects the changes in a type of eye movement characteristic of the subject over time, such as changes in eye position / fixation point / sagging path. To better process and analyze each eye movement data sequence, this embodiment converts each eye movement data sequence into a time series matrix.

[0032] For example, a time series matrix can be viewed as a three-dimensional matrix of time series, where each row represents a time step and each column represents a different eye movement feature. In other words, a time series matrix can reflect various eye movement features at each time step.

[0033] In one embodiment, acquiring a plurality of eye-tracking data sequences includes: Data is acquired from several sensors, and different feature extraction algorithms are used to extract features from the data acquired from different sensors to obtain the eye-tracking data sequence; wherein the sensors are deployed at different locations.

[0034] Specifically, unlike previous head-mounted eye-tracking devices, this embodiment uses fixed sensors to acquire the eye-tracking data sequence of the observed subject. Each sensor is deployed in a different location, and the data collected by each sensor is input into a different feature extraction algorithm to obtain multiple eye-tracking data sequences. Each eye-tracking data sequence is used to reflect the changes of different categories of eye-tracking features over time.

[0035] For example, sensors or miniature eye-tracking devices could be embedded in the left and right A-pillars of a car to acquire eye-tracking data sequences of the driver's observations of the left and right rearview mirrors; similarly, sensors could be embedded in the main instrument panel to acquire eye-tracking data sequences of the driver's observations of the main instrument panel. Figure 2 As shown, each circle represents the location of the sensor's mounting point. Sensors at mounting points 1 and 2 primarily detect the subject's glancing data and sensitivity to the left and right rearview mirrors; sensor data at mounting point 3 primarily detects the subject's attention to the dashboard, such as sensitivity to speed, fatigue level, and whether there are any abnormal driving conditions.

[0036] In one embodiment, determining the time series matrix based on each of the eye-tracking data sequences includes: Each of the aforementioned eye-tracking data sequences is preprocessed, wherein the preprocessing includes at least one of the following processing methods: data denoising, data alignment, data interpolation, and data downsampling. The time series matrix is ​​determined based on the preprocessed eye-tracking data sequences.

[0037] The raw data is characterized by high noise and high dimensionality, making direct processing computationally expensive. Therefore, this embodiment requires preprocessing the raw eye-tracking data sequences, including denoising, calibration, alignment, interpolation, and downsampling. Specifically, denoising can be achieved using mean filtering to remove noise. Data alignment involves synchronizing different eye-tracking data sequences temporally to extract a complete time series. Then, data from different time points / time periods within this time series are aligned and synchronized, followed by interpolation. Given the continuity of eye-tracking data, this embodiment uses fitting interpolation. Downsampling can be performed using maximum or average downsampling. Due to the rapid eye movement and jumping speeds and poor robustness, this embodiment uses average downsampling. Finally, a time series matrix is ​​constructed from the preprocessed eye-tracking data sequences, accurately reflecting the changes in various eye-tracking features over time.

[0038] Step S200: Process the time series matrix based on the attention mechanism to obtain the attention weight matrix.

[0039] Specifically, in order to extract more effective feature data, this embodiment uses an attention mechanism to calculate the attention distribution of the time series matrix and obtain an attention weight matrix to distinguish the importance of different parts of the time series matrix.

[0040] In one embodiment, processing the time series matrix based on the attention mechanism to obtain the attention weight matrix includes: The query matrix, key matrix, and value matrix of the attention mechanism are determined based on the time series matrix, and the attention weight matrix is ​​obtained through the query matrix, the key matrix, and the value matrix.

[0041] Specifically, the attention mechanism consists of three matrices: the query matrix (Q), the key matrix (K), and the value matrix (V). First, the query matrix, key matrix, and value matrix are calculated based on the time series matrix. Then, the attention distribution of the time series matrix is ​​calculated based on these three matrices to obtain the attention weight matrix, which distinguishes the attention levels of different parts of the time series matrix.

[0042] In one embodiment, the computational method for the attention mechanism includes: Multiply the time series matrix by the Q-weight matrix, K-weight matrix, and V-weight matrix respectively to obtain the query matrix, key matrix, and value matrix; The time series matrix is ​​divided into several local blocks based on the query matrix, key matrix, and value matrix, and the attention weights corresponding to each local block are obtained. The attention weights of each local block are concatenated, and the attention weight matrix is ​​determined based on the concatenation result.

[0043] Specifically, the time series matrix requiring attention calculation is divided into multiple time steps, each corresponding to a position or time point in the time series matrix. The attention mechanism consists of three parts: a Q-weight matrix, a K-weight matrix, and a V-weight matrix. These three weight matrices are multiplied by the input data to obtain the query matrix (Q matrix), the key matrix (K matrix), and the value matrix (V matrix). For each time step of the time series matrix, the query matrix reflects the feature vector of that time step, the key matrix reflects the correlation between that time step and all other time steps except itself, and the value matrix reflects the values ​​of the feature vectors of all other time steps. The Q matrix is ​​generated from the input of the current time step and can be used to determine the parts of the time series matrix that require attention. The K matrix is ​​generated from the input of each time step in the time series matrix and is used to describe the position in the time series matrix, thereby calculating the correlation between the current time step and other time steps (i.e., the correlation between different positions in the time series matrix). The V matrix contains information from the time series matrix and corresponds to the key vector; in the attention mechanism, it is used to calculate a weighted average to capture relevant information from the time series matrix. The time series matrix is ​​split into several local blocks guided by the Q, K, and V matrices. For each local block, an attention score is calculated to obtain the attention weight for that local block. Finally, the calculation results of all local blocks are concatenated (or weighted summed) to obtain the attention weight matrix used to adjust the feature weights.

[0044] For example, assuming the time series matrix is ​​T, T can be viewed as a three-dimensional tensor, where the three dimensions represent eye position, fixation point, and saccade path. Inputting the time series matrix into an attention model, and performing a linear transformation on the time series matrix within the attention model, yields... , , ,in, , , The Q-weight matrix, K-weight matrix, and V-weight matrix are randomly initialized. Each eye-tracking feature (i.e., a class of eye-tracking features within a time series) in the time series matrix is ​​defined as a time step. Q corresponds to the feature vector at the current time step; K corresponds to other time steps besides the current time step, representing the correlation between the feature vectors at other time steps and the feature vector of Q; V is the value of the feature vector corresponding to K, used to calculate the weighted average. Based on Q, K, and V, the time series matrix is ​​divided into several local blocks, and attention weights are calculated for each local block. Finally, all attention weights are concatenated, and the final output attention weight matrix Y is obtained based on the concatenation result.

[0045] In one embodiment, concatenating the attention weights of each local block and determining the attention weight matrix based on the concatenation result includes: The attention weight matrix is ​​obtained by multiplying the concatenated attention weights by the time series matrix.

[0046] Specifically, the attention weight matrix is ​​obtained by concatenating the attention weights of all local blocks and then multiplying them by the original time series matrix.

[0047] For example, the formula for calculating the attention mechanism is as follows: ; ; ; Among them, block i Let represent the i-th local block; b represents the total number of local blocks; Describes the dimension of K. , , The learned weight matrix, express function, This indicates a pooling operation. This indicates a splicing operation. Y represents the attention weight of the i-th local block; Y represents the attention weight matrix.

[0048] Step S300: Construct an eye-tracking spatiotemporal map based on the time series matrix, and extract features from the eye-tracking spatiotemporal map to obtain a spatiotemporal feature map, wherein the eye-tracking spatiotemporal map is used to reflect the spatiotemporal relationship of each data point in the time series matrix.

[0049] Specifically, in this embodiment, an eye-tracking spatiotemporal map is constructed using a time-series matrix. This map reflects the spatiotemporal relationships of data points within the time-series matrix. Then, feature extraction is performed on the eye-tracking spatiotemporal map to obtain a spatiotemporal feature map, thereby extracting the effective temporal and / or spatial features of each data point in the time-series matrix.

[0050] In one embodiment, constructing the eye-tracking spatiotemporal map based on the time series matrix includes: Graph nodes are constructed based on each data point in the time series matrix; Based on the spatiotemporal relationship of each data point in the time series matrix, construct the edges between each graph node; The eye-tracking spatiotemporal graph is obtained based on each of the graph nodes and the edges between them.

[0051] Nearest neighbor (NN) technology is a method used to determine the distance or similarity between data points, typically used to calculate the temporal relationship between each data point, i.e., the degree of proximity of each data point in time or space. This embodiment uses NN technology to construct an eye-tracking spatiotemporal graph corresponding to a time series matrix: a graph node is generated for each data point in the time series matrix; taking two data points as an example, the spatiotemporal relationship between the two data points is used to determine whether an edge exists between the graph nodes corresponding to these two data points. Finally, the eye-tracking spatiotemporal graph is obtained based on the constructed graph nodes and edges.

[0052] In one implementation, the eye-tracking spatiotemporal graph is an adjacency matrix, where the rows and columns correspond to graph nodes, and the elements of the adjacency matrix indicate whether there are edges (connections) between graph nodes. For example, for eye-tracking data, a binary adjacency matrix is ​​typically used, where 1 indicates a connection between two nodes, and 0 indicates no connection.

[0053] In one embodiment, the step of extracting features from the eye-tracking spatiotemporal map to obtain a spatiotemporal feature map includes: The eye-tracking spatiotemporal map is input into a spatiotemporal convolution model to obtain the spatiotemporal feature map.

[0054] Specifically, spatiotemporal convolution is used to perform convolution operations on the constructed eye-tracking spatiotemporal map, which allows for feature extraction and learning in both temporal and spatial dimensions.

[0055] For example, the calculation formula for the spatiotemporal convolution model is shown below: ; Where X represents the input eye-tracking spatiotemporal map; The output spatiotemporal feature map is represented by T; the temporal length is represented by k; the spatial size of the convolution kernel is represented by W; and the convolution kernel is represented by W, which can be obtained through the model's learning process.

[0056] Step S400: Determine the eye-tracking feature map based on the attention weight matrix and the spatiotemporal feature map.

[0057] Specifically, this embodiment combines the attention weight matrix with the spatiotemporal feature map to obtain a weighted eye-tracking feature map. Since the eye-tracking feature map integrates both attention weights and spatiotemporal features, it can reflect a more comprehensive and effective feature representation of eye-tracking data.

[0058] In one embodiment, determining the eye-tracking feature map based on the attention weight matrix and the spatiotemporal feature map includes: The eye-tracking feature map is obtained by performing a dot product operation on the attention weight matrix and the spatiotemporal feature map.

[0059] Specifically, this embodiment fuses the attention weight matrix and the spatiotemporal feature map through dot product calculation to obtain a weighted eye-tracking feature map. The attention mechanism adaptively assigns different weights to different neighboring nodes (i.e., adjacent graph nodes) to focus on the most relevant graph nodes. The eye-tracking feature map obtained after the dot product operation fuses eye-tracking features and attention weights from different time steps, which can reflect a more comprehensive and effective feature representation of eye-tracking data.

[0060] For example, this embodiment pre-defines a feature fusion module that combines the attention weight matrix and spatiotemporal features. Figure 1 The input feature fusion module is used. This module performs a dot product operation on the attention weight matrix and the spatiotemporal feature map to obtain a weighted eye-tracking feature map. The calculation formula for the feature fusion module is as follows: ; in, Y new It is an eye movement feature map.

[0061] In one embodiment, the method further includes: The eye-tracking feature map is input into a preset target prediction model to obtain prediction information corresponding to each eye-tracking data sequence.

[0062] Specifically, eye-tracking feature maps can be input into target prediction models, such as the ST-GCN model, to achieve model training tasks or recognition / prediction tasks for various eye-tracking data sequences, such as driver fatigue prediction or region of interest prediction.

[0063] In one embodiment, inputting the eye-tracking feature map into a preset target prediction model includes: Based on the input data format of the target prediction model, the eye-tracking feature map is normalized and standardized. The normalized and standardized eye-tracking feature map is input into the target prediction model.

[0064] Specifically, before inputting the eye-tracking feature map into the target prediction model, it needs to be normalized and standardized to ensure that the value range and scale of the eye-tracking feature map are within a reasonable range.

[0065] In summary, the advantages of this application are: (1) This application transforms multiple eye-tracking data sequences into a time series matrix, determines the attention weight matrix through the time series matrix and the attention mechanism to distinguish the importance of different parts in the time series matrix; constructs an eye-tracking spatiotemporal map through the time series matrix and extracts features from it to obtain a multi-dimensional spatiotemporal feature map; finally, the attention weight matrix and the spatiotemporal feature map are fused to obtain a weighted eye-tracking feature map, thereby obtaining a more comprehensive and effective feature representation of the eye-tracking data.

[0066] (2) This application obtains multiple categories of eye movement features by deploying multiple sensors in different locations and inputting the collected data from each sensor into different feature extraction algorithms. By performing data denoising, data alignment, and data interpolation on the original collected eye movement data sequence, the integrity, reliability, and efficiency of data analysis can be improved, so that the constructed time series matrix accurately reflects the changes of different eye movement features over time.

[0067] (3) This application uses the nearest neighbor technique to construct the eye-tracking spatiotemporal graph corresponding to the time series matrix. The graph nodes represent each data point in the time series matrix, and the edges represent the proximity of each data point in time and / or space, so as to better analyze the spatiotemporal relationship of each data point in the time series matrix.

[0068] Next, with reference to the accompanying drawings, an eye movement feature map acquisition device according to an embodiment of this application is described.

[0069] like Figure 3 As shown, the eye-tracking feature map acquisition device 10 includes: an acquisition module 100, an attention module 200, a graph analysis module 300, and a fusion module 400.

[0070] Specifically, the acquisition module 100 is used to acquire several eye-tracking data sequences and determine a time series matrix based on each of the eye-tracking data sequences; Attention module 200 is used to process the time series matrix based on the attention mechanism to obtain an attention weight matrix; The graph analysis module 300 is used to construct an eye-tracking spatiotemporal graph based on the time series matrix, and to extract features from the eye-tracking spatiotemporal graph to obtain a spatiotemporal feature map, wherein the eye-tracking spatiotemporal graph is used to reflect the spatiotemporal relationship of each data point in the time series matrix; The fusion module 400 is used to determine the eye-tracking feature map based on the attention weight matrix and the spatiotemporal feature map.

[0071] It should be noted that the foregoing explanation of the method for obtaining eye movement feature maps also applies to the eye movement feature map acquisition device of this embodiment, and will not be repeated here.

[0072] Figure 4A schematic diagram of the structure of a terminal device provided in an embodiment of this application. The terminal device may include: The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0073] When the processor 402 executes the program, it implements the eye-tracking feature map acquisition method provided in the above embodiments.

[0074] Furthermore, the terminal equipment also includes: Communication interface 403 is used for communication between memory 401 and processor 402.

[0075] The memory 401 is used to store computer programs that can run on the processor 402.

[0076] The memory 401 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0077] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0078] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0079] Processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0080] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for obtaining eye-tracking feature maps.

[0081] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0082] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0083] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0084] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can read and execute instructions from or in conjunction with such an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically by optically scanning paper or other media, then editing, interpreting or otherwise processing them as necessary, and then storing them in computer memory.

[0085] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0086] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0088] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for obtaining eye-tracking feature maps, characterized in that, Includes the following steps: Acquire several eye-tracking data sequences, and determine a time series matrix based on each of the eye-tracking data sequences; The time series matrix is ​​processed based on an attention mechanism to obtain an attention weight matrix, including: multiplying the time series matrix by the Q-weight matrix, K-weight matrix, and V-weight matrix in the attention mechanism to obtain a query matrix, a key matrix, and a value matrix; splitting the time series matrix into several local blocks according to the query matrix, the key matrix, and the value matrix, and obtaining the attention weight corresponding to each local block; concatenating the attention weights of each local block, and determining the attention weight matrix based on the concatenation result. Constructing an eye-tracking spatiotemporal graph based on the time series matrix includes: constructing graph nodes based on each data point in the time series matrix; constructing edges between the graph nodes based on the spatiotemporal relationships of the data points in the time series matrix; and obtaining the eye-tracking spatiotemporal graph based on the graph nodes and the edges between the graph nodes. Feature extraction is performed on the eye-tracking spatiotemporal map to obtain a spatiotemporal feature map, wherein the eye-tracking spatiotemporal map is used to reflect the spatiotemporal relationship of each data point in the time series matrix; The eye-tracking feature map is determined based on the attention weight matrix and the spatiotemporal feature map.

2. The method for obtaining eye-tracking feature maps as described in claim 1, characterized in that, The acquisition of several eye-tracking data sequences includes: Data is acquired from several sensors, and different feature extraction algorithms are used to extract features from the data acquired from different sensors to obtain the eye-tracking data sequence; wherein the sensors are deployed at different locations.

3. The method for obtaining eye-tracking feature maps as described in claim 1, characterized in that, The step of determining the time series matrix based on each of the eye-tracking data sequences includes: Each of the aforementioned eye-tracking data sequences is preprocessed, wherein the preprocessing includes at least one of the following processing methods: data denoising, data alignment, data interpolation, and data downsampling. The time series matrix is ​​determined based on the preprocessed eye-tracking data sequences.

4. The method for obtaining eye-tracking feature maps as described in claim 1, characterized in that, The step of extracting features from the eye-tracking spatiotemporal map to obtain a spatiotemporal feature map includes: The eye-tracking spatiotemporal map is input into a spatiotemporal convolution model to obtain the spatiotemporal feature map.

5. The method for obtaining eye-tracking feature maps as described in claim 1, characterized in that, The step of determining the eye-tracking feature map based on the attention weight matrix and the spatiotemporal feature map includes: The eye-tracking feature map is obtained by performing a dot product operation on the attention weight matrix and the spatiotemporal feature map.

6. A device for acquiring eye-tracking feature maps, characterized in that, include: The acquisition module is used to acquire several eye-tracking data sequences and determine a time series matrix based on each of the eye-tracking data sequences. An attention module is used to process the time series matrix based on an attention mechanism to obtain an attention weight matrix. This includes: multiplying the time series matrix by the Q-weight matrix, K-weight matrix, and V-weight matrix in the attention mechanism to obtain a query matrix, a key matrix, and a value matrix; splitting the time series matrix into several local blocks based on the query matrix, the key matrix, and the value matrix, and obtaining the attention weight corresponding to each local block; concatenating the attention weights of each local block, and determining the attention weight matrix based on the concatenation result. The graph analysis module is used to construct an eye-tracking spatiotemporal graph based on the time series matrix, including: constructing graph nodes according to each data point in the time series matrix; constructing edges between the graph nodes according to the spatiotemporal relationships of the data points in the time series matrix; and obtaining the eye-tracking spatiotemporal graph based on the graph nodes and the edges between the graph nodes. Feature extraction is performed on the eye-tracking spatiotemporal map to obtain a spatiotemporal feature map, wherein the eye-tracking spatiotemporal map is used to reflect the spatiotemporal relationship of each data point in the time series matrix; The fusion module is used to determine the eye-tracking feature map based on the attention weight matrix and the spatiotemporal feature map.

7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and an eye-tracking feature map acquisition program stored in the memory and executable on the processor. When the processor executes the eye-tracking feature map acquisition program, it implements the steps of the eye-tracking feature map acquisition method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an eye-tracking feature map acquisition program, which, when executed by a processor, implements the steps of the eye-tracking feature map acquisition method as described in any one of claims 1-5.

Images