2D screen fixation point prediction method based on state space

Abstract

The invention discloses a 2D screen fixation point prediction method based on a state space. The method is mainly characterized by comprising the following steps: (1) extracting local features of a face image by using a CNN network-based local feature extraction module to obtain an initial feature map; (2) inputting the initial feature map into an SSM-based global feature extraction module, performing global feature fusion by adopting SS2D (2D selective scanning) and FFN to obtain a final feature map, and obtaining a gaze direction through a gaze direction fusion regression module; and (3) determining a mapping matrix of a fixation point mapping module by using the user face image and fixation point coordinate data obtained in the calibration stage, and finally converting the fixation direction in the prediction stage into a fixation point on a 2D screen through the fixation point mapping module. According to the method, the degree of freedom of the user and the accuracy of fixation point prediction can be effectively improved.

Description

Technical Field

[0001] The present invention relates to a computer vision gaze point prediction technology, and in particular to a 2D screen gaze point prediction method based on state space. Background Art

[0002] With the rapid development of computers, smart phones and other devices, more and more people's work, entertainment, life, etc. are concentrated on 2D screens such as display screens and mobile phones. Therefore, the prediction of gaze points on 2D screens has gradually become an important direction in the field of computer vision. Research on 2D screen gaze point prediction technology can understand and predict the distribution of users' visual attention when watching screen content. This research has considerable value in many application scenarios such as improving user experience, personalized content recommendation, and human-computer interaction. For example, by predicting the user's gaze point, you can design an interface and layout that is more in line with the user's visual habits, thereby improving the user experience. 2D screen gaze point prediction is to locate the user's gaze position on the 2D screen through image processing technology and computer vision algorithms.

[0003] 2D screen gaze point prediction estimates and predicts people's gaze points on 2D screens through model-based methods or appearance-based methods. For example, some existing eye tracker devices use geometric eye models to predict gaze points. Based on appearance, CNN can be used to process eye images or face images to estimate the prediction point. It is also possible to use the transformer architecture and its global information capture capabilities to make more accurate gaze point predictions. Existing 2D screen gaze point prediction methods have failed to achieve a good balance in terms of prediction accuracy, cost, and user freedom. Therefore, it is necessary to develop more effective models to obtain more accurate gaze point prediction results at low cost and high user freedom. Summary of the invention

[0004] The purpose of the present invention is to provide a 2D screen gaze point prediction method based on state space. The method uses a CNN network to extract local features, and then fuses global information through an SSM global feature extraction module to predict the gaze direction. Finally, the gaze direction is mapped into a gaze point of the 2D screen through a gaze point mapping module.

[0005] The technical solution for achieving the purpose of the present invention is:

[0006] A 2D screen gaze point prediction method based on state space comprises the following steps:

[0007] (1) During the calibration phase, calibration points appear on the screen in sequence. The user looks at the calibration points in sequence. The network camera acquires the corresponding user image, records the coordinates of the calibration points and the facial image obtained by performing face detection on the user image.

[0008] (2) In the prediction stage, the user makes arbitrary gazes, the webcam obtains the user image, and the face image is obtained through face detection.

[0009] (3) Design a hybrid architecture model that can predict the gaze point of face images. The model includes a local feature extraction module based on the CNN network, a global feature extraction module based on the SSM network, a gaze direction fusion regression module, and a gaze point mapping module.

[0010] (4) The face image in the calibration stage in step (1) or the face image in the prediction stage in step (2) is input into the local feature extraction module in step (3), and the local features of the face image are extracted through the CNN network to obtain an initial feature map.

[0011] (5) The initial feature map obtained in step (4) is input into the SSM-based global feature extraction module in step (3), which uses SS2D (2D selective scanning) and FFN to perform global feature fusion to obtain the final feature map.

[0012] (6) The final feature map obtained in step (5) is input into the gaze direction fusion regression module to obtain the gaze direction.

[0013] (7) Using the gaze direction obtained in step (6) using the face image in the calibration phase as input and the gaze point coordinates recorded in step (1), a mapping matrix from gaze direction to gaze point in the gaze point mapping module is determined by the minimum regression method.

[0014] (8) The gaze direction obtained in step (6) using the face image in the prediction stage as input is input into the gaze point mapping module in step (7) in which the mapping matrix is ​​determined, thereby obtaining the final gaze point in the prediction stage.

[0015] Furthermore, in step (5), the initial feature map is input into the global feature extraction module in step (3), and SS2D (2D selective scanning) and FFN are used to perform global feature fusion to obtain the final feature map.

[0016] Furthermore, step (8) uses Euclidean distance as an evaluation index, where:

[0017] x(x1,x2), y(y1,y2) are the coordinates of the predicted gaze point and the actual gaze point respectively.

[0018] The Euclidean distance between two points d(x,y) is calculated as follows

[0019]

[0020] The Euclidean distance can directly reflect the gap between the predicted result and the actual result of the fixation point. The smaller the Euclidean distance between two points, the better the prediction result of the fixation point.

[0021] The 2D screen gaze point prediction method based on a network camera proposed in the present invention has the following advantages over the traditional gaze point prediction method based on visual saliency:

[0022] (1) The present invention uses a two-stage gaze point prediction method, that is, first predicting the gaze direction and then mapping the gaze direction to obtain the gaze point. This method effectively improves the user's freedom.

[0023] (2) The present invention designs a global feature extraction module based on SSM, which can effectively capture the global information in the feature map and improve the accuracy of gaze point prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of the 2D screen gaze point prediction method based on state space proposed by the present invention. DETAILED DESCRIPTION

[0025] The method of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0026] Step 1: Get the face image in the calibration phase and record the coordinates of the calibration points;

[0027] During the calibration phase, the user looks at the calibration points that appear on the screen in turn. The webcam acquires the corresponding user image and performs face detection to obtain the face image, while recording the coordinates of the calibration points.

[0028] Step 2: Obtain the face image in the prediction stage;

[0029] After the calibration phase is completed, the prediction phase is carried out. In this phase, the user makes an arbitrary gaze, the webcam obtains the user image, and the face image is obtained through face detection.

[0030] Step 3: State-space based fixation prediction model

[0031] In order to better extract and fuse the local and global features of facial images, the model includes a CNN-based local feature extraction module, an SSM-based global feature extraction module, a gaze direction fusion regression module, and a gaze point mapping module.

[0032] Step 4: Input the face image in the calibration stage in step 1 or the face image in the prediction stage in step 2 into the local feature extraction module in step 3, and extract the local features of the face image through the CNN network to obtain an initial feature map.

[0033] Step 5: The initial feature map obtained in step 4 is input into the SSM-based global feature extraction module in step 3, which uses SS2D (2D selective scanning) and FFN to perform global feature fusion to obtain the final feature map.

[0034] Step 6: Input the final feature map obtained in step 5 into the gaze direction fusion regression module to obtain the gaze direction.

[0035] Step 7: Use the gaze direction obtained in step 6 using the face image in the calibration phase as input and the gaze point coordinates recorded in step 1 to determine the mapping matrix from the gaze direction to the gaze point in the gaze point mapping module through the minimum regression method.

[0036] Step 8: The gaze direction obtained in step 6 using the face image in the prediction stage as input is input into the gaze point mapping module whose mapping matrix is ​​determined in step 7 to obtain the final gaze point in the prediction stage.

[0037] In order to test the prediction performance of step 8, the Euclidean distance is used as the evaluation index, where:

[0038] x(x1,x2), y(y1,y2) are the coordinates of the predicted gaze point and the actual gaze point respectively.

[0039] The Euclidean distance between two points d(x,y) is calculated as follows

[0040]

[0041] The Euclidean distance can directly reflect the gap between the predicted result and the actual result of the fixation point. The smaller the Euclidean distance between two points, the better the prediction result of the fixation point.

[0042] As an improvement to the 2D screen gaze point prediction model, we proposed a state-space based hybrid architecture model, in which local features are extracted on the face image through the CNN network, global features are extracted through the global feature extraction module of SSM, and finally the prediction direction is obtained through the fusion regression module and the final gaze point is obtained through the gaze point mapping module.

Claims

1. A 2D screen gaze point prediction method based on state space, characterized by: The following steps are involved: (1) During the calibration phase, calibration points appear on the screen in sequence. The user looks at the calibration points in sequence. The webcam obtains the corresponding user image, records the coordinates of the calibration points and the facial image obtained by performing face detection on the user image; (2) In the prediction stage, the user makes arbitrary gazes, the webcam obtains the user image, and the face image is obtained through face detection; (3) Design a hybrid architecture model that can predict gaze points in face images. The model includes a local feature extraction module based on CNN, a global feature extraction module based on SSM, a gaze direction fusion regression module, and a gaze point mapping module. (4) Inputting the face image in the calibration phase in step (1) or the face image in the prediction phase in step (2) into the local feature extraction module in step (3), extracting the local features of the face image through the CNN network to obtain an initial feature map; (5) The initial feature map obtained in step (4) is input into the SSM-based global feature extraction module in step (3), which uses SS2D (2D selective scanning) and FFN to perform global feature fusion to obtain the final feature map; (6) Input the final feature map obtained in step (5) into the gaze direction fusion regression module to obtain the gaze direction; (7) using the gaze direction obtained in step (6) using the face image in the calibration phase as input and the gaze point coordinates recorded in step (1) to determine the mapping matrix from the gaze direction to the gaze point in the gaze point mapping module through a minimum regression method; (8) The gaze direction obtained in step (6) using the face image in the prediction stage as input is input into the gaze point mapping module whose mapping matrix is ​​determined in step (7) to obtain the final gaze point in the prediction stage.

2. The state-space 2D screen gaze point prediction method according to claim 1, characterized in that: In step (5), the initial feature map is input into the global feature extraction module in step (3), and the SSM method is used to perform global feature fusion to obtain the final feature map.

3. The state-space 2D screen gaze point prediction method according to claim 1, characterized in that: Step (8) uses Euclidean distance as the evaluation index, where: are the coordinates of the predicted gaze point and the actual gaze point respectively. The Euclidean distance between the two points , the calculation formula is as follows The Euclidean distance can directly reflect the gap between the predicted result and the actual result of the fixation point. The smaller the Euclidean distance between two points, the better the prediction result of the fixation point.

Images