A first-view gaze point prediction method based on attention shift

Abstract

The application provides a first-view gaze point prediction method based on attention shift, comprising: extracting an optical flow image; constructing a first-view gaze point prediction model based on attention shift; inputting the optical flow image into the first-view gaze point prediction model to obtain spatial features and time features; obtaining an initial visual saliency image and an attention image based on the spatial features and the time features; and fusing the initial visual saliency image and the attention image to generate a final gaze point prediction image. The application extracts spatial and temporal features of an original image at multiple scales, fully utilizes time sequence information and high-level semantic information for saliency detection, predicts subsequent attention from previous gaze by modeling attention shift, and finally combines a visual saliency model to fuse into a final gaze point prediction image, thereby improving the accuracy of gaze point prediction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision gaze point prediction, and particularly relates to a first-view gaze point prediction method based on attention shift. BACKGROUND

[0002] With the rapid development of motion cameras and other devices, more and more self-centered first-view videos are produced, which makes first-view gaze point prediction an emerging direction in the field of computer vision. The first-view video is a video that observes and shoots objective things from the angle of what the photographer sees with his own eyes. Studying the gaze area in the first-view has a significant role in observing human attention and understanding human activity intention. At the same time, this research has considerable value in virtual reality (VR), augmented reality (AR) and other application scenarios. For example, the current VR faces the problem of high hardware requirements for full-scene fine rendering. If the gaze area of a person can be predicted and local fine rendering is performed, the hardware cost can be greatly reduced. First-view gaze point prediction is to locate the gaze area of the human eye in the first-person video through image processing technology and computer vision algorithm.

[0003] First-view gaze point prediction estimates and predicts the area that people may gaze at in an image or video. For example, by using implicit cues in the behavior of the camera wearer, first-view gaze point prediction is performed. A deep model and an adversarial network are used to predict the self-centered gaze position in future video frames. Time attention shift is incorporated into a saliency-based gaze point prediction model. A multi-modal network can also be used to use video and inertial measurement unit data to make more accurate self-centered gaze point prediction. The transformer-based architecture can also solve the problem of gaze point prediction. The existing first-view gaze point prediction method still has certain room for improvement in prediction accuracy. Therefore, it is necessary to develop a more effective model to obtain more accurate gaze point prediction results. SUMMARY

[0004] To solve the above technical problems, the application provides a first-view gaze point prediction method based on attention shift, which uses a dual-stream I3D network to extract time features and multi-scale spatial features, generates a visual saliency image and an attention image through a saliency detection module and an attention shift module respectively, and finally generates gaze point prediction for an image or a video by fusion, to predict the human gaze area and improve the ability of further visual information processing.

[0005] The application provides a first-view gaze point prediction method based on attention shift, which comprises:

[0006] extracting an optical flow image;

[0007] Construct a first-view gaze point prediction model based on attention shift;

[0008] Input the optical flow image into the first-view gaze point prediction model, obtain spatial features and temporal features, obtain an initial visual saliency image and an attention image based on the spatial features and the temporal features, and generate a final gaze point prediction image based on the initial visual saliency image and the attention image.

[0009] Optionally, the first-view gaze point prediction model comprises a feature encoding module, a saliency prediction module, an attention shift module, and a late fusion module; wherein the feature encoding module is a feature encoding module introducing an Inflated 3D network.

[0010] The feature encoding module and the late fusion module are connected through the parallel saliency prediction module and the gaze state prediction module.

[0011] Optionally, obtaining the spatial features and the temporal features comprises:

[0012] Input the optical flow image into the feature encoding module, and extract the spatial features and the temporal features using the Inflated 3D network respectively.

[0013] Optionally, obtaining the initial visual saliency image comprises:

[0014] Input the spatial features and the temporal features into the saliency prediction module, perform feature fusion by element-by-element summation to obtain the initial visual saliency image.

[0015] Optionally, the attention shift module comprises a channel weight extractor, a gaze state predictor, and an LSTM-based weight predictor.

[0016] The weight predictor is connected with the channel weight extractor and the gaze state predictor respectively.

[0017] Optionally, obtaining the attention image comprises:

[0018] Input the maximum value coordinates of a preset image and the spatial features into the channel weight extractor to obtain a first channel weight; wherein the preset image is the gaze point prediction image predicted last time, and the first channel weight is a feature representation channel weight around the gaze point last time.

[0019] Input the temporal features predicted last time into the gaze state predictor to obtain a gaze state probability.

[0020] input the first channel weight into the weight predictor to obtain a second channel weight; wherein the second channel weight is a channel weight representing an attention region at a current time;

[0021] obtain a third channel weight based on the second channel weight and the gaze state probability; wherein the third channel weight is a final channel weight at the current time;

[0022] weight the third channel weight and the spatial feature to obtain the attention image.

[0023] Optionally, generating a final gaze point prediction image comprises:

[0024] input the initial visual saliency image and the attention image into the late fusion module to obtain the final gaze point prediction image; and record the coordinates of the maximum value in the gaze point prediction image for next use.

[0025] Optionally, the prediction method further comprises verifying the gaze point prediction image using Area Under Cuve.

[0026] Compared with the prior art, the present application has the following advantages and technical effects:

[0027] The present application can fully extract visual saliency features in images or videos by using a double-flow I3D network for multi-scale feature extraction.

[0028] The present application designs an attention shift module, which can model attention shift in human task processes, and combines with bottom-up saliency to improve the accuracy of gaze point prediction. BRIEF DESCRIPTION OF DRAWINGS

[0029] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of the present application illustrated in the drawings and their descriptions are used to explain the present application and are not intended to limit the present application. In the drawings:

[0030] Figure 1 A first perspective gaze point prediction method flowchart of the embodiments of the present application. DETAILED DESCRIPTION

[0031] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0032] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0033] The present application provides a first perspective gaze point prediction method based on attention shift, comprising:

[0034] extracting an optical flow image;

[0035] constructing a first perspective gaze point prediction model based on attention shift;

[0036] inputting the optical flow image into the first perspective gaze point prediction model to obtain spatial features and temporal features, obtaining an initial visual saliency image and an attention image based on the spatial features and the temporal features, and generating a final gaze point prediction image based on the initial visual saliency image and the attention image.

[0037] Further, the first perspective gaze point prediction model comprises a feature encoding module, a saliency prediction module, an attention shift module and a late fusion module; wherein the feature encoding module is a feature encoding module introducing an Inflated 3D network.

[0038] The feature encoding module and the late fusion module are connected through a parallel saliency prediction module and a gaze state prediction module.

[0039] Further, obtaining spatial features and temporal features comprises:

[0040] inputting the optical flow image into the feature encoding module to extract spatial features and temporal features through the Inflated 3D network; wherein the spatial features are extracted by using a multi-scale feature fusion method.

[0041] Further, obtaining an initial visual saliency image comprises:

[0042] inputting the spatial features and the temporal features into the saliency prediction module to obtain the initial visual saliency image by using an element-by-element summation method for feature fusion.

[0043] Further, the attention shift module comprises a channel weight extractor, a gaze state predictor and an LSTM-based weight predictor.

[0044] The weight predictor is connected with the channel weight extractor and the gaze state predictor.

[0045] Further, obtaining an attention image comprises:

[0046] The maximum value coordinates of the preset image and the spatial feature are input into a channel weight extractor to obtain first channel weights; wherein the preset image is a last predicted gaze point prediction image, and the first channel weights are feature representation channel weights around the last predicted gaze point;

[0047] The last predicted temporal feature is input into a gaze state predictor to obtain gaze state probabilities;

[0048] The first channel weights are input into a weight predictor to obtain second channel weights; wherein the second channel weights are channel weights representing attention regions at the current moment;

[0049] The second channel weights and the gaze state probabilities are used to obtain third channel weights; wherein the third channel weights are final channel weights at the current moment;

[0050] The third channel weights and the spatial feature are weighted to obtain an attention image.

[0051] Further, generating a final gaze point prediction image comprises:

[0052] The initial visual saliency image and the attention image are input into a late fusion module to obtain the final gaze point prediction image; and the coordinates of the maximum value in the gaze point prediction image are recorded for use next time.

[0053] Further, the prediction method further comprises verifying the gaze point prediction image using Area Under Cuve.

[0054] Embodiment

[0055] As shown in the figure, the first view gaze point prediction method based on attention shift provided by the embodiment specifically comprises the following steps: Figure 1

[0056] Step 1: Extract the optical flow frame of the image or video;

[0057] The optical flow image contains information of object motion, and one optical flow image can be extracted from every two pictures or video frames. The optical flow extraction tool dense flow of GPU is used for optical flow extraction.

[0058] Step 2: First view gaze point prediction model based on attention shift;

[0059] In order to better extract visual saliency features, the Inflated 3D (I3D) network is introduced into the feature encoding module of the model, which can fully extract the temporal information in the image or video frame. In addition, the model also designs a saliency prediction module, including an attention shift module of a channel weight extractor, a gaze state predictor and a weight predictor based on LSTM, and a late fusion module.​

[0060] Step 3: The original image or video in step 1 and the extracted optical flow image are input into the feature encoding module in step 2 to extract spatial features and temporal features respectively through the I3D network, wherein the spatial features are extracted using multi-scale feature fusion.

[0061] Step 4: The spatio-temporal features obtained in step 3 are input into the saliency prediction module in step 2, and feature fusion is performed by element-by-element summation to obtain an initial visual saliency image.

[0062] Step 5: The maximum value coordinates of the last predicted gaze map and the spatial features obtained in step 3 are input into the channel weight extractor, and the coordinates are cropped in the spatial features to extract the feature representation channel weight around the last gaze point through average pooling.

[0063] Step 6: The last predicted temporal feature representation obtained in step 3 is input into the gaze state predictor to obtain a gaze state probability, which represents the possibility of gaze at the last time.

[0064] Step 7: The channel weight extracted in step 5 is output as the channel weight representing the attention region at the current time by using the LSTM-based weight predictor.

[0065] Step 8: The final channel weight at the current time is calculated by the channel weight output in step 7 and the gaze state obtained in step 6.

[0066] Step 9: The channel weight extracted in step 8 and the spatial features obtained in step 3 are weighted to obtain an attention image.

[0067] Step 10: The visual saliency image obtained in step 4 and the attention image obtained in step 9 are input into the post-fusion module to obtain the final gaze point prediction image. The coordinates of the maximum value in the image are recorded as the coordinates used in step 5 for the next prediction.

[0068] In order to test the performance of the gaze point prediction in step 10, the test set is used for verification in this embodiment. Area Under Cuve (AUC) is used as the evaluation index in this embodiment.

[0069] AUC is defined as the area enclosed by the coordinate axes under the ROC curve.

[0070] The horizontal coordinate of ROC is false positive rate (FPR), and the calculation formula is as follows

[0071]

[0072] Wherein FP is the false detected gaze position, N is all non-gaze positions;

[0073] The ordinate of ROC is true positive rate (TPR), and the calculation formula is as follows

[0074]

[0075] Wherein TP is the correct detected gaze position, P is all gaze positions;

[0076] AUC can quantitatively reflect the model performance measured based on ROC curve, and is a performance index for measuring the advantages and disadvantages of a learner. The calculation method is to integrate along the horizontal axis of ROC, and the larger the AUC is, the better the performance of the significance detection is.

[0077] As an improvement of the gaze point prediction model, the embodiment proposes a hybrid architecture model based on attention shift. Among them, the dual-flow I3D network is used for spatio-temporal feature extraction at multiple scales, which fully utilizes the time sequence information and high-level semantic information for saliency detection. By modeling the attention shift, the subsequent attention is predicted from the previous gaze. Finally, the visual saliency model is combined to form the final gaze point prediction map, which improves the accuracy of gaze point prediction.

[0078] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A first-person gaze point prediction method based on attention shifting, characterized in that, include: Extract optical flow images; Construct a first-person gaze prediction model based on attention shift; The optical flow image is input into the first viewpoint gaze prediction model to obtain spatial and temporal features. Based on the spatial and temporal features, an initial visual saliency image and an attention image are obtained. Based on the initial visual saliency image and the attention image, they are fused to generate the final gaze prediction image. The first viewpoint gaze prediction model includes: a feature encoding module, a saliency prediction module, an attention transfer module, and a post-fusion module; wherein, the feature encoding module is a feature encoding module incorporating an Inflated 3D network; The feature encoding module and the post-fusion module are connected through the parallel saliency prediction module and gaze state prediction module; Obtaining the spatial and temporal features includes: The optical flow image is input into the feature encoding module, and the spatial and temporal features are extracted using the Inflated 3D network. The attention transfer module includes: a channel weight extractor, a gaze state predictor, and an LSTM-based weight predictor. The weight predictor is connected to the channel weight extractor and the gaze state predictor, respectively. Obtaining the attention image includes: The maximum value coordinates of the preset image and the spatial features are input into the channel weight extractor to obtain the first channel weight; wherein, the preset image is: the previously predicted gaze point image, and the first channel weight is: the feature representation channel weight around the previous gaze point; The previously predicted temporal features are input into the gaze state predictor to obtain the gaze state probability; The first channel weight is input into the weight predictor to obtain the second channel weight; wherein, the second channel weight is: the channel weight representing the attention region at the current time; Based on the second channel weight and the gaze state probability, the third channel weight is obtained; wherein, the third channel weight is: the final channel weight at the current moment; The attention image is obtained by weighting the third channel weight and the spatial features.

2. The first-view gaze point prediction method based on attention transfer according to claim 1, characterized in that, Obtaining the initial visual saliency image includes: The spatial and temporal features are input into the saliency prediction module, and feature fusion is performed using an element-wise summation method to obtain the initial visual saliency image.

3. The first-view gaze point prediction method based on attention transfer according to claim 1, characterized in that, Generating the final gaze prediction image includes: The initial visual saliency image and the attention image are input into the post-fusion module to obtain the final fixation prediction image; and the coordinates of the maximum value in the fixation prediction image are recorded for the next use.

4. The first-view gaze point prediction method based on attention transfer according to claim 1, characterized in that, The prediction method also includes validating the gaze point prediction image using Area Under Cuve.

Images