A device for detecting amnesia based on visual attention representation

Abstract

The application discloses a kind of based on visual attention representation's amnesia detection device, device includes: constructing dataset and extracting visual modal feature module, constructs dataset, extracts the static local feature and dynamic global feature of short video;Build short video memorability prediction network module based on visual attention representation, including: spatial attention unit, feature fusion unit, semantic guided attention unit and loss function unit with complementary decision;The extracted feature is input into the short video memorability prediction network module based on visual attention representation built and trained.The application achieves high-precision short video memorability prediction.

Description

Technical Field

[0001] This invention relates to the field of video memory prediction, and more particularly to an amnesia detection device based on visual attention representation. Background Technology

[0002] Visual memorability, as a representative abstract semantic term, is an indicator that measures the degree to which different observers remember visual instances after a period of time. The confirmation that visual instance memorability scores are independent of time and unaffected by individual judgment allows for a clear definition of visual memorability and makes it objectively measurable.

[0003] Research on video memorability largely follows that of image memorability. Videos are a synthesis of visual, auditory, textual, and motion information, typically conveying richer media content than images. Using only static visual features, such as color and object statistics, is usually insufficient to describe the richness of video content, and predictive models that rely entirely on low-level visual features may yield unsatisfactory results. The inherent gap between the limited descriptive power of low-level features and the richness of high-level semantics perceived by humans is a key bottleneck in inferring visual memorability.

[0004] Currently, research on video memorability is still in its early stages. Han et al. used functional magnetic resonance imaging of the brain to establish a computational model and made the first attempt at predicting video memorability (Han J, Chen C, Shao L, et al. Learning computational models of video memorability from fMRI brainimaging[J]. IEEE transactions on cybernetics, 2014, 45(8):1692-1703.). Before constructing a memorability predictor, Shekhar et al. studied various visual features, including Convolutional 3D (C3D) features, semantic features, saliency features, dense trajectories, and color features, and established an effective method to estimate video memorability, and further extended it into a possible video summarization method (S. Shekhar, D. Singal, H. Singh, M. Kedia, and A. Shetty, “Show and recall: Learning what makes videos memorable,” in 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), 2017, pp.2730–2739.). Cohendet et al. designed a novel protocol based on the image memory score collection protocol to collect long-term video memorability annotations (R. Cohendet, K. Yadati, NQK Duong, and C.-H. Demarty, “Annotating, understanding, and predicting long-term video memorability,” in Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, 2018, pp. 178–186.). Constantin et al. published a task predicting media memorability and released the MediaEval 2019 dataset to promote research on video memorability, which is the largest dataset to date for video memorability prediction (MG Constantin, B. Ionescu, C.-H. Demarty, NQ Duong, X. Alameda-Pineda, and M. “The predicting media memorability task at mediaeval2019,” in Proceedings of MediaEval workshop, 2019. Newman et al. collected a dynamic video memorability dataset containing human annotations with different delay intervals (A. Newman, C. Fosco, V. Casser, A. Lee, B. McNamara, and A. Oliva, “Multimodal memorability: Modeling effects of semantics and decay on video memorability,” in Proceedings of European Conference on Computer Vision, 2020, pp. 223–240.). From the perspective of video memorability modeling, these attempts focus on investigating the importance of features that contribute to video memorability, such as low-level visual, emotional, and semantic features. Analysis of the current state of research in related fields both domestically and internationally shows that the prediction of video memorability is still in its initial stage. Although some related studies have shown continuous improvement in experimental results, the ability to predict video memorability remains poor to date. Therefore, a high-accuracy method for predicting video memorability is needed. Summary of the Invention

[0005] This invention provides an amnesia detection device based on visual attention representation. This invention achieves high-precision prediction of short video memory, as detailed below:

[0006] An amnesia detection device based on visual attention representation, the device comprising:

[0007] The module for constructing a dataset and extracting visual modal features constructs the dataset and extracts static local features and dynamic global features from short videos.

[0008] A short video memory prediction network module based on visual attention representation is constructed, including: a spatial attention unit, a feature fusion unit, a semantically guided attention unit, and a loss function unit with complementary decision-making.

[0009] The extracted features are input into the constructed short video memory prediction network module based on visual attention representation for training.

[0010] The feature fusion unit utilizes different semantic levels of dynamic global features to transform static local features into a semantically aware representation. This unit uses low-rank bilinear pooling with tanh activation to process static local features. and dynamic global features G n Alignment and fusion are performed to obtain semantically guided feature representations.

[0011] The semantically guided attention unit consists of a semantically guided attention component, a GRU component, and a dynamic regression network component. This unit uses a semantically guided attention mechanism to distribute the semantically guided attention weights. Embedded semantically guided feature representation right The representational capabilities are further enhanced to obtain the final semantic guidance representation of short videos. Will The initialization vector of GRUs is obtained by feeding it into a fully connected layer and average pooling. Obtained through GRUs The temporal dependencies between the elements are determined, and the final summary representation of the video semantics is obtained from the hidden layer of the last GRU. The data is fed into a dynamic regression network consisting of two fully connected layers and average pooling to obtain dynamic memory prediction scores.

[0012] Among them, the obtained static memory score and dynamic memory prediction score Substitute the corresponding static loss function L into each value. static and dynamic loss function L dynamic In this study, complementary strategies were used to guide network training. This was achieved through analysis of static memory scores. and dynamic memory scores The weighted sums are then used to obtain the final memory score predicted by the network.

[0013] The beneficial effects of the technical solution provided by this invention are:

[0014] 1) For the task of predicting the memorability of short videos, considering the spatial dependencies and temporal information between video frames, a static feature extractor and a dynamic feature extractor are introduced to model the static local information and dynamic global information in the video respectively, so as to obtain the static local features and the dynamic global features of the video, and to deeply explore the memorability elements in the visual information.

[0015] 2) To address the inherent correlation between static local features and dynamic global features in short videos, low-rank bilinear pooling is introduced to aggregate the memory elements of different visual features, generating a more complete semantic representation of short videos.

[0016] 3) To address the inherent sequential dependencies between static local features and video semantic representation, spatial attention and semantically guided attention mechanisms are introduced to condense the spatial dependencies in keyframes and the temporal dependencies in short video semantic representation.

[0017] 4) This invention makes full use of the visual information of short videos, deeply explores the memory elements contained in different visual features, and considers the dependency relationship between different features, which is conducive to improving the prediction accuracy of short video memory prediction tasks.

[0018] 5) To address the problem of predicting the memorability of short videos in a single-modal scenario, this invention fully considers the spatial dependency between video keyframes and the guiding role of multi-level video semantics on keyframe information. By extracting semantically guided feature representations jointly in the spatiotemporal domain through feature fusion, this invention simultaneously mines the memorable elements contained in frame-level static local information and video dynamic global information through a dual-stream structure.

[0019] 6) Based on the spatial attention mechanism and semantically guided attention mechanism proposed in this invention, the memorable information of each feature connotation is mined and enhanced, and more memorable elements are aggregated from the joint representation of spatial location and spatiotemporal domain. In addition, two types of loss functions with complementary decision rules are associated with the corresponding attention modules to guide the training of the proposed network and improve network performance.

[0020] 7) Based on a trained model, this invention can provide doctors with objective data analysis for the diagnosis of diseases such as amnesia. Using short videos as stimuli, it objectively records the patient's memory of different short videos. Based on the gap between the patient's memory and the memorability of the short videos, it can determine whether the patient suffers from amnesia or other diseases. This invention can not only serve as a diagnostic tool for amnesia and other diseases, but also help doctors objectively measure the severity of a patient's amnesia and develop more targeted treatment plans based on the patient's specific situation. Furthermore, the method of this invention can also be applied to multiple fields such as learning, advertising effectiveness evaluation, and online education, helping people better understand and utilize short video resources, improving learning efficiency and advertising effectiveness. Attached Figure Description

[0021] Figure 1 This is a network diagram of a short video memory prediction method based on visual attention representation;

[0022] Figure 2This is a schematic diagram illustrating the specific extraction process of static local features and dynamic global features;

[0023] Figure 3 This is a schematic diagram illustrating the prediction results of the present invention on the MediaEval 2019 short video memory prediction dataset. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0025] Example 1

[0026] A device for detecting amnesia based on visual attention representation, the device comprising:

[0027] Part 1: Building the Dataset and Extracting Visual Modal Features

[0028] The specific operation of this module is as follows:

[0029] 1) Construct the dataset: Download the MediaEval 2019 dataset, and divide 87.5% of the short videos in the dataset into the training set and 12.5% ​​into the test set.

[0030] 2) Visual modal feature extraction: For each original short video, extract static local features and dynamic global features respectively. Let the two types of visual modal features be: static local features Dynamic global features in, Let i be the feature vector at the i-th position in the feature map of the f-th keyframe of the n-th video. Let be the semantic representation vector of the l-th layer of the n-th video.

[0031] Static local features are used to characterize higher-level semantic information of individual keyframes in the spatial dimension. Dynamic global features characterize the overall video content by leveraging temporal dependencies embedded in consecutive keyframes.

[0032] Part Two: Constructing a short video memory prediction network module based on visual attention representation:

[0033] The network module can be divided into four parts: a spatial attention unit, a feature fusion unit, a semantically guided attention unit, and a loss function unit with complementary decision-making.

[0034] 1) Spatial Attention Unit: This unit consists of a spatial attention component, a GRU (Gated Recurrent Unit) component, and a static regression network component. To aggregate local features and their spatial dependencies, this unit adaptively captures high-order dependencies from static local representations by incorporating a spatial attention mechanism into the GRUs. The GRUs initialization vector is obtained by feeding the GRUs into a fully connected layer and then performing average pooling. The GRU representation vector at step t-1. Embedded static local features In this process, the network is helped to focus on static memory elements, resulting in the spatial attention weight vector at step t. Spatial attention weights Embedded Obtain the spatial attention module weighting vector in the t-th step of GRUs. The GRU representation vector at step t. The vector represented by GRUs in the previous step Weighted vectors of spatial attention modules Update. The state vectors of all GRU steps are input into a static regression network consisting of a two-layer fully connected network and average pooling to obtain the static memory score.

[0035] 2) Feature Fusion Unit: This unit utilizes different semantic levels of dynamic global features to transform static local features into a semantically aware representation. This unit uses low-rank bilinear pooling with tanh activation to fuse static local features. and dynamic global features G n Alignment and fusion are performed to obtain semantically guided feature representations.

[0036] 3) Semantically Guided Attention Unit: This unit consists of a semantically guided attention component, a GRU component, and a dynamic regression network component. This unit uses a semantically guided attention mechanism to distribute the semantically guided attention weights. Embedded semantically guided feature representation right The representational capabilities are further enhanced to obtain the final semantic guidance representation of short videos. Will The initialization vector of GRUs is obtained by feeding it into a fully connected layer and average pooling. Obtained through GRUs The temporal dependencies between the elements are determined, and the final summary representation of the video semantics is obtained from the hidden layer of the last GRU. The data is fed into a dynamic regression network consisting of two fully connected layers and average pooling to obtain dynamic memory prediction scores.

[0037] 4) Loss function unit with complementary decision-making: the obtained static memory score and dynamic memory prediction score Substitute the corresponding static loss function L into each value. static and dynamic loss function L dynamic In this study, complementary strategies were used to guide network training. This was achieved through analysis of static memory scores. and dynamic memory scores The weighted sums are then used to obtain the final memory score predicted by the network.

[0038] Step 3, Model Training Module:

[0039] The extracted features are input into the constructed short video memory prediction network module based on visual attention representation for training.

[0040] Step 4: Practical Application

[0041] The trained model achieved state-of-the-art results on the MediaEval 2019 dataset. This device not only explores new directions for future research on short video memory prediction but also solves the current problem of low accuracy in short video memory prediction. This expands its application beyond predicting short video memory to include medical use for the diagnosis and treatment of amnesia, Alzheimer's disease, and similar conditions. In modern medicine, the diagnosis of amnesia relies primarily on the doctor's experience and the patient's self-report. However, this method is often subjective and prone to error. The network for predicting short video memory in this invention can improve its accuracy through objective data analysis and continuous training and adjustment, better adapting to the needs of amnesia patients and providing doctors with more accurate diagnostic information. This invention solves the technical problems of predicting short video memory based on trained models for amnesia and similar diseases, providing a new approach and method for future medical diagnosis and treatment.

[0042] Example 2

[0043] The following is combined with Figures 1-3 The scheme in Example 1 will be described in detail below:

[0044] Step 1: Constructing the dataset and extracting visual modal features:

[0045] The specific operation of this module is as follows:

[0046] 1) This embodiment of the invention uses the MediaEval 2019 dataset, which contains 8000 silent short videos of various video types. All videos are 7 seconds long and contain 24 frames per second. 87.5% of the short videos in the dataset are randomly selected as the training set, and the remaining 12.5% ​​are used as the test set.

[0047] 2) Extract the static local features and dynamic global features of the short video respectively:

[0048] For static local features, for each short video V n Seven image frames are extracted at equal time intervals. As a keyframe, the static feature extractor f static (·) represents a ResNet-50, where the final average pooling and fully connected layers are replaced with adaptive average pooling. The keyframes are input sequentially into f. static (·) Extract deep features to obtain the output features of the static feature extraction module.

[0049]

[0050] in, This represents the f-th keyframe of the n-th video. Let R represent the feature vector at the i-th position of the static local feature map of the f-th keyframe. Each keyframe's static local feature map has I = W × H positions. W, H, and C represent the width, height, and number of channels of the feature map, respectively, and R represents the real number field.

[0051] For dynamic global features, short videos are input into a 3D ResNet. The outputs of the three convolutional blocks (res3, res4, and res5) in the 3D ResNet are used to extract three levels of video semantic features. The semantics and temporal scale of these different levels of video features are modeled using a Temporal Pyramid Network (TPN). The output features of the TPN are then processed through maxpooling and concatenation operations to obtain the output features of the dynamic feature extraction module.

[0052] G n =f dynamic (V n )∈R L×D

[0053] Among them, f dynamic (·) indicates a dynamic feature extractor. is the semantic representation vector of the l-th layer, where L and D are the number of semantic layers and the feature dimension of the dynamic global feature.

[0054] The second step is to build a short video memory prediction network module based on visual attention representation using the deep learning framework PyTorch.

[0055] The short video memory prediction network module based on visual attention representation consists of four parts: a spatial attention unit, a feature fusion unit, a semantically guided attention unit, and a loss function unit with complementary decision-making.

[0056] The following is a detailed introduction to these four parts:

[0057] 1) Spatial Attention Unit: Composed of a spatial attention part, a GRU part, and a static regression network part. In order to aggregate local features and their spatial dependencies, this unit adaptively captures high-order dependencies from static local representations by incorporating a spatial attention mechanism into the GRUs.

[0058] ① By using static local features The initial state vector of the GRU is obtained by feeding it into a fully connected layer and then performing average pooling. and use The formula for initializing GRUs is as follows:

[0059]

[0060] Among them, f init_1 (·) represents a fully connected layer with a hyperbolic activation function tanh(·).

[0061] ② The GRU representation vector at step t-1 is transformed by three fully connected layers and a standard softmax operation. Embedded Obtain the spatial attention weight vector at step t. The formula is as follows:

[0062]

[0063] Where, Q∈R I×C , K∈R C×C These are learnable network weights. spa This is the spatial attention equation.

[0064] ③ Adjust spatial attention weights Embedded Obtain the spatial attention unit weighting vector in the t-th step of GRUs. The formula is as follows:

[0065]

[0066] ④Let As the representation vector of the GRUs at step t, it is derived from the representation vector of the previous step. Weighted vector of spatial attention unit The formula is as follows:

[0067]

[0068] Among them, f gru (·) represents GRUs, d h It represents the dimension of the vector.

[0069] ⑤ Input the state vectors of all GRU steps into a static regression network consisting of two fully connected layers and average pooling to obtain the static memory score, as shown in the following formula:

[0070]

[0071] Where T represents the number of steps in GRUs, F represents the total number of keyframes, and f map (·) is implemented by a two-layer fully connected network with a single output neuron and a linear activation function.

[0072] 2) Feature Fusion Unit: This unit utilizes different semantic levels of dynamic global features to transform static local features into semantically guided feature representations. Through low-rank bilinear pooling, static local features are... and dynamic global features G n Alignment and fusion are performed to obtain semantically guided feature representations. The formula is as follows:

[0073]

[0074] in, and These are two embedding matrices used to align the two types of features. and These are learnable network weights. D a and D b It is a feature representation guided by matrices and semantics. The dimension of . · Represents element-wise multiplication.

[0075] 3) Semantic-guided attention unit: It consists of a semantic-guided attention part, a GRU part, and a dynamic regression network part.

[0076] ①Semantic-guided feature representation The initial state vector of GRUs is obtained by feeding it into a fully connected layer and average pooling. and use vectors The formula for initializing GRUs is as follows:

[0077]

[0078] Among them, f init_2(·) represents a fully connected layer, L is the number of semantic layers for dynamic global features, and I is the unknown quantity of the static local feature map for each keyframe.

[0079] ②Semantic-guided feature representation The semantic guidance attention weights are obtained through a fully connected layer and a standard softmax operation. The formula is as follows:

[0080]

[0081] Among them, f saa (·) represents a fully connected layer

[0082] ③ Impart semantically guided attention weights Embedded semantically guided feature representation The final semantic guidance representation of the short video is obtained through average pooling. The formula is as follows:

[0083]

[0084] ④ Obtained through GRUs The temporal dependencies between the elements are determined, and the final summary representation of the video semantics is obtained from the hidden layer of the last GRU. The formula is as follows:

[0085]

[0086] in, It is a learnable affine transformation matrix, D c Let f represent the hidden layer feature dimension of GRUs, and ReLU(·) represent the activation function. mutual Represents the gate function. This represents the current state vector of the GRUs. f represents the output vector of the gated function. gru This refers to GRUs.

[0087] ⑤ The final summary representation of video semantics The data is fed into a dynamic regression network consisting of two fully connected layers and average pooling to obtain the dynamic memory prediction score, as shown in the following formula:

[0088]

[0089] Two types of loss functions with complementary decision-making are applied to the corresponding attention units, while simultaneously guiding the training of the entire network to improve its predictive performance. This is further demonstrated by applying static memory scores. and dynamic memory scores The weighted sums are then used to obtain the final memory score predicted by the network.

[0090] The following is a detailed introduction:

[0091] Static loss function L static The formula is as follows:

[0092]

[0093] The first term represents the true memory score y of the video. n and static memory scores The standard mean square error between the two terms, the second term is the joint penalty norm of l1 and l2 to encourage spatial attention units to explore more memory-related elements at all locations, and λ is the weighting coefficient. The spatial attention weights for a video are represented by the following formula:

[0094]

[0095] ⑥ Dynamic loss function L dynamic The formula is as follows:

[0096]

[0097] in, f represents dynamic memory fractions. Θ (·) and Rank(·) represent ranking functions, and γ is the weight coefficient.

[0098] ⑦ Through static memory fractions and dynamic memory scores The weighted sums are then used to obtain the final memory score predicted by the network. The formula is as follows:

[0099]

[0100] Where σ1 is the static memory score weight.

[0101] Step 3: Model Training

[0102] During training, the Adam stochastic gradient descent optimizer was used, with an initial learning rate of 0.00001, a batch size of 4, and an epoch of 30. After training, the network model and parameters were saved.

[0103] Step 4, Specific Applications:

[0104] By obtaining the trained memory prediction model through the above steps, the amnesia detection device based on visual attention representation proposed in this invention is completed. This device can be specifically applied in three areas: early diagnosis, disease monitoring, and treatment assessment for amnesia and other diseases.

[0105] The following is a detailed introduction to these three aspects:

[0106] 1) Early Diagnosis: In the early diagnosis of diseases such as amnesia, doctors can construct targeted short video datasets according to actual needs. For example, based on the patient's age, gender, and occupation, short video clips on specific themes can be selected, and the present invention can be used to assess the memory performance of these short videos. By playing short videos on relevant themes for the patient and analyzing their memory performance while watching the videos, if the patient's overall memory performance is lower than the assessment of the present invention, it is believed that the patient may have amnesia or other diseases. This helps doctors detect memory problems in patients at an early stage, further assess their cognitive abilities, and provide appropriate treatment.

[0107] 2) Disease Monitoring: Objectively assessing the memory status of patients with amnesia has always been a medical challenge. To prevent repeated short videos from reinforcing the patient's memory and leading to misjudgments of the condition, this invention allows for the continuous supplementation of short video examples. Doctors can periodically play non-repeating short videos during the patient's treatment for long-term monitoring, in order to detect any signs of memory decline or cognitive deterioration early. This helps to adjust the treatment plan in a timely manner to slow the progression of the disease.

[0108] 3) Assess treatment effectiveness: At the end of the patient's treatment, the doctor can play short videos that have been assessed by this invention and have the same memory properties, compare the patient's memory performance before and after watching the short videos, and determine whether the patient has been cured.

[0109] Figure 3 This is an example of the prediction results of this invention on the MediaEval 2019 short video memory prediction dataset. It can be seen that the method proposed in this invention achieves a relatively ideal prediction effect on video memory scores.

[0110] In summary, this invention proposes a deep spatiotemporal collaborative network for attentional representations in video memory prediction. Spatial and semantically guided attention units model static local information and dynamic global information in the video, respectively. A two-stream network architecture integrates these spatial and semantically guided attention units to simultaneously capture frame-level static local information and dynamic global information in the video. Specifically, spatial attention units aggregate more memory-related elements from spatial locations, feature fusion units achieve semantic alignment and intermediate fusion of local and global information, and semantically guided attention units supplement the spatiotemporal joint representation with more memory-related elements. Furthermore, two types of loss functions with complementary decision rules are associated with the corresponding attention units to guide the training of the proposed network and improve network performance.

[0111] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0112] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0113] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A device for detecting amnesia based on visual attention representation, characterized in that, The device comprises: a dataset construction and visual modality feature extraction module, which constructs a dataset and extracts static local features and dynamic global features of the short video; a short video memorability prediction network module based on visual attention representation, which comprises a spatial attention unit, a feature fusion unit, a semantic-guided attention unit, and a loss function unit with complementary decision-making; the extracted features are input into the short video memorability prediction network module based on visual attention representation to be trained, and the trained memorability prediction model is used to evaluate the memorability of the short video; The feature fusion unit converts the static local feature into a semantic-guided feature representation by using different semantic levels of the dynamic global feature, and aligns and fuses the static local feature and the dynamic global feature by low-rank bilinear pooling to obtain the semantic-guided feature representation and the dynamic global feature The formula is as follows:​ ； wherein, and are two embedding matrices for aligning two types of features, and are learnable network weights, and are matrix and semantic guided feature representations of dimensionality, denotes multiplication between elements. the semantic-guided attention unit is composed of a semantic-guided attention part, a GRUs part, and a dynamic regression network part, Semantically guided feature representation into a fully connected layer and average pooling to get the initial state vector of GRUs and the vector Initialize the GRUs; ； wherein, represents a fully connected layer, L is the number of semantic layers of dynamic global features, and I is the unknown number of static local feature maps of each key frame. Semantically guided feature representation The semantic guided attention weights are obtained by a fully connected layer and a standard softmax operation ; ； wherein, represents a fully connected layer; Semantically guided attention weights Embedding semantically guided feature representations , and the final semantically guided representation of the short video is obtained through an average pooling operation ; ； obtained by GRUs time-dependent relationship between the frames and the last GRUs' hidden layer obtains the final summary representation of the video semantics ; ； wherein, is a learnable affine transformation matrix, denotes the hidden layer feature dimension of GRUs, denotes an activation function, denotes a gating function, denotes the current state vector of GRUs, denotes the output vector of the gating function, denotes GRUs; Final aggregated representation of video semantics The video is fed into a dynamic recurrent network consisting of two fully connected layers and average pooling to obtain a dynamic memorability prediction score.

2. The device for detecting amnesia based on visual attention representation according to claim 1, wherein, the dynamic global feature is: Three levels of video semantic features are extracted by using the convolution blocks res3, res4 and res5 in the 3D ResNet; the semantics and time scale of the video semantic features are modeled by using the TPN, max pooling and concatenate operations are performed on the output features, and the output features : ； wherein, denotes a dynamic feature extractor, is the first layer semantic representation vector, and is the number of semantic layers and feature dimensions of the dynamic global feature, and R denotes the real number field, denotes a short video. 3.The device for detecting amnesia based on visual attention representation of claim 1, wherein, the spatial attention unit is composed of a spatial attention part, a GRUs part, and a static regression network part, the state vectors of all GRUs steps are input into a static regression network composed of two fully connected networks and an average pooling to obtain a static memorability score, as follows: ； wherein, represents the number of steps of the GRUs, F denotes the total number of key frames, implemented by a two-layer fully connected network with a single output neuron and a linear activation function.

Images