Video memorability prediction method, apparatus, device, and storage medium

Abstract

The present disclosure provides a video memorability prediction method, device, equipment and storage medium. The video memorability prediction method of the present disclosure comprises: obtaining a plurality of video frame level features of an input video; performing visual multi-level coding on the plurality of video frame level features to obtain global features, time domain features and local features, and splicing the global features, the time domain features and the local features to obtain visual multi-level coding features; obtaining text features of a paired text of the input video; extracting optical flow features of the input video; guiding and enhancing the visual multi-level coding features using the text features to obtain text-guided visual features; obtaining a text memorability prediction score, a video frame memorability prediction score and an optical flow memorability prediction score; and fusing the text memorability prediction score, the optical flow memorability prediction score and the video frame memorability prediction score to obtain a video memorability prediction score of the input video. The present disclosure effectively improves the accuracy and precision of video memorability prediction by fully extracting video features and fully utilizing the consistency of text and vision.

Description

Technical Field

[0001] This disclosure belongs to the field of computer vision technology, and in particular relates to a video memory prediction method, apparatus, device and storage medium based on visual multi-level coding and text-guided visual feature enhancement. Background Technology

[0002] Humans retain different things to varying degrees. Research shows that the degree to which people retain different things is related to the perceptual information they receive from those things, such as visual information. This indicates that video recall is related to the content carried by the video itself, and can be predicted based on the video's content.

[0003] From an artificial intelligence perspective, enabling computers to mimic human recall patterns across different videos based on their content is significant. Video recall prediction tasks aim to allow computers to automatically predict the degree of recall for a video based on the information provided. Specifically, video recall is defined as the ability to remember a video after a certain time span.

[0004] Video memory has broad application prospects. Today, we inevitably face a vast amount of video in all aspects of life. Browsing through large amounts of video is time-consuming and energy-intensive. Video memory can help select memorable content from videos and discard less memorable segments, often considered irrelevant or useless.

[0005] Visual information in videos is a major factor contributing to varying levels of human memorability, highlighting the significant importance of extracting effective visual features for video recall. Existing video recall prediction techniques rely on general neural networks to extract visual features, such as using ResNet for frame-level features or I3D networks for video-level features. However, these methods lack specificity for the video recall prediction task. This is because video recall prediction differs from general computer vision tasks like image classification or action recognition. In those tasks, label generation is objective; a picture of a dog will always be classified as a dog. However, the degree to which a person remembers the same video varies greatly, indicating a degree of subjectivity in recall scores. Visual features extracted based on general neural networks cannot reflect how humans remember videos.

[0006] Text is a high-level summary of visual content, containing complex semantic information. In video memory prediction, many text-based auxiliary methods have been adopted, such as jointly training video memory prediction and text generation tasks to implicitly utilize text features, or fusing visual memory scores and text memory scores. However, these techniques cannot effectively utilize the consistency between text and vision. Summary of the Invention

[0007] To address the problems existing in the aforementioned video memory technology, this disclosure proposes a video memory prediction method, apparatus, device, and storage medium based on visual multi-level coding and text-guided visual feature enhancement.

[0008] The first aspect of this disclosure provides a video memory prediction method, comprising:

[0009] Multiple video frame-level features of the input video are obtained using multiple video frames of the input video;

[0010] Visual multi-level encoding is performed on multiple video frame-level features of the input video to obtain global features, temporal features and local features, and the global features, temporal features and local features are concatenated to obtain the visual multi-level encoded features of the input video;

[0011] Obtain the text features of the text paired with the input video;

[0012] Extract the optical flow map from the input video and obtain the optical flow features of the optical flow map;

[0013] The text features of the text paired with the input video are used to guide and enhance the visual multi-level coding features of the input video, thereby obtaining the text-guided visual features of the input video.

[0014] Obtain the text memory prediction score of the text features, the video frame memory prediction score of the text-guided visual features, and the optical flow memory prediction score of the optical flow features;

[0015] The late-stage fusion method based on an adaptive strategy fuses the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score to obtain the video memory prediction score of the input video.

[0016] In some embodiments of the first aspect of this disclosure, the global features are obtained by performing an average pooling operation on multiple video frame-level features of the input video.

[0017] In some embodiments of the first aspect of this disclosure, the temporal features are obtained by: feeding multiple frequency frame-level features of the input video into a bidirectional gated recurrent unit network, extracting each hidden state output of the bidirectional gated recurrent unit network, and performing an average pooling operation on all hidden state outputs of the bidirectional gated recurrent unit network to obtain the temporal features.

[0018] In some embodiments of the first aspect of this disclosure, the local features are obtained by: processing each hidden state output of the bidirectional gated recurrent unit network sequentially through multiple one-dimensional convolutional neural networks with different kernel sizes and the same number of kernels, processing with nonlinear activation functions, and performing average pooling operations to obtain multiple local features with different receptive fields, and then concatenating the multiple local features with different receptive fields.

[0019] In some embodiments of the first aspect of this disclosure, obtaining the text features of the text paired with the input video includes: cleaning the sentences in the text paired with each input video, storing the sentences in the form of a list; adding a predetermined identifier to the list and inputting it into a BERT model, taking the vector corresponding to the predetermined identifier in the last layer of the BERT model as the sentence feature, and the sentence feature being the text features of the text paired with the input video.

[0020] In some embodiments of the first aspect of this disclosure, guiding and enhancing the visual multilevel coding features of the input video using text features of the text paired with the input video includes: segmenting the visual multilevel coding features into multiple segments, and mapping the segmented segments sequentially through a fully connected layer, a ReLU activation function, and a fully connected layer to a visual-text common space; mapping the text features through a fully connected layer, a ReLU activation function, and a fully connected layer to a visual-text common space; adding the mapped text features and multiple segments of the mapped visual multilevel coding features, and then processing them sequentially through a tanh activation function and a fully connected layer to obtain importance scores for each segment; mapping the importance scores of each segment of the visual multilevel coding features to a probability space to obtain normalized importance scores for each segment; and using the normalized importance scores as the weights of the segments, weighted summing of the segments of the visual multilevel coding features to generate the text-guided visual features.

[0021] In some embodiments of the first aspect of this disclosure, obtaining the text memory prediction score of the text features, the video frame memory prediction score of the text-guided visual features, and the optical flow memory prediction score of the optical flow features includes: processing the optical flow features using an optical flow multilayer perceptron network to generate an optical flow memory prediction score; processing the text features using a text multilayer perceptron network to generate a text memory prediction score; and processing the text-guided visual features using a video frame multilayer perceptron network to generate a video frame memory prediction score; wherein the optical flow multilayer perceptron network, the text multilayer perceptron network, and the video frame multilayer perceptron network are all trained using mean squared error as a loss function.

[0022] In some embodiments of the first aspect of this disclosure, the late fusion method based on an adaptive strategy fuses the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score, comprising: assigning weights to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score using the late fusion method based on an adaptive strategy; and performing a weighted summation of the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score using the assigned weights to obtain the video memory prediction score of the input video.

[0023] In some embodiments of the first aspect of this disclosure, the method of using an adaptive strategy-based late fusion approach to assign weights to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score includes: assigning weights to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score respectively to determine a weight allocation strategy; updating the weight values ​​of the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score according to a predetermined step size to update the weight allocation strategy; and evaluating each weight allocation strategy using the Spearman coefficient as an evaluation metric, and determining the weight allocation strategy that achieves the highest Spearman coefficient as the final weight allocation strategy.

[0024] A second aspect of this disclosure provides a video memory prediction device, comprising:

[0025] A video frame-level feature extraction module is used to obtain multiple video frame-level features of the input video using multiple video frames;

[0026] A visual multi-level coding module is used to perform visual multi-level coding on multiple video frame-level features of the input video to obtain global features, temporal features and local features, and to concatenate the global features, temporal features and local features to obtain the visual multi-level coded features of the input video;

[0027] The text feature extraction module is used to obtain the text features of the text paired with the input video;

[0028] An optical flow feature extraction module is used to extract the optical flow map of the input video and obtain the optical flow features of the optical flow map;

[0029] The text-guided visual feature module is used to guide and enhance the visual multi-level coding features of the input video using the text features of the text paired with the input video, so as to obtain the text-guided visual features of the input video.

[0030] A multimodal memory score prediction module is used to obtain the text memory prediction score of the text features, the video frame memory prediction score of the text-guided visual features, and the optical flow memory prediction score of the optical flow features.

[0031] The multimodal memory score fusion module is used to fuse the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score using a late fusion method based on an adaptive strategy to obtain the video memory prediction score of the input video.

[0032] In some embodiments of the second aspect of this disclosure, the visual multi-level coding module is specifically used to obtain the global features by performing average pooling on multiple video frame-level features of the input video; and / or, the visual multi-level coding module is specifically used to obtain the temporal features by: feeding multiple frequency frame-level features of the input video into a bidirectional gated recurrent unit network, extracting each hidden state output of the bidirectional gated recurrent unit network, and performing average pooling on all hidden state outputs of the bidirectional gated recurrent unit network to obtain the temporal features; and / or, the visual multi-level coding module is specifically used to obtain the local features by: processing each hidden state output of the bidirectional gated recurrent unit network sequentially through multiple one-dimensional convolutional neural networks with different kernel sizes and the same number of kernels, processing with a nonlinear activation function, and performing average pooling to obtain multiple local features with different receptive fields, and concatenating the multiple local features with different receptive fields.

[0033] In some embodiments of the second aspect of this disclosure, the text-guided visual feature module is specifically used to obtain the text-guided visual features of the input video in the following manner: The visual multi-level coding features are segmented into multiple segments, and the segmented segments are sequentially processed through a fully connected layer, a ReLU activation function, and a fully connected layer to map to a visual-text common space; the text features are processed through a fully connected layer, a ReLU activation function, and a fully connected layer to map to the visual-text common space; the mapped text features and multiple segments of the mapped visual multi-level coding features are added together and then sequentially processed through a tanh activation function and a fully connected layer to obtain the importance score of each segment; the importance scores of each segment of the visual multi-level coding features are mapped to a probability space to obtain a normalized importance score of each segment; using the normalized importance score as the weight of the segment, the segments of the visual multi-level coding features are weighted and summed to generate the text-guided visual features.

[0034] In some embodiments of the second aspect of this disclosure, the multimodal memory score prediction module is specifically used to: process the optical flow features using an optical flow multilayer perceptron network to generate an optical flow memory prediction score; process the text features using a text multilayer perceptron network to generate a text memory prediction score; and process the text-guided visual features using a video frame multilayer perceptron network to generate a video frame memory prediction score; wherein the optical flow multilayer perceptron network, the text multilayer perceptron network, and the video frame multilayer perceptron network are all trained using mean squared error as a loss function.

[0035] In some embodiments of the second aspect of this disclosure, the multimodal memory score fusion module is specifically configured to: assign weights to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score using a late fusion method based on an adaptive strategy; and perform a weighted summation of the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score using the assigned weights to obtain the video memory prediction score of the input video.

[0036] In some embodiments of the second aspect of this disclosure, the multimodal memory score fusion module is specifically configured to allocate weights in the following manner: assigning weights to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score respectively to determine a weight allocation strategy; updating the weight values ​​of the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score according to a predetermined step size to update the weight allocation strategy; and evaluating each weight allocation strategy using the Spearman coefficient as an evaluation metric, and determining the weight allocation strategy that achieves the highest Spearman coefficient as the final weight allocation strategy.

[0037] A third aspect of this disclosure provides an electronic device comprising:

[0038] Memory, the memory storing execution instructions; and

[0039] A processor that executes the execution instructions stored in the memory, causing the processor to perform the video memory prediction method described above.

[0040] A fourth aspect of this disclosure provides a readable storage medium storing executable instructions that, when executed by a processor, are used to implement the video memory prediction method described above.

[0041] This disclosure provides a video memorability prediction method based on visual multi-level coding and text-guided visual enhancement. Visual multi-level coding extracts global, temporal, and local features. Global features reflect the overall impression people have of a video when they remember it; temporal features show the progression of events over time; and local features reveal specific details. Given that text is a high-level summary of visual content and that there is semantic consistency between text and video, when annotating text, people tend to focus on important or interesting visual information in the video—often highly memorable segments. This means that the semantic information covered by the text often corresponds to highly memorable video segments, indicating that text-guided visual features are more relevant to visual memorability. Therefore, this disclosure uses text-guided visual enhancement before determining the video memorability prediction score. Thus, by fully extracting video features and leveraging the consistency between text and visual elements, this disclosure effectively improves the accuracy and precision of video memorability prediction. Attached Figure Description

[0042] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0043] Figure 1 This is a flowchart illustrating a video memory prediction method according to some embodiments of the present disclosure;

[0044] Figure 2 This is a flowchart illustrating visual multilevel coding according to some embodiments of the present disclosure;

[0045] Figure 3 This is a schematic diagram illustrating the implementation process of text guidance and visual enhancement according to some embodiments of this disclosure;

[0046] Figure 4 This is a schematic flowchart of multimodal memory score fusion according to some embodiments of the present disclosure;

[0047] Figures 5a to 5f These are example images of different video frames from the input video;

[0048] Figure 6 This is a schematic block diagram of a video memory prediction device that employs a hardware implementation of a processing system, according to one embodiment of this disclosure. Detailed Implementation

[0049] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0050] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0051] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0052] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0053] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0054] For descriptive purposes, this disclosure may use spatial relative terms such as “below,” “under,” “below,” “down,” “above,” “above,” “higher,” and “side (e.g., in a “sidewall”)” to describe the relationship between one component and another component as shown in the accompanying drawings. In addition to the orientations depicted in the drawings, the spatial relative terms are also intended to encompass different orientations of the device during use, operation, and / or manufacture. For example, if the device in the drawings is flipped, a component described as “below” or “under” another component or feature would subsequently be positioned “above” said other component or feature. Thus, the exemplary term “below” can encompass both “above” and “below” orientations. Furthermore, the device may be otherwise positioned (e.g., rotated 90 degrees or in other orientations), thus interpreting the spatial relative descriptive terms used herein accordingly.

[0055] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0056] The following text combines Figures 1 to 6 The specific embodiments of this disclosure will be described in detail.

[0057] Figure 1 A flowchart illustrating the video memory prediction method based on visual multilevel coding and text-guided visual feature enhancement disclosed herein is shown.

[0058] For example, see Figure 1 The method disclosed herein can be implemented using the following models: an optical flow feature extractor, a video frame feature extractor, a text feature extractor, a visual multilevel coding module for text guidance and visual enhancement, an optical flow multilayer perceptron network, a video frame multilayer perceptron network, and a text multilayer perceptron network. It should be noted that this model is merely an example. In practical applications, other models or model structures can also be used to implement the method disclosed herein, and this disclosure does not impose any limitations on this.

[0059] In some embodiments of this disclosure, such as Figure 1 As shown, the video memory prediction method disclosed herein may include the following steps:

[0060] Step S100: Obtain multiple video frame-level features of the input video using multiple video frames;

[0061] Specifically, for each input video, n frames are extracted uniformly, and each of the n video frames is input into a video frame feature extractor (e.g., a pre-trained neural network) to extract and save video frame-level features. In this way, multiple video frame-level features for each input video can be obtained.

[0062] Where n is the number of video frames extracted, which can be flexibly set according to actual needs. For example, n can be 8.

[0063] Step S200: Obtain global features, temporal features and local features by performing visual multi-level coding on multiple video frame-level features of the input video, and concatenate the global features, temporal features and local features to obtain the visual multi-level coded features of the input video;

[0064] Specifically, the multiple video frame-level features extracted in step S100 are input into the visual multi-level coding module to extract global features, temporal features and local features. Then, the global features, temporal features and local features are concatenated to obtain the visual multi-level coding features of the input video.

[0065] Step S300: Obtain the text features of the text paired with the input video;

[0066] Specifically, the text paired with each input video can be cleaned and then fed into a text feature extractor (e.g., a pre-trained BERT model) to extract sentence-level features, which are the text features of the text paired with the input video.

[0067] Step S400: Extract the optical flow map from the input video and obtain the optical flow features of the optical flow map;

[0068] Specifically, an optical flow map is extracted for each input video, and the extracted optical flow map is fed into an optical flow feature extractor (e.g., a pre-trained I3D model) to extract optical flow features.

[0069] Step S500: Use the text features of the text paired with the input video to guide and enhance the visual multi-level coding features of the input video, and obtain the text-guided visual features of the input video.

[0070] That is, the text features extracted in step S300 are used to guide and enhance the visual multi-level coding features generated in step S200 to obtain the text-guided visual features of the input video.

[0071] Step S600: Obtain the text memory prediction score of text features, the video frame memory prediction score of text-guided visual features, and the optical flow memory prediction score of optical flow features;

[0072] Specifically, the text features extracted in step S300, the optical flow features extracted in step S400, and the text-guided visual features generated in step S500 can be input into different multilayer perceptron networks for processing to obtain corresponding video memory scores, which are used as text memory prediction scores, optical flow memory prediction scores, and video frame memory prediction scores, respectively.

[0073] like Figure 1 As shown, the text features extracted in step S300 are input into a text multilayer perceptron network for processing to obtain a text memory prediction score; the optical flow features extracted in step S400 are input into an optical flow multilayer perceptron network for processing to obtain an optical flow memory prediction score; and the text-guided visual features generated in step S500 are input into a video frame multilayer perceptron network for processing to obtain a video frame memory prediction score.

[0074] Among them, optical flow multilayer perceptron networks, video frame multilayer perceptron networks, and text multilayer perceptron networks can all be trained using mean squared error as the loss function.

[0075] Step S700: The late fusion method based on an adaptive strategy fuses the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score to obtain the video memory prediction score of the input video.

[0076] Specifically, a late-stage fusion method based on an adaptive strategy can be used to assign weights to the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score, and then use the assigned weights to perform a weighted sum of the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score to obtain the video memory prediction score of the input video.

[0077] In some implementations, the late-stage fusion method based on an adaptive strategy assigns weights to the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score. This includes determining a weighting strategy for the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score based on the late-stage fusion method, and using the Spearman coefficient as an evaluation metric to evaluate the weighting strategy to select the weighting strategy with the highest Spearman coefficient to determine the weights of the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score.

[0078] This disclosure effectively improves the precision and accuracy of video memory prediction by applying visual multilevel coding and text-guided visual feature enhancement in the video memory prediction method.

[0079] In some implementations, step S100 can be achieved through the following steps:

[0080] Step a1: Use Python's OpenCV package to extract n frames evenly from the input video;

[0081] Step a2: Input each video frame from step a1 into the CLIP model to extract features, and take the output of the last layer on the visual side of the CLIP model as the video frame level feature.

[0082] If n is 8, the extracted video frame-level features can be represented as [v1, v2, ..., v8|, v i Let i represent the video frame-level feature at the time corresponding to the i-th frame, where i = 1, 2, ..., 8.

[0083] The CLIP model is a pre-trained model on a dataset containing 400 million image-text pairs. Using the Transformer architecture as the network infrastructure, pre-training the model on a large dataset can improve its generalization ability and zero-shot transfer learning ability. CLIP has achieved good results on multiple tasks following the above paradigm. Therefore, this disclosure uses the CLIP model to extract video frame-level features.

[0084] Figure 2 A flowchart illustrating the visual multi-level encoding process in step S200 is shown. Figure 2 As shown, the visual multi-level coding in step S200 may include the following steps:

[0085] Step S201: Perform average pooling on the n video frame-level features of each input video to obtain the global feature f1.

[0086] The average pooling operation is performed as shown in equation (1):

[0087]

[0088] Where n represents the number of frames to extract from the input video; for example, n can be set to 8, v t It is the video frame-level feature corresponding to the time of frame t.

[0089] Average pooling layers can capture recurring semantic information in video frames, which represents the most frequently occurring content in the video. Humans tend to form an overall impression of the most frequently occurring content in a video. Therefore, this disclosure generates global features through average pooling operations.

[0090] Step S202: Feed the 8 video frame-level features of each input video into a bidirectional gated recurrent unit network (GRU), extract each hidden state output of the bidirectional gated recurrent unit network, and perform average pooling operation on all hidden state outputs of the bidirectional gated recurrent unit network to obtain temporal features f2.

[0091] Extracting the output of each hidden state of the bidirectional gated recurrent unit network can include: forward propagating the output of each hidden state. and backpropagation output The concatenation is output as the hidden state at the current moment. As shown in equations (2) and (3).

[0092]

[0093] In equation (2), GRU forward It is a forward GRU in a bidirectional GRU network. This means the hidden state of the previous time step in the forward GRU. This means the input to the forward GRU at the current moment. It is the hidden state of the forward GRU at the current moment.

[0094]

[0095] In equation (3), GRU backward It is the reverse GRU of a bidirectional GRU network. It is the hidden state of the previous time step in the reverse GRU. This means that the input to the inverse GRU at the current moment, where n is the number of frames extracted from the video. It is the hidden state of the reverse GRU at the current moment.

[0096] The average pooling operation is performed on all hidden state outputs of the bidirectional gated recurrent unit network as shown in Equation (4).

[0097]

[0098] In equation (4), n is the number of frames extracted from the input video, and h t f1 is the hidden state of the bidirectional GRU network at time t, and f2 is the time-domain feature.

[0099] In step S100, this disclosure extracts multiple video frames. However, these frames are independent of each other and lack interaction. The GRU network is suitable for handling this kind of temporal problem. Since this disclosure only extracts n frames in total, there will be no long dependency problem. The video frames are fed into the GRU network in sequence to model the temporal relationship between the video frames. The hidden state of the GRU network contains the temporal relationship between multiple video frames. This disclosure obtains temporal features by performing average pooling on the hidden state of the GRU network. When humans remember videos, they remember the development process of events in the video, which is exactly what the temporal features express.

[0100] Step S203: The output of each hidden state of the bidirectional gated recurrent unit network is processed by multiple one-dimensional convolutional neural networks with different kernel sizes but the same number of kernels, processed by nonlinear activation functions and average pooling operations to obtain multiple local features with different receptive fields. The multiple local features with different receptive fields are concatenated to obtain the local features f3 of the input video.

[0101] Specifically, the hidden state output of each bidirectional gated recurrent unit network in step S202 can be processed by four one-dimensional convolutional neural networks with kernel sizes of 2, 3, 4, and 5 and a kernel number of 512, processed by nonlinear activation functions (e.g., ReLU), and average pooling operations to obtain multiple local features with different receptive fields, namely [m2, m3, m4, m5], as shown in Equation (5). These multiple local features with different receptive fields are then concatenated as the final local feature f3.

[0102] m k =mean-pooling(ReLU(Conv1d(H))) (5)

[0103] In equation (5), k represents the size of different convolutional kernels, H is composed of the hidden states of the bidirectional GRU at each time step S202, i.e., H is [h1, h2, ..., h8], m k These are local features obtained from a one-dimensional convolutional neural network with a kernel size of k and a kernel count of 512.

[0104] One-dimensional convolutional neural networks can capture detailed information between video frame-level features at different scales in a domain. This disclosure uses one-dimensional convolutional neural networks of various scales to extract local details at different scales. Then, the ReLU nonlinear activation function is used to increase the nonlinear expressive power of the network. The resulting local features can effectively reflect the local information in the video. When humans remember a video, they remember the local details in the video, which can be reflected by the local features of this disclosure.

[0105] Step S204: Concatenate the global feature f1 obtained in step S201, the temporal feature f2 obtained in step S202, and the local feature f3 obtained in step S203 to generate the final visual multi-level coding feature f.

[0106] Concatenation is the simplest feature-level fusion method, and features at different levels carry different aspects of information that humans can remember from a video. Therefore, this disclosure can achieve visual multi-level encoding of multiple video-level features with low complexity.

[0107] In some implementations, an exemplary process for obtaining the text features of the text paired with the input video in step S300 may include the following steps:

[0108] Step b1: Clean the sentences in the text paired with each input video and store the sentences in the form of a list;

[0109] Here, text cleaning can include operations such as converting uppercase letters to lowercase letters, removing punctuation marks, and separating words. Taking sentences as an example, storing the cleaned text as a list means storing the cleaned sentences in the text as a list in Python.

[0110] Because the text annotated by the experimenters has various grammatical and punctuation problems that are not conducive to computer processing, this disclosure first cleans the text to form a uniform format that is conducive to computer processing before extracting text features, thereby improving the efficiency and accuracy of text feature extraction.

[0111] Step b2: After adding the [CLS] identifier to the list obtained in step Sb1, input it into the pre-trained BERT model, and take the vector corresponding to the [CLS] identifier in the last layer of the BERT model as the sentence feature f. t The sentence feature f t This refers to the text features of the text paired with the input video.

[0112] Here, the BERT model is a pre-trained BERT model on a large corpus. Based on the Transformer architecture, the BERT model can capture bidirectional relationships within sentences. Pre-trained on a large corpus, it achieved state-of-the-art performance on 11 natural language processing tasks, demonstrating its strong transfer learning capabilities. This disclosure uses a pre-trained BERT model to effectively extract global semantic information from the text, where [CLS] represents the corresponding vector. This allows the extracted text features to better represent the global semantic information of the text paired with the input video, thus more accurately capturing the true information of the input video.

[0113] In some implementations, an exemplary process for extracting the optical flow map of the input video and obtaining the optical flow features of the optical flow map in step S400 may include the following steps:

[0114] Step c1: Use the TV-L1 method to extract the dense optical flow of the input video and save the optical flow images in the vertical and horizontal directions.

[0115] The TV-L1 method is a classic optical flow image extraction method, and the extracted optical flow has good results. This disclosure uses TV-L1 to extract optical flow images, which is not only easy to implement, but also has good compatibility.

[0116] Step c2: Input the optical flow image extracted in step c1 into the I3D model to extract features, and take the output vector of the Mixed_5c layer as the optical flow feature f of the input video. o .

[0117] The I3D model is a pre-trained I3D model on the ImageNet and Kinetics datasets. This disclosure uses the I3D model, which achieves good performance on the large-scale action recognition dataset Kinetics, to extract optical flow features. By inputting dense optical flow, this disclosure can more accurately capture motion information in the input video and generate optical flow features that can characterize the motion in the video.

[0118] Figure 3 A schematic diagram illustrating an exemplary implementation flow of text guidance and visual enhancement in step S500 is shown. Figure 3 As shown, the visual enhancement of text guidance in step S500 may include the following steps:

[0119] Step S501: Divide the visual multi-level coding feature f generated in step S204 into multiple segments [x1, x2, ..., x...]. L The segmented fragments are then sequentially passed through a fully connected layer, a ReLU activation function, and another fully connected layer to map them into the visual-text common space, as shown in Equation (6):

[0120]

[0121] In equation (6), x i It is the i-th visual multi-level encoded feature fragment among multiple fragments obtained by segmenting visual multi-level encoded features, U v W v These are the weights of the two fully connected layers on the visual side. These are the mapped visual features.

[0122] This liter divides the visual multi-level coding features extracted in step S204 into multiple segments to calculate the importance of each segment to video memorability, and maps each segment to the visual-text common space to calculate the importance score. The structure of fully connected layers, ReLU activation function, and fully connected layers ensures that visual features can be mapped to the same semantic space as text features.

[0123] Step S502: The text features are mapped to the visual-text common space through another fully connected layer, ReLU activation function, and fully connected layer, as shown in Equation (7):

[0124]

[0125] In equation (7), f t These are the sentence features extracted in step b2, U t W t These are the weights of the two fully connected layers on the text side. These are the mapped text features.

[0126] Therefore, a structure similar to that in step S501 can be used to map text features into the visual-text common space to guide visual features.

[0127] Step S503: Add multiple segments of the mapped text features and mapped visual multilevel coding features, and then input them into the tanh activation function and fully connected layer to obtain the importance scores of each segment of the visual multilevel coding features;

[0128] The importance score of each segment can be obtained by equation (8):

[0129]

[0130] In equation (8), W is the weight of the fully connected layer, e i It is the current visual multi-level encoded feature fragment x i Importance score.

[0131] Within the visual-text common space, mapped text features and mapped visual features of multiple segments interact through a simple addition operation. An importance score of the current visual multi-level encoded feature segment is obtained through a tanh activation function and a fully connected layer. This importance score indicates the degree of importance of the current visual multi-level encoded feature segment to video memorability, guided by text features and combined with its own semantic information.

[0132] Step S504: Assign importance scores [e1, e2, ..., e] to each segment of the visual multi-level encoded features. LThe input softmax layer is mapped to the probability space to obtain the normalized importance scores [α1, α2, ..., α] of each segment. L ].

[0133] The importance scores obtained through fully connected layers are not suitable for weighted summation. This disclosure maps them to the probability [0, 1] space through a softmax layer, ensuring that the sum is 1, which is suitable for weighted summation.

[0134] Step S505: Using the normalized importance score obtained in step S504 as the weight of the segment, the segments obtained by segmenting the visual multi-level coding features in step S501 are weighted and summed to generate text guidance visual features.

[0135] The weighted summation in step S505 is shown in equation (9):

[0136]

[0137] In equation (9), f v To generate visual features that guide the text, x i It is the i-th visual multi-level encoded feature fragment among multiple fragments obtained by segmenting visual multi-level encoded features, α i It is the i-th visual multi-level encoded feature fragment x i The normalized importance score.

[0138] Thus, guided by text features, and utilizing the consistency between text and visual semantic information, visual segment features with high similarity to text occupy a higher proportion, generating visual features that are more suitable for video memorability, i.e., text-guided visual features.

[0139] In some implementations, step S600 may include the following steps:

[0140] Step d1: Extract sentence features f from step b2. t Input text into a multilayer perceptron network to generate text memory prediction scores.

[0141] The text multilayer perceptron network can be composed of fully connected layers, ReLU activation function, Dropout layer, fully connected layers, ReLU activation function, Dropout layer, fully connected layers, and Sigmoid activation function, and is trained using mean squared error as the loss function.

[0142] The loss function of the text multilayer perceptron network is shown in equation (10):

[0143]

[0144] In equation (10), y is the actual memory score of the video, obtained by the experimenters through experiments. The memory score predicted by the text multilayer perceptron network;

[0145] Step d2: Input the optical flow features obtained in step c2 into the optical flow multilayer perceptron network to generate optical flow memory prediction scores;

[0146] Here, the structure and loss function of the optical flow multilayer perceptron network can be the same as those of the text multilayer perceptron network.

[0147] Step d3: Input the text-guided visual features from step S505 into the multilayer perceptron network of the video frame to generate a video frame memory prediction score.

[0148] Here, the structure and loss function of the video frame multilayer perceptron network can be the same as those of the text multilayer perceptron network.

[0149] In practical applications, steps d1, d2, and d3 can be executed synchronously or in a certain order. This disclosure does not restrict the execution order of steps d1, d2, and d3.

[0150] As can be seen from the above, since text features, optical flow features, and text-guided visual features already have sufficiently powerful feature representation capabilities, this disclosure can predict the memory score of the corresponding modality using a simple multilayer perceptron structure, and update the parameters of the corresponding multilayer perceptron network using mean squared error loss. In this way, processing efficiency can be further improved while ensuring high accuracy in video memory prediction.

[0151] Figure 4 A flowchart illustrating the multimodal memory-based score fusion process in step S700 is shown. Figure 4 As shown, step S700 may include:

[0152] Step S701: Assign weights to the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score respectively to determine the weight allocation strategy; update the weight values ​​of the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score by a predetermined step size (e.g., 0.05) while ensuring that the sum of the weights is 1, so as to update the weight allocation strategy; evaluate each weight allocation strategy using the Spearman coefficient as an evaluation index, and determine the weight allocation strategy that can obtain the highest Spearman coefficient as the final weight allocation strategy; perform a weighted summation of the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score according to the final weight allocation strategy to obtain the video memory prediction score of the input video.

[0153] The weighting strategy includes the weight values ​​for text memory prediction scores, optical flow memory prediction scores, and video frame memory prediction scores.

[0154] The Spearman coefficients can be obtained from equation (11):

[0155]

[0156] In equation (11), N is the number of input videos in the dataset, and z i To rank the true memory scores of the input videos by the size of their values ​​in the dataset, The value of the video memory score obtained after this update is ranked in the dataset, where ρ is the Spearman coefficient.

[0157] Studies have shown that the aforementioned late fusion method is more suitable for video memory prediction tasks. This disclosure also integrates memory scores from multiple modalities based on the idea of ​​late fusion, but uses an adaptive weight allocation strategy to assign higher weights to modalities that play a greater role in predicting memory scores. This is more conducive to improving the accuracy and precision of video memory prediction.

[0158] Figures 5a to 5f Different video frames of the input video are shown. Using the technical solution of this disclosure, video memory prediction is performed on the input video, yielding the following results: Figure 5a The video memory prediction score for the video frame shown is 0.652. Figure 5b The video memory prediction score for the video frame shown is 0.877. Figure 5c The video memory prediction score for the video frame shown is 0.936. Figure 5d The video memory prediction score for the video frame shown is 0.724. Figure 5e The video memory prediction score for the video frame shown is 0.843. Figure 5f The video memory prediction score for the shown video frame is 0.902.652. Figure 5a The video memory prediction score for the video frame shown is 0.652.

[0159] This disclosure provides a video memorability prediction method based on visual multi-level coding and text-guided visual enhancement. Visual multi-level coding extracts global, temporal, and local features. Global features reflect the overall impression people have of a video when they remember it; temporal features show the progression of events over time; and local features reveal specific details. Given that text is a high-level summary of visual content and that there is semantic consistency between text and video, when annotating text, people tend to focus on important or interesting visual information in the video—often highly memorable segments. This means that the semantic information covered by the text often corresponds to highly memorable video segments, indicating that text-guided visual features are more relevant to visual memorability. Therefore, this disclosure uses text-guided visual enhancement before determining the video memorability prediction score. Thus, by fully extracting video features and leveraging the consistency between text and visual elements, this disclosure effectively improves the accuracy and precision of video memorability prediction.

[0160] The technical effects of this disclosure are explained below with reference to experimental results:

[0161] For example, the technical solution disclosed herein is tested on the Memento10k dataset, which includes 10,000 videos, each accompanied by a text description. The dataset is fixedly divided into 7,000 videos as the training set, 1,500 videos as the validation set, and 1,500 videos as the test set. Each video is labeled with a true video memory score, which is generated by an experiment called The Memento Video Memory Game. In short, if 70 out of 100 people can recall watching a video after watching it again, the true video memory score of that video is defined as 0.7.

[0162] Table 1 shows a comparison of the results of this disclosure and existing techniques on the Memento10k dataset, using the Spearman coefficient as the evaluation metric. The technical solution of this disclosure achieves a higher Spearman coefficient, which is the effect of visual multilevel coding and text-guided visual enhancement. This demonstrates that visual features enhanced by visual multilevel coding and text-guided visual enhancement are more suitable for video memory prediction, significantly improving the accuracy and precision of video memory prediction.

[0163] Method Name Spearman coefficient This disclosure 0.684 Existing methods 0.663

[0164] Table 1

[0165] Figure 6This is a schematic block diagram of a video memory prediction device 800 implemented using a hardware processing system according to one embodiment of the present disclosure.

[0166] The apparatus may include corresponding modules that perform one or more steps in the flowchart above. Therefore, each or more steps in the flowchart above can be performed by a corresponding module, and the apparatus may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0167] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 900 connects various circuits, including one or more processors 1000, memory 1100, and / or hardware modules. Bus 900 can also connect various other circuits 1200, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0168] Bus 900 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0169] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0170] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0171] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0172] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0173] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware. The program can be stored in a readable storage medium, and when executed, the program includes one or a combination of the steps of the method implementation.

[0174] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0175] like Figure 6 As shown, the video memory prediction device 800 according to some embodiments of the present disclosure may include:

[0176] The video frame-level feature extraction module 802 is used to obtain multiple video frame-level features of the input video using multiple video frames;

[0177] The visual multi-level coding module 804 is used to perform visual multi-level coding on multiple video frame-level features of the input video to obtain global features, temporal features and local features, and to concatenate the global features, temporal features and local features to obtain the visual multi-level coded features of the input video.

[0178] The text feature extraction module 806 is used to obtain the text features of the text paired with the input video;

[0179] Optical flow feature extraction module 808 is used to extract the optical flow map of the input video and obtain the optical flow features of the optical flow map;

[0180] The text-guided visual feature module 810 is used to guide and enhance the visual multi-level coding features of the input video using the text features of the text paired with the input video, so as to obtain the text-guided visual features of the input video.

[0181] The multimodal memory score prediction module 812 is used to obtain the text memory prediction score of the text features, the video frame memory prediction score of the text-guided visual features, and the optical flow memory prediction score of the optical flow features.

[0182] The multimodal memory score fusion module 814 is used to fuse the text memory prediction score, optical flow memory prediction score and video frame memory prediction score using a late fusion method based on an adaptive strategy to obtain the video memory prediction score of the input video.

[0183] In some implementations, the visual multi-level coding module 804 is specifically used to obtain the global features by performing an average pooling operation on multiple video frame-level features of the input video.

[0184] In some implementations, the visual multi-level coding module 804 is specifically used to obtain the temporal features by: feeding multiple frequency frame-level features of the input video into a bidirectional gated recurrent unit network, extracting each hidden state output of the bidirectional gated recurrent unit network, and performing an average pooling operation on all hidden state outputs of the bidirectional gated recurrent unit network to obtain the temporal features.

[0185] In some implementations, the visual multi-level coding module 804 is specifically used to obtain the local features by: processing each hidden state output of the bidirectional gated recurrent unit network sequentially through multiple one-dimensional convolutional neural networks with different kernel sizes and the same number of kernels, processing with nonlinear activation functions, and performing average pooling operations to obtain multiple local features with different receptive fields, and then concatenating the multiple local features with different receptive fields.

[0186] In some implementations, the text-guided visual feature module 810 is specifically used to obtain the text-guided visual features of the input video in the following manner:

[0187] The visual multi-level coding features are segmented into multiple segments, and the segmented segments are sequentially passed through a fully connected layer, a ReLU activation function, and another fully connected layer to map them into the visual-text common space.

[0188] The text features are mapped to the visual-text common space through a fully connected layer, a ReLU activation function, and another fully connected layer.

[0189] The importance scores of each segment are obtained by adding the mapped text features and the mapped visual multi-level encoded features together and then processing them sequentially through the tanh activation function and a fully connected layer.

[0190] The importance scores of each segment of the visual multi-level coding feature are mapped to the probability space to obtain the normalized importance scores of each segment.

[0191] Using the normalized importance score as the weight of the segment, the segments of the visual multi-level coding feature are weighted and summed to generate the text-guided visual feature.

[0192] In some implementations, the multimodal memory score prediction module 812 is specifically used to: process the optical flow features using an optical flow multilayer perceptron network to generate an optical flow memory prediction score; process the text features using a text multilayer perceptron network to generate a text memory prediction score; and process the text-guided visual features using a video frame multilayer perceptron network to generate a video frame memory prediction score; wherein the optical flow multilayer perceptron network, the text multilayer perceptron network, and the video frame multilayer perceptron network are all trained using mean squared error as the loss function.

[0193] In some implementations, the multimodal memory score fusion module 814 is specifically used to: assign weights to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score using a late fusion method based on an adaptive strategy; and perform a weighted summation of the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score using the assigned weights to obtain the video memory prediction score of the input video.

[0194] In some implementations, the multimodal memory score fusion module 814 is specifically used to allocate weights in the following manner:

[0195] Weights are assigned to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score respectively to determine the weight allocation strategy;

[0196] The weighting strategy is updated by updating the weight values ​​of the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score according to a predetermined step size.

[0197] Each weight allocation strategy is evaluated using the Spearman coefficient as an evaluation metric, and the weight allocation strategy that achieves the highest Spearman coefficient is determined as the final weight allocation strategy.

[0198] This disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor or other hardware module executing the execution instructions stored in the memory, causing the processor or other hardware module to perform the above-described video memory prediction method.

[0199] This disclosure also provides a readable storage medium storing execution instructions, which, when executed by a processor, are used to implement the above-described video memory prediction method.

[0200] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0201] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0202] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A video memory prediction method, characterized in that, include: Multiple video frame-level features of the input video are obtained using multiple video frames of the input video; Visual multi-level encoding is performed on multiple video frame-level features of the input video to obtain global features, temporal features and local features, and the global features, temporal features and local features are concatenated to obtain the visual multi-level encoded features of the input video; Obtain the text features of the text paired with the input video; Extract the optical flow map from the input video and obtain the optical flow features of the optical flow map; The text features of the text paired with the input video are used to guide and enhance the visual multi-level coding features of the input video, thereby obtaining the text-guided visual features of the input video. Obtain the text memory prediction score of the text features, the video frame memory prediction score of the text-guided visual features, and the optical flow memory prediction score of the optical flow features; An adaptive late-stage fusion method fuses the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score to obtain the video memory prediction score of the input video. The global features are obtained by performing average pooling on multiple video frame-level features of the input video. The temporal features are obtained by feeding multiple frequency frame-level features of the input video into a bidirectional gated recurrent unit (BRN) network, extracting each hidden state output of the BRN network, and performing average pooling on all hidden state outputs of the BRN network to obtain the temporal features. The local features are obtained by sequentially processing each hidden state output of the BRN network through multiple one-dimensional convolutional neural networks with different kernel sizes and the same number of kernels, processing with a nonlinear activation function, and performing average pooling to obtain multiple local features with different receptive fields, and then concatenating the multiple local features with different receptive fields.

2. The video memory prediction method according to claim 1, characterized in that, The step of obtaining the text features of the text paired with the input video includes: Clean the sentences in the text paired with each input video and store the sentences in the form of a list; After adding a predetermined identifier to the list, the data is input into the BERT model. The vector corresponding to the predetermined identifier in the last layer of the BERT model is taken as the sentence feature. The sentence feature is the text feature of the text paired with the input video.

3. The video memory prediction method according to claim 1, characterized in that, The step of guiding and enhancing the visual multi-level coding features of the input video using the text features of the text paired with the input video includes: The visual multi-level coding features are segmented into multiple segments, and the segmented segments are sequentially passed through a fully connected layer, a ReLU activation function, and another fully connected layer to map them into the visual-text common space. The text features are mapped to the visual-text common space through a fully connected layer, a ReLU activation function, and another fully connected layer. After adding multiple segments of the mapped text features and mapped visual multi-level encoded features, the sum is processed sequentially through the tanh activation function and a fully connected layer to obtain the importance score of each segment. The importance scores of each segment of the visual multi-level coding feature are mapped to the probability space to obtain the normalized importance scores of each segment. Using the normalized importance score as the weight of the segment, the segments of the visual multi-level coding feature are weighted and summed to generate the text-guided visual feature.

4. The video memory prediction method according to claim 1, characterized in that, The acquisition of the text memory prediction score of the text features, the video frame memory prediction score of the text-guided visual features, and the optical flow memory prediction score of the optical flow features includes: The optical flow features are processed using an optical flow multilayer perceptron network to generate an optical flow memory prediction score; The text features are processed using a text multilayer perceptron network to generate a text memory prediction score; The visual features of the text guidance are processed using a video frame multilayer perceptron network to generate a video frame memory prediction score. The optical flow multilayer perceptron network, the text multilayer perceptron network, and the video frame multilayer perceptron network are all trained using mean square error as the loss function.

5. The video memory prediction method according to claim 4, characterized in that, The late-stage fusion method based on an adaptive strategy fuses the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score, including: assigning weights to the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score using the late-stage fusion method based on an adaptive strategy; and performing a weighted summation of the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score using the assigned weights to obtain the video memory prediction score of the input video.

6. The video memory prediction method according to claim 5, characterized in that, The method for assigning weights to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score using an adaptive strategy-based late fusion method includes: assigning weights to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score respectively to determine a weight allocation strategy; updating the weight values ​​of the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score according to a predetermined step size to update the weight allocation strategy; and evaluating each weight allocation strategy using the Spearman coefficient as an evaluation metric, and determining the weight allocation strategy that achieves the highest Spearman coefficient as the final weight allocation strategy.

7. A video memory prediction device, characterized in that, include: A video frame-level feature extraction module is used to obtain multiple video frame-level features of the input video using multiple video frames; A visual multi-level coding module is used to perform visual multi-level coding on multiple video frame-level features of the input video to obtain global features, temporal features and local features, and to concatenate the global features, temporal features and local features to obtain the visual multi-level coded features of the input video; The text feature extraction module is used to obtain the text features of the text paired with the input video; An optical flow feature extraction module is used to extract the optical flow map of the input video and obtain the optical flow features of the optical flow map; The text-guided visual feature module is used to guide and enhance the visual multi-level coding features of the input video using the text features of the text paired with the input video, so as to obtain the text-guided visual features of the input video. A multimodal memory score prediction module is used to obtain the text memory prediction score of the text features, the video frame memory prediction score of the text-guided visual features, and the optical flow memory prediction score of the optical flow features. A multimodal memory score fusion module is used to fuse the text memory prediction score, optical flow memory prediction score, and video frame memory prediction score using a late-stage fusion method based on an adaptive strategy, to obtain the video memory prediction score of the input video. Specifically, the visual multi-level coding module is used to obtain the global features by performing average pooling on multiple video frame-level features of the input video; the visual multi-level coding module is used to obtain the temporal features by: feeding multiple frequency frame-level features of the input video into a bidirectional gated recurrent unit network, extracting each hidden state output of the bidirectional gated recurrent unit network, and performing average pooling on all hidden state outputs of the bidirectional gated recurrent unit network to obtain the temporal features; the visual multi-level coding module is used to obtain the local features by: processing each hidden state output of the bidirectional gated recurrent unit network sequentially through multiple one-dimensional convolutional neural networks with different kernel sizes and the same number of kernels, processing with a nonlinear activation function, and performing average pooling to obtain multiple local features with different receptive fields, and concatenating the multiple local features with different receptive fields.

8. The video memory prediction device according to claim 7, characterized in that, The text-guided visual feature module is specifically used to obtain the text-guided visual features of the input video in the following manner: The visual multi-level coding features are segmented into multiple segments, and the segmented segments are sequentially passed through a fully connected layer, a ReLU activation function, and another fully connected layer to map them into the visual-text common space. The text features are mapped to the visual-text common space through a fully connected layer, a ReLU activation function, and another fully connected layer. After adding multiple segments of the mapped text features and mapped visual multi-level encoded features, the sum is processed sequentially through the tanh activation function and a fully connected layer to obtain the importance score of each segment. The importance scores of each segment of the visual multi-level coding feature are mapped to the probability space to obtain the normalized importance scores of each segment. Using the normalized importance score as the weight of the segment, the segments of the visual multi-level coding feature are weighted and summed to generate the text-guided visual feature.

9. The video memory prediction device according to claim 8, characterized in that, The multimodal memory score prediction module is specifically used to: process the optical flow features using an optical flow multilayer perceptron network to generate an optical flow memory prediction score; process the text features using a text multilayer perceptron network to generate a text memory prediction score; and process the text-guided visual features using a video frame multilayer perceptron network to generate a video frame memory prediction score; wherein the optical flow multilayer perceptron network, the text multilayer perceptron network, and the video frame multilayer perceptron network are all trained using mean squared error as the loss function.

10. The video memory prediction device according to claim 9, characterized in that, The multimodal memory score fusion module is specifically used to: assign weights to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score using a late fusion method based on an adaptive strategy; and to perform a weighted summation of the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score using the assigned weights to obtain the video memory prediction score of the input video.

11. The video memory prediction device according to claim 10, characterized in that, The multimodal memory score fusion module is specifically used to allocate weights in the following manner: assigning weights to the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score respectively to determine a weight allocation strategy; and updating the weight values ​​of the text memory prediction score, the optical flow memory prediction score, and the video frame memory prediction score according to a predetermined step size to update the weight allocation strategy. Each weight allocation strategy is evaluated using the Spearman coefficient as an evaluation metric, and the weight allocation strategy that achieves the highest Spearman coefficient is determined as the final weight allocation strategy.

12. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the video memory prediction method according to any one of claims 1 to 6.

13. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the video memory prediction method according to any one of claims 1 to 6.

Images