A video description method and system for spatio-temporal feature semantic modeling of a video

Abstract

The application discloses a video description method and system for video space-time feature semantic modeling, and belongs to the technical field of computer vision, and comprises the following steps: S1, feature extraction; S2, feature fusion; S3, querying corresponding long-short term time sequence motion features from CLIP features, indirectly learning potential semantics of the long-short term time sequence motion features, and obtaining action features with semantic information; S4, using object features and action features to construct object association graphs and action association graphs respectively, then fusing the object association graphs and the action association graphs to construct object-action association graphs; and S5, feeding object-action association graph data into a GraphTransformer for processing to obtain object-action association tensors, then feeding the object-action association tensors and the CLIP features into a Transformer network for training to generate a final video description for video space-time feature semantic modeling.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a video description method and system for spatiotemporal feature semantic modeling of video. Background Technology

[0002] With the widespread adoption of digital video equipment and the explosive growth of internet video content, users face a massive amount of video data every day, leading to a continuous increase in the demand for video content retrieval and browsing. Traditional text retrieval technologies cannot meet users' needs for video content queries. Therefore, a technology is needed that can convert video content into text descriptions so that users can find videos of interest through text retrieval. In addition, video description technology based on spatiotemporal feature semantic modeling can also be applied in fields such as assisting the blind and disabled, human-computer interaction and intelligent systems, and AI-powered automatic narration.

[0003] Traditional video description methods for spatiotemporal feature semantic modeling utilize template-based and retrieval-based methods. With the rapid development of deep learning-related research, more and more researchers are using deep learning-based methods to conduct research. Deep learning-based research methods can be divided into three main categories: video description methods based on encoder-decoder spatiotemporal feature semantic modeling, video description methods based on attention, and video description methods based on graph neural networks.

[0004] All three types of methods mentioned above suffer from the problem that the generated descriptions do not match the actual video content. This is mainly reflected in the fact that the descriptions of the main activities of objects in the video remain at the preliminary semantic level, without considering the context of the actions in the video frame sequence or the potential deeper semantic information.

[0005] Therefore, there is an urgent need to provide a video description method based on temporal motion feature modeling and latent semantic extraction for video spatiotemporal feature semantic modeling to solve the above problems. Summary of the Invention

[0006] The technical solution, in order to solve the technical problems mentioned above, is as follows: Based on temporal motion feature modeling and latent semantic extraction technology, at the motion feature level, long-term temporal motion features are extracted through a self-attention mechanism, and short-term temporal motion features are extracted through a cross-attention mechanism. Then, the long-term and short-term temporal motion features are fused to perform long- and short-term modeling of the motion features. At the latent semantic extraction level, the features most relevant to the long- and short-term temporal motion features are queried from the multimodal feature CLIP to implicitly obtain the latent semantic information of the long- and short-term temporal motion features, thereby obtaining action features with semantic information.

[0007] The first objective of this invention is to provide a video description method for spatiotemporal feature semantic modeling, comprising the following steps: S1. Feature extraction; the features include object features, motion features, and CLIP features; S2, Feature Fusion; specifically including: Long-term temporal modeling of motion features in video frames is performed using a self-attention mechanism to obtain long-term temporal motion features. Short-term temporal modeling of motion features of adjacent video frames is performed using a cross-attention mechanism to obtain short-term temporal motion features. The long-term and short-term temporal motion features are fused to obtain long-short-term temporal motion features; S3. Using long and short-term temporal motion features as the query for alignment attention, and CLIP features as the key and value for alignment attention, the corresponding long and short-term temporal motion features are queried from the CLIP features, and the latent semantics of the long and short-term temporal motion features are indirectly learned to obtain action features with semantic information. S4. Construct object association graphs and action association graphs using object features and action features respectively, and then merge the object association graphs and action association graphs to construct object-action association graphs; S5. Input the object action association graph data into GraphTransformer to obtain the object action association tensor. Then, input the object action association tensor and CLIP features into the Transformer network for training to generate the final video description for video spatiotemporal feature semantic modeling.

[0008] Preferably, during the feature extraction process: The process of obtaining the object features is as follows: First, extract T video frames from the original video, T∈[20, 40]. Use an object detection model to detect bounding boxes in each frame. Then, use RoIAlign technology to extract object features from each bounding box. At the same time, set the maximum number of bounding boxes in each frame to Nt, where Nt is a natural number greater than 0. Filter object features based on confidence level. If the number of objects detected in a frame is not less than Nt, only the Nt objects with the highest confidence level are retained. Meanwhile, objects with a confidence level lower than 0.5 are filtered out. If the number of objects detected in a frame is less than Nt, zero-value row vectors are used to fill the frame, ensuring that the number of objects in each frame is Nt. The process of obtaining the motion features is as follows: the motion features of each overlapping frame sequence of length K are extracted using the action recognition model, where K is a natural number greater than 0. For each sequence of length K in T frames, motion features are extracted from the last pooling layer of the action recognition model to obtain a set of motion features M={m1,m2,...,mT}, where mT represents the motion features extracted from the Tth frame, and M represents all motion features of T video frames in a video. The CLIP feature acquisition process is as follows: the video description dataset for video spatiotemporal feature semantic modeling is fed into the visual-text retrieval model for video-text retrieval tasks, a contrastive language-image pre-trained model for video description for video spatiotemporal feature semantic modeling is retrained, and then visual-text aligned CLIP features are extracted from the contrastive language-image pre-trained model.

[0009] Preferably, the feature fusion process is as follows: The motion feature sequence M={m1,m2,...,mT} is used as input. The self-attention mechanism is used to perform weighted calculation between the motion features of the current video frame and the motion features of the remaining video frames. This allows the model to take into account the global information of the entire sequence when calculating the attention weights, thereby capturing the features of the long-term context. This enables long-term temporal modeling of the motion features, resulting in long-term temporal motion features. The motion feature sequence M={m1,m2,...,mT} is used as input. The cross-attention mechanism is used to perform weighted calculation on the motion features of the current video frame and the motion features of the adjacent video frames. The feature information of the adjacent frames is weighted and converged into the motion features of the current video frame. The weighted convergence process enables the model to learn the motion features from the adjacent video frames in the motion features of the current video frame, thereby capturing short-term context features. This allows for short-term temporal modeling of the motion features, resulting in short-term temporal motion features. The long-term and short-term temporal motion features are concatenated to obtain the merged long-short-term temporal motion features. These features are then input into a fully connected layer for further processing to complete subsequent tasks.

[0010] Preferably, S3 includes: First, the long-term and short-term temporal motion features are used as the Query, and the CLIP features are used as the Key and Value. The features corresponding to the long-term and short-term temporal motion features are retrieved from the CLIP features. Then, the degree of correlation between the long-term and short-term temporal motion features and the CLIP features is measured by calculating the attention score between the Query and the Key. Subsequently, the attention score is normalized by the softmax function to obtain the attention weight between the Query and the Value. The Value is then weighted and aggregated according to the obtained attention weight, thereby implicitly incorporating the latent semantic information related to the long-term and short-term temporal motion features to obtain action features with semantic information.

[0011] Preferably, S4 includes: S401. Using an object detection model, extract Nt object features in each video frame as nodes of a graph, with object features as node attributes. Use a double-loop loop to traverse the relationships between the Nt objects in the current video frame and the Nt objects in adjacent video frames, calculate the similarity between objects, and use it as edges of the graph to measure the correlation between objects in the video frame sequence, construct an object relationship matrix, and thus obtain an object association graph. At the same time, construct hash table one, using video ID and video frame ID as keys and object features as values. S402. The above action features are represented as nodes. Then, based on the continuity of time, the similarity between action features between adjacent video frames is calculated and used as edges of the graph to measure the correlation of action features in the video frame sequence. An action relationship matrix is ​​constructed to obtain an action association graph. At the same time, a second hash table is constructed, with video ID and video frame ID as keys and action features as values. S403. Based on the video ID and video frame ID, find the corresponding object features and action features in the two hash tables, and then use the object features as nodes and the action features as edges to obtain the object action association graph.

[0012] Preferably, S5 includes: The object action relationship graph is fed into the GraphTransformer for processing. Each node is mapped to a vector representation containing the feature information of that node. At the same time, each edge in the graph is also mapped to a vector representation to represent the relationship or connection between nodes. These vector representations of nodes and edges are combined to form the object action relationship tensor that the Transformer can encode, which is then passed as input to the subsequent Transformer model. The object action association tensor and CLIP features are fed into the Transformer for encoding and decoding to obtain the final video description based on the spatiotemporal semantic model of the video.

[0013] The second objective of this patent is to provide a video description system for spatiotemporal feature semantic modeling, comprising: The feature extraction module includes object features, motion features, and CLIP features. The feature fusion module includes the following fusion process: Long-term temporal modeling of motion features in video frames is performed using a self-attention mechanism to obtain long-term temporal motion features. Short-term temporal modeling of motion features of adjacent video frames is performed using a cross-attention mechanism to obtain short-term temporal motion features. The long-term and short-term temporal motion features are fused to obtain long-short-term temporal motion features; The semantic information acquisition module uses long and short-term temporal motion features as the query for alignment attention and CLIP features as the key and value for alignment attention. It queries the corresponding long and short-term temporal motion features from the CLIP features, indirectly learns the latent semantics of the long and short-term temporal motion features, and obtains action features with semantic information. The object-action association graph construction module uses object features and action features to construct object association graphs and action association graphs respectively, and then merges the object association graphs and action association graphs to construct the object-action association graph. The output module feeds the object action association graph data into the GraphTransformer to obtain the object action association tensor. Then, the object action association tensor and CLIP features are fed into the Transformer network for training to generate the final video description for video spatiotemporal feature semantic modeling.

[0014] The third objective of this patent is to provide a computer program product, including a computer program that, when executed by a processor, implements the video description method for video spatiotemporal feature semantic modeling.

[0015] The fourth objective of this patent is to provide an information data processing terminal for implementing the video description method for spatiotemporal feature semantic modeling of the above-mentioned video.

[0016] The fifth objective of this patent is to provide a computer-readable storage medium, including instructions that, when executed on a computer, cause the computer to perform the video description method described above for video spatiotemporal feature semantic modeling.

[0017] The advantages and positive effects of this invention are as follows: This invention enables long and short-term sequence modeling of motion features at the temporal motion feature level, resulting in more fine-grained long and short-term temporal motion features. This not only helps the model to more accurately understand the contextual relationships of motion features in a video, but also helps the model capture subtle changes in motion features in the video, thereby gaining a more comprehensive understanding of motion features and their potential semantics. By using this temporal motion feature modeling approach, the model's understanding and prediction accuracy of motion features can be effectively improved, thereby enhancing the accuracy of video understanding and generating descriptions that are more consistent with the video content.

[0018] At the latent semantic level, this invention utilizes long and short-term temporal motion features to query CLIP features related to long and short-term temporal motion features, implicitly learning latent semantic information to obtain action features with latent semantics. In this way, not only can the action features be made as semantically consistent as possible with the text description, enhancing the consistency and relevance of cross-modal data, but the action features can also obtain richer semantic information, thereby more accurately expressing the meaning of the action. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a preferred embodiment of the present invention; Figure 2 This is a schematic diagram of the long-term self-attention structure for motion feature modeling in a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of the cross-attention structure for short-time modeling of motion features in a preferred embodiment of the present invention; Figure 4 This is a schematic diagram of the long- and short-time motion feature modeling structure in a preferred embodiment of the present invention; Figure 5 This is a schematic diagram of the motion feature-CLIP feature alignment attention structure in a preferred embodiment of the present invention. Detailed Implementation

[0020] To further understand the invention's content, features, and effects, the following embodiments are provided, and detailed descriptions are given in conjunction with the accompanying drawings.

[0021] One of the key aspects of video description tasks involving spatiotemporal feature semantic modeling lies in extracting and understanding the action features of objects in the video. These action features not only enrich the descriptive content but also enhance the model's ability to understand the video context and semantics. By analyzing the actions of objects, the model can more accurately infer their intentions, emotional states, and interactions with other objects, thereby generating more coherent and vivid descriptions. Figure 1As shown, this invention first uses self-attention to perform long-term modeling of motion features obtained from the action recognition model, and then uses cross-attention to perform short-term modeling. Then, the long and short-term features are fused to model the motion features in time sequence, resulting in long and short-term temporal motion features. In addition, an alignment attention is designed to learn the potential semantic information of long and short-term temporal motion features from the multimodal feature CLIP, resulting in action features containing semantic information. This allows for a better understanding of the actions in the video and enables a deeper understanding and description of the video content.

[0022] Please see Figures 1 to 5 .

[0023] First embodiment: A video description method based on spatiotemporal feature semantic modeling, comprising the following steps: S1. Feature Extraction; this involves obtaining object features, motion features, and features obtained through Contrastive Language-Image Pretraining (CLIP) from the video; specifically, it includes: Use object detection models to obtain object features in video frames; Motion features are extracted from frame sequences using an action recognition model; motion features refer to the raw features extracted from the action recognition model. CLIP features aligned with text (text descriptions corresponding to video segments in a video description dataset for video spatiotemporal feature semantic modeling) are extracted using a contrastive language-image pre-trained model. S2. Feature Fusion: This involves performing long-short-term temporal modeling on the motion features extracted from video frames to obtain long-short-term temporal motion features; specifically including: Long-term temporal modeling of motion features in video frames is performed using a self-attention mechanism to obtain long-term temporal motion features. Short-term temporal modeling of motion features of adjacent video frames is performed using a cross-attention mechanism to obtain short-term temporal motion features. The long-term and short-term temporal motion features are fused by a fusion layer to obtain long-term and short-term temporal motion features. S3. Using long-term and short-term temporal motion features as the query for alignment attention, and CLIP features as the key and value for alignment attention, the corresponding long-term and short-term temporal motion features are queried from the CLIP features to indirectly learn the latent semantics of the long-term and short-term temporal motion features, thus obtaining action features with semantic information. Action features refer to motion features that are temporally modeled from the original features and whose relevant semantic information is learned directly or indirectly. This step queries the corresponding long-term and short-term temporal motion features from the CLIP features to implicitly obtain the latent semantics of the long-term and short-term temporal motion features, thus obtaining action features with semantic information. S4. Construct object association graphs and action association graphs using object features and action features respectively, and then merge the object association graphs and action association graphs to construct object-action association graphs; S5. Input the object action association graph data into GraphTransformer to obtain the object action association tensor. Then, input the object action association tensor and CLIP features into the Transformer network for training to generate the final video description for video spatiotemporal feature semantic modeling.

[0024] To better understand the inventive concept of this application, each step is described in detail below; S1 specifically includes: S101. The process of obtaining the object features is as follows: First, extract T (T∈[20,40]) video frames from the original video. Use an object detection model (such as SSD, Faster R-CNN, YOLOv5, etc.) to detect bounding boxes in each frame. Then, use RoIAlign technology to extract object features from each bounding box. At the same time, set the maximum number of bounding boxes in each frame to Nt (Nt is a natural number greater than 0), and filter object features according to confidence. If the number of objects detected in a frame exceeds Nt, only the Nt objects with the highest confidence are retained; at the same time, objects with a confidence of less than 0.5 are also filtered out. If the number of objects detected in a frame is less than Nt, use zero-value row vectors to fill the gaps, ensuring that the number of objects in each frame is Nt. A set of object features O={o Nt 1,o Nt 2,...o Nt T}, where o Nt T This represents the Nt object features extracted from the T-th frame, and O represents the object features extracted from the entire video.

[0025] S102. The process of acquiring the motion features is as follows: using an action recognition model (such as I3D, C3D, R3D, etc.) to extract motion features of each overlapping frame sequence of length K (K is a natural number greater than 0, generally 2). For each sequence of length K in T frames, motion features are extracted from the last pooling layer of the model to obtain a set of motion features M={m1,m2,...,mT}, where mT represents the motion features extracted in the Tth frame, and M represents all motion features of T video frames in a video.

[0026] S103: The process of obtaining the CLIP features is as follows: The video description dataset (referring to a dataset containing video segments and corresponding text descriptions) used for video spatiotemporal feature semantic modeling is fed into a visual-text retrieval model for video-text retrieval tasks. A contrastive language-image pre-trained model for video descriptions used for video spatiotemporal feature semantic modeling is retrained. Then, using the contrastive language-image pre-trained model, visual-text aligned CLIP features of T frames are extracted, C={c1,c2,...,c...} T}, where C represents the CLIP feature extracted given T frames.

[0027] S2 specifically includes: S201. Long-term sequence modeling: The motion feature sequence M={m1,m2,...,mT} is used as input. The self-attention mechanism is used to perform weighted calculation between the motion features of the current video frame and the motion features of the remaining video frames. This allows the model to consider the global information of the entire sequence when calculating the attention weights, thereby capturing the features of the long-term context. This is used to perform long-term temporal modeling of the motion features and obtain long-term temporal motion features.

[0028] For example: The initial action features M = {m1, m2, ..., m...} T} as the input sequence, where m1, m2, ..., m T These are the elements in the input sequence. First, we define three weight matrices W. q W k and W v These are used to compute queries, keys, and values. For each input element m i We calculate its query Q using the following formula. i K i V i Q i =m i *W q K i =m i *W k V i= m i *W v。 Next, calculate each element m i With m j (m j The attention score (representing the remaining Ti features) is obtained by querying Q. i and K j The dot product of the transpose is used to obtain the attention score. long = Q i *K j TThen, adjust attention_scores long Multiply by the scaling factor, where the scaling factor is K. dim 0.5 (K) dim (K represents the feature dimension, here referring to action features), Attention_scores long =attention_scores long *scale 0.5 The attention weights are then obtained by normalization using the softmax function: attention_weights long =softmax{Attention_scores long Finally, calculate the output of all elements by comparing the attention score of each element with the corresponding V. i Multiply to obtain the final output motion. long =attention_weights long *V i M long ={m 1long ,m 2long ,...,m Tlong}

[0029] S202, Short-term sequence modeling: Taking the motion feature sequence M={m1,m2,...,mT} as input, the motion features of the current video frame are weighted and calculated with the motion features of adjacent video frames using a cross-attention mechanism. The feature information of adjacent frames is weighted and converged into the motion features of the current video frame. This weighted convergence process enables the model to learn the motion features from adjacent video frames in the motion features of the current video frame, thereby capturing short-term context features. This allows for short-term temporal modeling of motion features, resulting in short-term temporal motion features.

[0030] For example: Given the initial motion features M = {m1, m2, ..., m...} T As the input sequence, for short-time sequence modeling, we need to use the motion features of the current frame as Q to query similar features in the next frame. Therefore, we first define three weight matrices W. q W k and W v They are used to compute queries, keys, and values ​​for each input element m. i Q i =m i *W q ,K i =m i *W k Vi= m i *W v。 Then, calculate Q. i and K i+1 The dot product is used to obtain the attention score. short = Q i *K i+1 T Then, adjust attention_scores short Multiply by the scaling factor, where the scaling factor is K. dim 0.5 , (K dim (Attention_scores represents the feature dimension of K, specifically the action features) short =attention_scores short *scale 0.5 The attention weights are then obtained by normalization using the softmax function: attention_weights short =softmax{Attention_scores short Finally, calculate the output of all elements by comparing the attention score of each element with the corresponding V. i Multiply to obtain the final output motion. short =attention_weights short *V i , that is, M short ={m1 short m2 short ,...,ms Tshort}

[0031] S203, Long-term and short-term feature fusion: The long-term temporal motion features obtained by self-attention and the short-term temporal motion features obtained by cross-attention are concatenated to obtain the merged long-term and short-term temporal motion features. The long-term and short-term temporal motion features are then input into the fully connected layer for further processing to complete the subsequent tasks.

[0032] For example: the obtained M long Features and M short The features are fed into a fusion layer for concatenation, and then passed through a feedforward network layer to obtain the final long- and short-term temporal motion features action={a1,a2,...a... T}

[0033] S3 specifically includes: Using short-term and long-term temporal motion features as queries and CLIP features as keys and values, features corresponding to the short-term and long-term temporal motion features are retrieved from the CLIP features. Then, the degree of correlation between the short-term and long-term temporal motion features and CLIP features is measured by calculating the attention score between the query and the key. Next, the attention score is normalized using the softmax function to obtain the attention weight between the query and the value. The value (CLIP features) is then weighted and aggregated according to the obtained attention weight, thereby implicitly incorporating the latent semantic information related to the short-term and long-term temporal motion features to obtain action features with semantic information.

[0034] For example, the long and short time series motion features action={a1,a2,...a T As a query, the multimodal feature CLIP, C={c1,c2,...,c T Using these as keys and values, latent semantic information related to long- and short-term temporal motion features is learned from multimodal features. First, three weight matrices W are defined. q W k and W v These are used to compute queries, keys, and values. For each input element m i We calculate its query Q using the following formula. i K i V i Q i =m i *W q K i =c i *W k V i= c i *W v Here, the scaling factor is defined as scale=K. dim 0.5 (K) dim Let K represent the feature dimension (here referring to CLIP features). For each query, we have attention_score = (Q... i *K i T )*scale 0.5 Then, the attention weights are normalized using the softmax function to obtain the attention weights: attention_weight = softmax{attention_score, dim = -1}. Finally, the attention weights are compared with the corresponding V values. i Multiplying these together yields the final action feature: Action = {A1, A2, ..., A} T}

[0035] S4 specifically includes: S401. Using an object detection model, extract Nt object features from each video frame, which serve as nodes in a graph. Object features are treated as node attributes. A double-loop iterates through the relationships between the Nt objects in the current video frame and the Nt objects in adjacent video frames, calculating the similarity between objects and using this similarity as edges in the graph. This measures the correlation between objects in the video frame sequence, constructing an object relationship matrix to obtain the object association graph. Simultaneously, a hash table is constructed, using the video ID and video frame ID as keys and object features as values.

[0036] S402. Using the motion features obtained in S3 as nodes, and based on temporal continuity, calculate the similarity between motion features of adjacent video frames, using this similarity as edges of the graph to measure the correlation of motion features in the video frame sequence, thus constructing an motion relationship matrix to obtain an motion association graph. Simultaneously, construct hash table two, using video ID and video frame ID as keys and motion features as values.

[0037] S403. Based on the video ID and video frame ID, find the corresponding object features and action features in the two hash tables, and then use the object features as nodes and the action features as edges to obtain the object action association graph.

[0038] For example: constructing an object action relationship graph As explained above, the number of objects extracted in each video frame is the same. Therefore, between adjacent frames, we establish the relationship between objects by calculating the feature similarity of nodes between adjacent frames:

[0039] Among them, G object tij This represents the adjacency matrix of the object action relationship graph between adjacent frames, cos(o i t ,o j t+1 The ) represents the similarity between objects i and j.

[0040] Construction of Action Relationship Graph The final action feature Action={A1,A2,...A3} is obtained from S3. T Similarly, by calculating the relationships between actions in adjacent frames, the adjacency matrix of the action association graph is obtained:

[0041] Among them, G Action tij This represents the action adjacency matrix between adjacent frames, cos(Ai t A j t+1 ) represents action characteristic A i and A j The similarity between them.

[0042] Construction of object action relationship graph Merging the object action relationship graph and the action relationship graph together yields a new object action relationship graph G. Action object :

[0043] S5 specifically includes: S501. The object action relationship graph is fed into the GraphTransformer for processing. Each node is mapped to a vector representation containing its feature information. Simultaneously, each edge in the graph is also mapped to a vector representation, indicating the relationship or connection between nodes. These node and edge vector representations are combined to form the object action relationship tensor that the Transformer can encode, which can be passed as input to subsequent Transformer models.

[0044] S502. Input the object action association tensor and CLIP features into the Transformer for encoding and decoding to obtain the final video description based on the spatiotemporal semantic model of the video.

[0045] For example: Graph data G Action object The data is fed into a Graphtransformer for processing, resulting in a set of object-action association tensors F containing object features and action features. g As a local feature, and simultaneously, the CLIP feature C obtained in steps 1-3 is used as a global feature. Therefore, the local feature F is... g Together with the global feature C, it is fed into the Transformer for encoding and decoding. Finally, words are continuously decoded to obtain the final description.

[0046] A video description system for video spatiotemporal feature semantic modeling, used to implement the video description method for video spatiotemporal feature semantic modeling in the first embodiment, includes: The feature extraction module includes the following extraction process: Use object detection models to obtain object features in video frames; Motion features are extracted from frame sequences using an action recognition model; Extract CLIP features aligned with text using a contrastive language-image pre-trained model; The feature fusion module includes the following fusion process: Long-term temporal modeling of motion features in video frames is performed using a self-attention mechanism to obtain long-term temporal motion features. Short-term temporal modeling of motion features of adjacent video frames is performed using a cross-attention mechanism to obtain short-term temporal motion features. The long-term and short-term temporal motion features are fused to obtain long-short-term temporal motion features; The semantic information acquisition module uses long and short-term temporal motion features as the query for alignment attention and CLIP features as the key and value for alignment attention. It queries the corresponding long and short-term temporal motion features from the CLIP features, indirectly learns the latent semantics of the long and short-term temporal motion features, and obtains action features with semantic information. The object-action association graph construction module uses object features and action features to construct object association graphs and action association graphs respectively, and then merges the object association graphs and action association graphs to construct the object-action association graph. The output module feeds the object action association graph data into the GraphTransformer to obtain the object action association tensor. Then, the object action association tensor and CLIP features are fed into the Transformer network for training to generate the final video description for video spatiotemporal feature semantic modeling.

[0047] An information data processing terminal for implementing the above-mentioned video description method for spatiotemporal feature semantic modeling.

[0048] A computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the video description method described above for video spatiotemporal feature semantic modeling.

[0049] This invention models motion features on a time series level at the level of object motion features to obtain long-term and short-term temporal motion features. At the level of latent semantic extraction, it combines CLIP features with long-term and short-term temporal motion features, and learns latent semantic information from CLIP features guided by long-term and short-term temporal motion features. This helps video description models that model spatiotemporal features to better understand the actions and behaviors in the video, and helps to achieve a deeper understanding and interpretation of video content.

[0050] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented, in whole or in part, as a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0051] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall fall within the scope of the technical solution of the present invention.

Claims

1. A video description method for spatiotemporal feature semantic modeling, characterized in that, Includes the following steps: S1. Feature extraction; the features include object features, motion features, and CLIP features; S2, Feature Fusion; specifically including: Long-term temporal modeling of motion features in video frames is performed using a self-attention mechanism to obtain long-term temporal motion features. Short-term temporal modeling of motion features of adjacent video frames is performed using a cross-attention mechanism to obtain short-term temporal motion features. The long-term and short-term temporal motion features are fused to obtain long-short-term temporal motion features; S3. Using long and short-term temporal motion features as the query for alignment attention, and CLIP features as the key and value for alignment attention, the corresponding long and short-term temporal motion features are queried from the CLIP features, and the latent semantics of the long and short-term temporal motion features are indirectly learned to obtain action features with semantic information. S4. Construct object association graphs and action association graphs using object features and action features respectively, and then merge the object association graphs and action association graphs to construct object-action association graphs; S5. Input the object action association graph data into GraphTransformer to obtain the object action association tensor. Then, input the object action association tensor and CLIP features into the Transformer network for training to generate the final video description for video spatiotemporal feature semantic modeling.

2. The video description method for spatiotemporal feature semantic modeling according to claim 1, characterized in that, During the feature extraction process: The process of obtaining the object features is as follows: First, extract T video frames from the original video, T∈[20, 40]. Use an object detection model to detect bounding boxes in each frame. Then, use RoIAlign technology to extract object features from each bounding box. At the same time, set the maximum number of bounding boxes in each frame to Nt, where Nt is a natural number greater than 0. Filter object features based on confidence level. If the number of objects detected in a frame is not less than Nt, only the Nt objects with the highest confidence level are retained. Meanwhile, objects with a confidence level lower than 0.5 are filtered out. If the number of objects detected in a frame is less than Nt, zero-value row vectors are used to fill the frame, ensuring that the number of objects in each frame is Nt. The process of obtaining the motion features is as follows: the motion features of each overlapping frame sequence of length K are extracted using the action recognition model, where K is a natural number greater than 0. For each sequence of length K in T frames, motion features are extracted from the last pooling layer of the action recognition model to obtain a set of motion features M={m1,m2,...,mT}, where mT represents the motion features extracted from the Tth frame, and M represents all motion features of T video frames in a video. The CLIP feature acquisition process is as follows: the video description dataset for video spatiotemporal feature semantic modeling is fed into the visual-text retrieval model for video-text retrieval tasks, a contrastive language-image pre-trained model for video description for video spatiotemporal feature semantic modeling is retrained, and then visual-text aligned CLIP features are extracted from the contrastive language-image pre-trained model.

3. The video description method for spatiotemporal feature semantic modeling according to claim 2, characterized in that, The feature fusion process is as follows: The motion feature sequence M={m1,m2,...,mT} is used as input. The self-attention mechanism is used to perform weighted calculation between the motion features of the current video frame and the motion features of the remaining video frames. This allows the model to take into account the global information of the entire sequence when calculating the attention weights, thereby capturing the features of the long-term context. This enables long-term temporal modeling of the motion features, resulting in long-term temporal motion features. The motion feature sequence M={m1,m2,...,mT} is used as input. The cross-attention mechanism is used to perform weighted calculation on the motion features of the current video frame and the motion features of the adjacent video frames. The feature information of the adjacent frames is weighted and converged into the motion features of the current video frame. The weighted convergence process enables the model to learn the motion features from the adjacent video frames in the motion features of the current video frame, thereby capturing short-term context features. This allows for short-term temporal modeling of the motion features, resulting in short-term temporal motion features. The long-term and short-term temporal motion features are concatenated to obtain the merged long-short-term temporal motion features. These features are then input into a fully connected layer for further processing to complete subsequent tasks.

4. The video description method for spatiotemporal feature semantic modeling according to claim 3, characterized in that, S3 include: First, the long-term and short-term temporal motion features are used as the Query, and the CLIP features are used as the Key and Value. The features corresponding to the long-term and short-term temporal motion features are retrieved from the CLIP features. Then, the degree of correlation between the long-term and short-term temporal motion features and the CLIP features is measured by calculating the attention score between the Query and the Key. Subsequently, the attention score is normalized by the softmax function to obtain the attention weight between the Query and the Value. The Value is then weighted and aggregated according to the obtained attention weight, thereby implicitly incorporating the latent semantic information related to the long-term and short-term temporal motion features to obtain action features with semantic information.

5. The video description method for spatiotemporal feature semantic modeling according to claim 4, characterized in that, S4 include: S401. Using an object detection model, extract Nt object features in each video frame as nodes of a graph, with object features as node attributes. Use a double-loop loop to traverse the relationships between the Nt objects in the current video frame and the Nt objects in adjacent video frames, calculate the similarity between objects, and use it as edges of the graph to measure the correlation between objects in the video frame sequence, construct an object relationship matrix, and thus obtain an object association graph. At the same time, construct hash table one, using video ID and video frame ID as keys and object features as values. S402. The above action features are represented as nodes. Then, based on the continuity of time, the similarity between action features between adjacent video frames is calculated and used as edges of the graph to measure the correlation of action features in the video frame sequence. An action relationship matrix is ​​constructed to obtain an action association graph. At the same time, a second hash table is constructed, with video ID and video frame ID as keys and action features as values. S403. Based on the video ID and video frame ID, find the corresponding object features and action features in the two hash tables, and then use the object features as nodes and the action features as edges to obtain the object action association graph.

6. The video description method for spatiotemporal feature semantic modeling according to claim 5, characterized in that, S5 include: The object action relationship graph is fed into the GraphTransformer for processing. Each node is mapped to a vector representation containing the feature information of that node. At the same time, each edge in the graph is also mapped to a vector representation to represent the relationship or connection between nodes. These vector representations of nodes and edges are combined to form the object action relationship tensor that the Transformer can encode, which is then passed as input to the subsequent Transformer model. The object action association tensor and CLIP features are fed into the Transformer for encoding and decoding to obtain the final video description based on the spatiotemporal semantic model of the video.

7. A video description system for spatiotemporal feature semantic modeling, characterized in that, include: The feature extraction module includes object features, motion features, and CLIP features. The feature fusion module includes the following fusion process: Long-term temporal modeling of motion features in video frames is performed using a self-attention mechanism to obtain long-term temporal motion features. Short-term temporal modeling of motion features of adjacent video frames is performed using a cross-attention mechanism to obtain short-term temporal motion features. The long-term and short-term temporal motion features are fused to obtain long-short-term temporal motion features; The semantic information acquisition module uses long and short-term temporal motion features as the query for alignment attention and CLIP features as the key and value for alignment attention. It queries the corresponding long and short-term temporal motion features from the CLIP features, indirectly learns the latent semantics of the long and short-term temporal motion features, and obtains action features with semantic information. The object-action association graph construction module uses object features and action features to construct object association graphs and action association graphs respectively, and then merges the object association graphs and action association graphs to construct the object-action association graph. The output module feeds the object action association graph data into the GraphTransformer to obtain the object action association tensor. Then, the object action association tensor and CLIP features are fed into the Transformer network for training to generate the final video description for spatiotemporal feature semantic modeling.

8. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the video description method for video spatiotemporal feature semantic modeling as described in any one of claims 1 to 6.

9. An information data processing terminal for implementing the video description method for spatiotemporal feature semantic modeling of video according to any one of claims 1-6.

10. A computer-readable storage medium comprising instructions, when executed on a computer, causing the computer to perform the video description method for video spatiotemporal feature semantic modeling as described in any one of claims 1-6.

Images