Video description generation method and system based on key semantics and dynamic detail information perception

Abstract

The invention provides a video description generation method and system based on key semantics and dynamic detail information perception, and the method comprises the steps: constructing a video description generation model which comprises a key semantics extraction encoder, a difference detail perception encoder, a global basic encoder and a decoder; performing video frame sampling on a video, and extracting initial image frame features of the sampled video frames by using an image encoder; extracting action key semantic features and object key semantic features in the initial image frame features by using a key semantic extraction encoder; using a difference detail perception encoder to extract difference detail features in the initial image frame features; extracting global basic features in the initial image frame features by using a global basic encoder; and carrying out multi-modal fusion on the key semantic features, the difference detail features and the global basic features, and inputting into a decoder to generate natural language description of the video. According to the method, the accuracy of video description generation can be effectively improved, and more comprehensive detail information is captured.

Description

Technical Field

[0001] This invention relates to the field of video description generation technology, and specifically to a method and system for video description generation based on the perception of key semantics and dynamic detail information. Background Technology

[0002] Video description generation is a key task that integrates computer vision and natural language processing technologies. It aims to provide accurate and coherent textual descriptions of video content and has wide application value in fields such as video retrieval, content recommendation, motion analysis, and auxiliary technologies.

[0003] Video description generation not only requires models to have a deep understanding of the high-level visual semantics in video content, but also to be able to flexibly transform this semantic information into natural language descriptions with rich semantic expressions and grammatical structures. The former involves the identification and modeling of objects, scenes, and contexts in the video, while capturing their dynamic behaviors and events that evolve over time; the latter emphasizes how to effectively bridge the semantic gap between visual and textual information and fully model the syntactic and semantic structures of language. Early video description generation methods were mostly based on template- or rule-driven model frameworks, but due to their inflexibility and fixed paradigms, they were quickly replaced by more flexible deep learning-based encoding and decoding frameworks.

[0004] The encoder-decoder-based model framework dominates the current video captioning field and has gradually evolved into two main research directions. The first direction focuses on deep modeling and feature extraction of video visual content, including introducing more representative multimodal features (such as 3D action features, object detection features, and audio features), and using complex structural modeling to capture spatial-temporal dynamic changes and inter-object interactions in videos. For example, Zhang et al.'s 2019 CVPR paper, "Object-aware aggregation with bidirectional temporal graph for video captioning," uses Mask R-CNN to obtain object-level features and constructs a bidirectional temporal graph model to characterize the behavioral evolution of objects. Zhang et al.'s 2020 CVPR paper, "Object relational graph with teacher-recommended learning for video captioning," introduces graph neural networks (GNNs) to further model intra- and inter-frame object interactions. They enhance visual representation by introducing multimodal features or utilize complex modeling techniques to mine object interactions and spatiotemporal changes to obtain comprehensive video semantic information. While such methods can effectively mine rich visual information, they often overlook the semantic gap between the visual and textual modalities, making it difficult for models to accurately identify the key semantic information in the text description. The second research direction focuses on bridging the semantic differences between the visual and linguistic modalities. This typically involves aligning video features with keywords or phrases extracted from the descriptive text. For example, the 2019 ICCV paper by Hou et al., "Joint syntax representation learning and visual cue translation for video captioning," proposed a translation strategy combining grammatical structure and visual cues. It utilizes part-of-speech (POS) tags to construct sentence templates, guiding the model to generate grammatically correct descriptions. The 2021 AAAI paper by Hobin et al., "Semantic grouping network for video captioning," achieves semantic decoding of keyframes by clustering and aligning video frames with words or phrases in the captions.The paper "Hierarchical Modular Network for Video Captioning" published by Ye et al. at CVPR in 2022 utilizes multi-level supervision signals extracted from descriptive text, including entities, predicates, and sentences, to guide the model in mining more semantically valuable visual information. However, this type of approach has certain limitations: on the one hand, isolated words are difficult to provide complete contextual information, which may lead to the generated results deviating from the true semantics; on the other hand, focusing solely on visual information related to the training text can easily overlook other important details in the video, especially those scenes or behaviors that appear less frequently in the training corpus. Summary of the Invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a method and system for video description generation based on the perception of key semantics and dynamic detail information.

[0006] To achieve the above-mentioned objectives of the present invention, the present invention provides a method for video description generation based on key semantics and dynamic detail information perception, comprising the following steps:

[0007] The video is sampled frame by frame, and the initial image frame features of the sampled video frames are extracted using a pre-trained image encoder;

[0008] A video description generation model is constructed, which includes a key semantic extraction encoder, a difference detail-aware encoder, a global basic encoder, and a decoder.

[0009] The key semantic extraction encoder extracts action key semantic features and object key semantic features from the initial image frame features; the difference detail awareness encoder extracts difference detail features from the initial image frame features; and the global basic encoder extracts global basic features from the initial image frame features.

[0010] Multimodal fusion of key semantic features of actions, key semantic features of objects, differential detail features and global basic features is performed and input into the decoder to generate a natural language description of the video;

[0011] During training, the original frame-level auxiliary description features, frame-level auxiliary description features with missing action semantics, and frame-level auxiliary description features with missing object semantics are extracted simultaneously. Semantic reconstruction is performed based on the aforementioned key action semantic features, key object semantic features, frame-level auxiliary description features with missing action semantics, and frame-level auxiliary description features with missing object semantics. The mean squared error loss during the semantic reconstruction process is calculated to supervise the key semantic extraction encoder. Frame instance-level comparative learning is performed based on the differential detail features and the original frame-level auxiliary description features. The comparative learning loss is calculated to supervise the differential detail-aware encoder.

[0012] Optionally, an action key semantic extraction encoder is used to extract action key semantic features from the features of the initial image frame;

[0013] An object key semantic extraction encoder is used to extract object key semantic features from the features of the initial image frame;

[0014] Supervision of the key semantic extraction encoder includes supervision of the action key semantic extraction encoder and the object key semantic extraction encoder.

[0015] Optionally, the methods for obtaining the original frame-level auxiliary description features, frame-level auxiliary description features lacking action semantics, and frame-level auxiliary description features lacking object semantics are as follows:

[0016] Obtain the original frame-level auxiliary description sentences for the sampled video frames;

[0017] Masking the key attribute words in the original frame-level auxiliary description sentences yields frame-level auxiliary description sentences with missing action semantics and frame-level auxiliary description sentences with missing object semantics.

[0018] A pre-trained encoder is used to obtain frame-level auxiliary description sentence features with missing action semantics, frame-level auxiliary description sentence features with missing object semantics, and original frame-level auxiliary description sentence features, respectively.

[0019] Optionally, during semantic reconstruction, the auxiliary text description sentence features corresponding to the missing key semantics of each sampled video frame can only be used to reconstruct the missing key semantics using the key semantic features of that frame.

[0020] The mean squared error loss is calculated based on the frame-level auxiliary description features with missing semantics output after semantic reconstruction and the original auxiliary text description sentence features to achieve supervised training.

[0021] Optionally, the mean squared error loss includes object semantic mean squared error loss and action semantic mean squared error loss;

[0022] The semantic mean squared error loss function for objects is:

[0023]

[0024] The action semantic mean squared error loss function is:

[0025]

[0026] Where T refers to the range of sampled video frames for a single video;

[0027]

[0028] ||·|| 2 This represents taking the square of the L2 norm of the vector difference. Then it represents the sum of squares of the differences between corresponding elements of two vectors, m∈{obj,act} represents the key semantic type, obj represents object semantics, act represents action semantics, and S m C represents the set of frame-level auxiliary descriptive sentence features that indicate semantic deficiencies. m Represents a set of key semantic features, TransDecoder SRm This indicates that the semantic reconstruction task is performed using the Decoder architecture in the standard Transformer. To reconstruct the features output after completing the missing semantics of the key semantic task, M diag This represents a diagonal attention mask, used to facilitate the generation of frame-level auxiliary descriptive sentence features for key semantic gaps in each sampled video frame. Only the key semantic features of this frame can be utilized To reconstruct the missing key semantics.

[0029] The semantic reconstruction task is as follows: after masking the key semantic attribute words in the complete original frame-level auxiliary description sentence, the remaining part is the context. Based on the features of the remaining part (i.e., the frame-level auxiliary description features with missing object / action semantics), the missing content needs to be filled in using the key semantic features of the object and the key semantic features of the action.

[0030] Optionally, during frame instance-level comparative learning,

[0031] Positive and negative sample pairs are constructed based on the range of sampled video frames from a single video, where the differential detail features of the i-th sampled video frame are... Its corresponding auxiliary description enhancement feature s′ i Constructing a positive sample pair, the differential detail features of the i-th sampled video frame The auxiliary descriptive enhancement feature s′ that does not correspond to any other auxiliary descriptive feature within the range of the sampled video frames. j (j≠i) form a negative sample pair, and the contrastive learning loss function is:

[0032]

[0033] Where T is the range of sampled video frames for a single video. The differential detail features of the i-th sampled video frame are defined by τ, which is the temperature coefficient. sim(a,b) represents the distance between vectors a and b measured by cosine similarity, where τ is the temperature coefficient.

[0034] Optionally, during frame instance-level comparative learning,

[0035] Positive and negative sample pairs are constructed based on the sampled video frame range of the batch video, and the differential detail features of the b-th video in the batch video are analyzed. Enhanced feature s′ of auxiliary description for the b-th video in a batch of videos (b，i) Forming positive sample pairs, differential detail features of the b-th video in a batch of videos. The auxiliary description enhancement feature s′ of the b-th video in the batch of videos (b，i) Additional auxiliary descriptions enhance features s′ (b′，j) To form negative sample pairs, the contrastive learning loss function is:

[0036]

[0037] Where B is the batch size during training, and T is the range of sampled video frames for a single video.

[0038] Optionally, a cross-entropy loss function is constructed to jointly supervise the base encoder, action key semantic extraction encoder, object key semantic extraction encoder, and differential detail feature extraction encoder during training; the cross-entropy loss function is:

[0039]

[0040] in,

[0041] P t (y t |Y ＜t V) = Softmax(h t W cls )

[0042] h t =TransDecoder lang (E ≤t C all )

[0043] E ≤t ={e1, e2, ..., e t}

[0044] e t =Embed(y t-1 )+PE

[0045] L represents the time series of the decoding process, V represents the sampled video frame sequence, PE represents positional encoding, Embed represents the learnable word matrix, and e t The word embedding input at time step t represents the word representation at time step (t-1) after word embedding and position encoding. (TransDecoder) lang This represents a language description generator based on the Transformer standard Decoder architecture, h t C represents the hidden layer output obtained at time step t. all For the final fused features, Softmax transforms the output into a probability distribution, P. t Then Y is the probability output at time step t. <t ={y1,y2,...y t-1} represents the sequence of description content generated at time step t, y t Y represents the word generated at time step t. <t * ={y1 * ,y2 * ,...y t-1 *} represents the actual subtitle segment y at time step t. t * W represents the real word at time step t. cls This represents the weighting parameter.

[0046] Optionally, during training, training is performed with the total loss function as the objective, wherein the total loss function is:

[0047]

[0048] Where, λ obj , λ act , λ xe All are weights. It is the object semantic mean squared error loss function. It is the action semantic mean squared error loss function. It is a contrastive learning loss function. It is the cross-entropy loss function used by the decoder to generate natural language descriptions of the video.

[0049] This application also proposes a video description generation system based on key semantics and dynamic detail information perception, comprising: a data receiving unit, a data processing unit, and an output unit; the data receiving unit is used to input video, and the data receiving unit and the data processing unit are connected and communicate with each other; the data processing unit performs video description on the video input by the data receiving unit according to the above-mentioned video description generation method based on key semantics and dynamic detail information perception, and outputs the result through the output unit; the data processing unit includes:

[0050] The video frame extraction module is used to sample video frames.

[0051] The initial image frame feature extraction module is used to extract initial image frame features from the sampled video frames, and each video obtains a feature sequence;

[0052] The key semantic extraction module is used to encode the extracted feature sequences and capture action key semantic features and object key semantic features at the level of sampled video frames;

[0053] The difference detail perception module is used to encode the extracted feature sequences and capture the dynamic difference detail features between sampled video frames;

[0054] The global basic encoder is used to encode the extracted feature sequences to obtain global basic features;

[0055] The fusion decoding module is used to fuse the captured key semantic features of actions, key semantic features of objects, difference details, and global basic features, and use the fused information to output the probability distribution of words, thereby continuously predicting the words of the next time step until the terminating word appears.

[0056] The beneficial effects of this invention are:

[0057] This invention designs a context-aware key semantic reconstruction task for the key semantic extraction module, used to train the model to extract key object and action semantic information from videos. The application of diagonal masks further enhances the model's attention to and expression of key semantic information in each frame. This invention also designs a frame-instance-level contrastive learning task for the differential detail perception module, used to train the model to focus on differential detail information dynamically changing between video frames. Constructing positive and negative sample pairs based on a batch of sampled video frames allows for the generation of more negative sample pairs, thereby further enhancing the model's discriminative ability. The global basic encoder is mainly used to capture the main content of the video. By effectively fusing the key semantics captured by the key semantic extraction module, the differential detail information captured by the differential detail perception encoder, and the global basic features of the video output by the global basic encoder, the accuracy of video description generation is not only effectively improved, but more comprehensive detail information is also captured.

[0058] Therefore, compared with existing methods, the proposed framework based on object key semantics, action key semantics, and dynamic difference detail feature information can generate semantically accurate and detailed video descriptions. The key semantic extraction encoder captures core semantic information at the video frame level, while the difference detail perception module captures dynamic difference detail information between frames. These two modules are designed in parallel, and the captured information, after being fused with the global basic information of the video, can generate semantically accurate and detailed video descriptions.

[0059] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0060] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0061] Figure 1 This is a schematic diagram of the principle of the present invention;

[0062] Figure 2 This is a schematic diagram of the principle structure of Embodiment 2. Detailed Implementation

[0063] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0064] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0065] Example 1

[0066] like Figure 1 As shown, this invention provides a method for generating video descriptions based on key semantics and dynamic detail information perception, comprising the following steps:

[0067] A video description generation model is constructed, which includes a key semantic extraction encoder, a difference detail-aware encoder, a global basic encoder, and a decoder.

[0068] The video is sampled frame by frame, and the initial image frame features of the sampled video frames are extracted using a pre-trained image encoder.

[0069] This embodiment uses the MSVD dataset. This dataset contains 1970 videos and approximately 80,000 English sentences, with an average of 40 sentences per video. 1200 videos are used as the training set, 100 videos as the validation set, and 670 videos as the test set.

[0070] For each video in the dataset, video frames are sampled, with the number of sampled video frames being T. In this embodiment, T is 20. An image encoder is then used to extract the initial image frame features of the sampled video frames, as shown in the following formula:

[0071] C = CLIP img (V)

[0072] Where V is the sampled video frame sequence, and C is the extracted initial image frame feature sequence. The image encoder network for extracting the initial image frame features can be AlexNet, GooLeNet, ResNet, or CLIP image encoder, etc. This embodiment uses the CLIP image encoder.

[0073] The extracted initial image frame features are input into the video description generation model, and the key semantic extraction encoder is used to extract action key semantic features and object key semantic features from the initial image frame features. Specifically, the action key semantic extraction encoder extracts action key semantic features from the initial image frame features; the object key semantic extraction encoder extracts object key semantic features from the initial image frame features. Both share the same encoder-decoder architecture but use independent parameters; this invention uses m∈{obj,act} to represent these two branches. The key semantic features are obtained as follows:

[0074]

[0075] Among them, TransEncoder KSEm The encoder representing object / action key semantic extraction uses the standard Transformer encoder architecture; m∈{obj,act} represents the key semantic type, where "obj" represents object semantics and "act" represents action semantics, c i m This represents the key semantic features of the object / action in the i-th frame.

[0076] A difference detail-aware encoder is used to extract difference detail features from the initial image frame features. This captures the differential detail information of inter-frame changes; these difference detail features are dynamic features. In this embodiment, the difference detail features are obtained as follows:

[0077]

[0078] Among them, TransEncoder DDP This represents the differential detail feature extraction encoder, c i d This represents the difference details of the i-th sampled video frame.

[0079] Extract using a global basic encoder Initial image frame features The global basic features are as follows:

[0080]

[0081] Where PE represents position code, c i g represents the global basic feature of the i-th frame.

[0082] The video's natural language description is generated by multimodal fusion of key semantic features of actions, key semantic features of objects, differential detail features, and global basic features, and then input into the decoder. The multimodal fusion process of key semantic features of actions, key semantic features of objects, differential detail features, and global basic features is as follows:

[0083]

[0084] in, C represents the fusion strategy for mean pooling. all These are the final fused features. The fused features are then fed into the decoder to generate a natural language description, as follows:

[0085] e t =Embed(y t-1 )+PE

[0086] E ≤t ={e1,e2,…,e t}

[0087] h t =TransDecoder lang (E ≤t C all )

[0088] P t (y t |Y ＜t V) = Softmax(h t W cls )

[0089] Where L is the time series of the decoding process (more precisely, L represents the sentence length of the actual description (i.e., the number of words in the actual description sentence); since one word is generated at each time step during decoding, L is also the time series of the decoding process), V represents the sampled video frame sequence, PE represents positional encoding, Embed represents the learnable word matrix, and e t The word embedding input at time step t represents the word representation at time step (t-1) after word embedding and position encoding. (TransDecoder) lang This represents a language description generator based on the standard Transformer decoder architecture. This generator can also use other recurrent neural networks such as GRU, LSTM, and Transformer to generate language descriptions. t P represents the hidden layer output obtained at time step t. Softmax transforms the output into a probability distribution. t Then Y is the probability output at time step t. <t ={y1,y2,...y t-1} represents the sequence of description content generated at time step t, y t Y represents the word generated at time step t. <t * ={y1 * ,y2 * ,...y t-1 *} represents the actual subtitle segment at time step t, y t * W represents the real word at time step t. cls This represents the weighting parameter.

[0090] During the training phase, original frame-level auxiliary description features, frame-level auxiliary description features lacking action semantics, and frame-level auxiliary description features lacking object semantics are also extracted simultaneously. The methods for obtaining the original frame-level auxiliary description features, frame-level auxiliary description features lacking action semantics, and frame-level auxiliary description features lacking object semantics are as follows:

[0091] (1) Obtain the original frame-level auxiliary description sentences of the sampled video frames. In this embodiment, the corresponding original frame-level auxiliary description sentences are obtained through the image-text pre-trained model BLIP, as follows:

[0092]

[0093] Where X is the original frame-level auxiliary description sentence set, x i w represents the original frame-level auxiliary description sentence obtained for the i-th sampled video frame. j L represents the j-th word in the sentence. iSentence x i The number of words in the sampled video frame sequence is V.

[0094] (2) Masking the key attribute words in the original frame-level auxiliary description sentences yields frame-level auxiliary description sentences lacking action semantics and frame-level auxiliary description sentences lacking object semantics. In this embodiment, the part-of-speech tag set for object attributes is defined as A. obj Define the set of part-of-speech tags for action attributes as A. act We utilize mature natural language processing tools to identify the part-of-speech tags of all words in the original set of frame-level auxiliary description sentences X. Through masking operations, we obtain frame-level auxiliary description sentences lacking action semantics and frame-level auxiliary description sentences lacking object semantics, respectively. The detailed operations are as follows:

[0095] A obj ={NN,NNS,NNP,NNPS}

[0096] A act ={VB,VBD,VBG,VBN,VBP,VBZ}

[0097]

[0098] P(w j ) represents any word w j The part of speech of x i m X represents the frame-level auxiliary description sentence corresponding to the i-th sampled video frame that lacks semantic meaning for the object or action. m Frame-level auxiliary description sentences representing missing semantics of objects or actions. surface Show the words after the masking operation, A m A represents obj and A act The set of words in the original frame-level auxiliary description sentence is masked if the word attribute belongs to the object attribute part-of-speech set or the action attribute part-of-speech set; otherwise, no processing is performed.

[0099] (3) Use the pre-trained CLIP model's text encoder to obtain the frame-level auxiliary description sentence features with missing action semantics, the frame-level auxiliary description sentence features with missing object semantics, and the original frame-level auxiliary description sentence features.

[0100]

[0101] Among them, s i m s represents the semantically missing frame-level auxiliary description sentence feature corresponding to the i-th sampled video frame. iThis represents the original frame-level auxiliary description sentence feature corresponding to the i-th sampled video frame.

[0102] Perform semantic reconstruction tasks: Perform semantic reconstruction based on key semantic features of actions, key semantic features of objects, frame-level auxiliary description features with missing action semantics, and frame-level auxiliary description features with missing object semantics. Calculate the mean squared error loss in the semantic reconstruction process and supervise the key semantic extraction encoder.

[0103] Specifically,

[0104]

[0105] Among them, TransDecoder SRm This indicates that the semantic reconstruction task is performed using the Decoder architecture in the standard Transformer. diag This represents a diagonal attention mask, the purpose of which is to facilitate the generation of frame-level auxiliary descriptive sentence features s corresponding to the key semantic missing information in each sampled video frame. i m Only the key semantic features c of this frame can be utilized i m To reconstruct the missing key semantics, thereby emphasizing C m Effective representation of the key semantic features of all sampled video frames. To reconstruct the features output after completing the missing semantics of the key semantic task, the mean squared error (MSE) loss is calculated between this feature and the original frame-level auxiliary description sentence features S to achieve supervised training. 2 This represents taking the square of the L2 norm of the vector difference. Then it represents the sum of squares of the differences between corresponding elements of two vectors, m∈{obj,act}. Let the semantic mean squared error loss function be the object's semantics. The mean squared error loss function for action semantics is used as the training objective to supervise the object key semantic extraction encoder and the action key semantic extraction encoder.

[0106] Perform frame instance-level contrastive learning: Perform frame instance-level contrastive learning based on difference detail features and original frame-level auxiliary description features, calculate the contrastive learning loss, and supervise the difference detail-aware encoder. Specifically, extract frame-level auxiliary description enhancement features from the original frame-level auxiliary description sentence features S, as follows:

[0107]

[0108] Among them, TransEncoder AT To assist the text encoder, s i ′ represents the auxiliary description enhancement feature corresponding to the i-th sampled video frame, and ci d They exist in the same feature space.

[0109] This embodiment provides two ranges for constructing positive and negative sample pairs.

[0110] The first method constructs positive and negative sample pairs based on the range of sampled video frames of a single video, i.e., the differential detail features of the i-th sampled video frame. Its corresponding auxiliary description enhancement feature s′ i This constitutes a positive sample pair, which is an auxiliary descriptive enhancement feature s′ that does not correspond to any other sampled video frame within the range of the video sample. j (j≠i) form a negative sample pair, and the contrastive learning loss is as follows:

[0111]

[0112] Where T is the range of sampled video frames for a single video, d refers to the temperature coefficient, τ is the temperature coefficient, sim(a,b) represents the distance between two vectors a and b measured by cosine similarity, and τ is the temperature coefficient.

[0113] The second method constructs positive and negative sample pairs based on the range of sampled video frames in a batch of videos, using B to represent the batch size during training. and These represent the differential detail features and auxiliary description enhancement features of the b-th video in a batch of videos of different sizes. With the corresponding s′ (b，i) Form positive sample pairs, with s′ (b′，j) (b′≠b or j≠i) form negative sample pairs, and the contrastive learning loss function is:

[0114]

[0115] During the training phase, a cross-entropy loss function L is also constructed. XE This serves as the training objective, enabling joint supervision of the global basic encoder, action key semantic extraction encoder, object key semantic extraction encoder, and differential detail-aware encoder. The cross-entropy loss function L... XE as follows:

[0116]

[0117] Therefore, the total loss function during the training process in this embodiment is:

[0118]

[0119] Where, λ obj , λ act , λ xeThese are the weights, which in this embodiment are 1.0, 1.0, and 1.5 respectively. If the contrastive learning loss function adopts the first case mentioned above, it can also be... Replace with

[0120] Throughout the entire training process, the total loss function described above is used as the overall objective to achieve complete training.

[0121] After the video description generation model is trained, the video for which a video description needs to be generated is input into the video description generation model. The model extracts key semantic features of the object, key semantic features of the action, differential detail features, and global basic features, and performs multimodal fusion. Finally, the model is input into the decoder to generate a natural language description of the video, thus completing the generation of the video description.

[0122] To demonstrate the superiority of this embodiment, the experimental results of this embodiment are compared with those of existing methods.

[0123] Existing Method 1: The method proposed by Zhong et al. in their 2023 paper "Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning" published at the AAAI conference.

[0124] Existing Method 2: The method proposed by Luo et al. in their 2024 paper "Globalsemantic enhancement network for video captioning" published in the journal Pattern Recognition.

[0125] The third existing method is the method proposed by Li et al. in their 2024 paper "Learning Hierarchical Modular Networks for Video Captioning" published in the IEEE TPAMI journal.

[0126] The experiment used the BLEU@4 metric to evaluate the accuracy of video description generation. BLEU@4 refers to the matching ratio of four-grams between the generated sentence and the manually annotated sentence. The higher the BLEU@4 value, the more accurate the video description generation result is, capturing more accurate semantics. Simultaneously, this experiment used CIDEr to evaluate the richness of video description generation. CIDEr is closer to the principle by which humans judge whether two sentences are similar because it utilizes TF-IDF (Term Frequency Inverse Document Frequency) to assign different weights to different n-grams. That is, frequently occurring phrases have lower weights, while infrequently occurring phrases are more special (have higher weights). People pay more attention to these infrequent but important words. This approach allows CIDEr to focus on more comprehensive detailed semantic information while capturing accurate core semantics. The higher the CIDEr value, the more accurate and richer the video description generation result is.

[0127] As shown in Table 1, this embodiment achieves better video description generation results. Existing methods lack accurate grasp of core semantics and neglect a large amount of detailed information, resulting in video description generation results that are inferior to those of this invention. The key semantic and dynamic detail information perception framework of this embodiment can generate semantically accurate and detail-rich video descriptions. The semantic reconstruction task performed by the key semantic extraction module guides the model to focus on the core semantic content of each frame, capturing more accurate and richer key semantic information; the frame instance-level contrastive learning task performed by the difference detail perception module guides the model to focus on inter-frame differences, capturing easily lost detail information, further enhancing the completeness of the description, and ultimately generating semantically accurate and detail-rich video descriptions.

[0128] Table 1. Comparative experimental results with existing methods

[0129] method accuracy Richness Existing Method 1 0.512 0.967 Existing Method 2 0.588 1.025 Existing Method 3 0.599 1.047 This embodiment 0.619 1.127

[0130] Example 2

[0131] This embodiment provides a video description generation system based on key semantics and dynamic detail information perception, including: a data receiving unit, a data processing unit, and an output unit; the data receiving unit is used to input video, and the data receiving unit and the data processing unit are connected and communicate with each other; the data processing unit performs video description on the video input by the data receiving unit according to the video description generation method based on key semantics and dynamic detail information perception described in Embodiment 1, and outputs the result through the output unit.

[0132] like Figure 2As shown, the data processing unit includes: a video frame extraction module, an initial image frame feature extraction module, a key semantic extraction module, a difference detail perception module, a global basic encoder, and a fusion decoding module. Among them,

[0133] The video frame extraction module is used to sample video frames; the initial image frame feature extraction module is used to extract initial image frame features from the sampled video frames, obtaining a feature sequence for each video; the key semantic extraction module is used to encode the extracted feature sequences, capturing action key semantic features and object key semantic features at the sampled video frame level; the difference detail perception module is used to encode the extracted feature sequences, capturing dynamic difference detail features between sampled video frames; the global basic encoder is used to encode the extracted feature sequences to obtain global basic features; the fusion decoding module is used to fuse the captured action key semantic features, object key semantic features, difference detail features, and global basic features, and uses the fused information to output the probability distribution of words, thereby continuously predicting the words of the next time step until the terminating word appears.

[0134] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0135] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for video description generation based on key semantic and dynamic detail information perception, characterized in that, The method comprises the following steps: video frame sampling is performed on a video, and an initial image frame feature of the sampled video frame is extracted by using a pre-trained image encoder; a video description generation model is constructed, the video description generation model comprising a key semantic extraction encoder, a difference detail perception encoder, a global basic encoder and a decoder; action key semantic features and object key semantic features in the initial image frame features are extracted by using the key semantic extraction encoder, difference detail features in the initial image frame features are extracted by using the difference detail perception encoder, and global basic features in the initial image frame features are extracted by using the global basic encoder; the action key semantic features, the object key semantic features, the difference detail features and the global basic features are subjected to multi-modal fusion, and a decoder is inputted to generate a natural language description of the video; during the training process, original frame-level auxiliary description features, frame-level auxiliary description features with missing action semantics and frame-level auxiliary description features with missing object semantics are also extracted synchronously; semantic reconstruction is performed based on the action key semantic features, the object key semantic features, the frame-level auxiliary description features with missing action semantics and the frame-level auxiliary description features with missing object semantics, mean square error loss in the semantic reconstruction process is calculated, the key semantic extraction encoder is supervised, frame instance-level contrast learning is performed based on the difference detail features and the original frame-level auxiliary description features, contrast learning loss is calculated, and the difference detail perception encoder is supervised.

2. The method for generating a video description based on key semantic and dynamic detail information perception according to claim 1, characterized in that: an action key semantic extraction encoder is used to extract action key semantic features in the initial image frame features; an object key semantic extraction encoder is used to extract object key semantic features in the initial image frame features; the supervision of the key semantic extraction encoder is the supervision of the action key semantic extraction encoder and the object key semantic extraction encoder.

3. The method for video description generation based on key semantics and dynamic details perception according to claim 1, characterized in that, The manner of obtaining the original frame-level auxiliary description features, the frame-level auxiliary description features with missing action semantics and the frame-level auxiliary description features with missing object semantics is as follows: original frame-level auxiliary description sentences of the sampled video frames are obtained; key attribute words in the original frame-level auxiliary description sentences are subjected to a mask operation to obtain frame-level auxiliary description sentences with missing action semantics and frame-level auxiliary description sentences with missing object semantics; a pre-trained encoder is used to obtain frame-level auxiliary description sentence features with missing action semantics, frame-level auxiliary description sentence features with missing object semantics and original frame-level auxiliary description sentence features.

4. The method for video description generation based on key semantics and dynamic details perception according to claim 1, characterized in that, During semantic reconstruction, the key semantic missing auxiliary text description sentence features corresponding to each sampled video frame can only use the key semantic features of the frame to reconstruct the missing key semantics; mean square error loss is calculated based on the frame-level auxiliary description features with missing semantics output after semantic reconstruction and the original auxiliary text description sentence features to realize supervised training.

5. The method for video description generation based on key semantics and dynamic details information perception according to claim 4, characterized in that, The mean square error loss comprises object semantic mean square error loss and action semantic mean square error loss; the object semantic mean square error loss function is as follows: the action semantic mean square error loss function is as follows: Wherein, T refers to the range of sampled video frames of a single video; ||·| 2 represents the square of L2 norm of vector difference, represents the sum of square of difference of corresponding elements of two vectors, m e {obj, act} represents the key semantic type, obj represents the object semantic, and act represents the action semantic, S m represents the frame-level auxiliary description sentence feature set with missing semantics, C m represents the key semantic feature set, TransDecoder SRm represents the semantic reconstruction task performed using the Decoder architecture in the standard Transformer, represents the feature after the missing semantics is completed for the output of the key semantic reconstruction task, M diag represents the diagonal attention mask, which is used to prompt the frame-level auxiliary description sentence feature corresponding to each sampling video frame to reconstruct the missing key semantic only using the key semantic feature of the frame. reconstruct the missing key semantic only using the key semantic feature of the frame.

6. The method for video description generation based on key semantics and dynamic details perception according to claim 1, characterized in that, When the frame instance-level contrast learning, Constructing positive and negative sample pairs based on a range of sampled video frames of a single video, where the differentiating detail features of the ithsampled video frame its corresponding auxiliary description enhancement feature s' i Constructing positive sample pairs, the differentiating detail features of the ithsampled video frame all other non-corresponding auxiliary description enhancement features s' within the range of the video sampled video frames j (j≠i) form negative sample pairs, and the contrastive learning loss function is: Wherein, T is the range of the sampling video frames of a single video, denotes the differentiated detail feature of the i-th sampling video frame, τ is a temperature coefficient, and sim(a, b) represents the distance between two vectors a and b by cosine similarity measure, and τ is a temperature coefficient.

7. The method for video description generation based on key semantics and dynamic details perception according to claim 1, characterized in that, When the frame instance-level contrast learning, Constructing positive and negative sample pairs based on a range of sampled video frames of a batch video, the differentiated detailed features of the bth video in the batch video The auxiliary description enhanced features s' of the bth video in the batch video (b，i) Forming a positive sample pair, the differentiated detailed features of the bth video in the batch video The auxiliary description enhanced features s' of the bth video in the batch video (b，i) except the auxiliary description enhanced features s' (b′，j) Forming a negative sample pair, the contrastive learning loss function is Wherein, B is the batch size in the training process, T is the range of sampled video frames of a single video, b'≠b, j≠i.

8. The method for video description generation based on key semantics and dynamic details information perception according to claim 1, characterized in that, The cross-entropy loss function is constructed to jointly supervise the basic encoder, the key semantic extraction encoder and the differential detail feature extraction encoder during the training process; the cross-entropy loss function is: Wherein, P t (y t | Y＜ t, V) = Softmax(h t W cls ) h t = TransDecoder lang (E ≤t , C all ) E ≤t = {e1, e2,..., en} where n is the number of elements in the set. t} e t = Embed(y t-1 )+PE L is the time sequence of decoding process, V represents the sampled sequence of video frames, PE represents the position encoding, Embed represents the learnable word matrix, e t represents the word embedding input at the t-th time step, which is the representation of the word at the t-1-th time step after word embedding and position encoding, TransDecoder lang represents the language description generator based on the standard Decoder architecture of Transformer, h t represents the hidden layer output obtained at the t-th time step, C all is the final fused feature, Softmax converts the output into a probability distribution, P t is the probability output at the t-th time step, Y <t = {y1, y2,... y t-1} represents the generated description content sequence at the t-th time step, y t represents the word generated at the t-th time step, Y <t * = {y1 * , y2 * ,... y t-1 *} represents the real subtitle segment at the t-th time step, y t * represents the real word at the t-th time step, W cls represents the weight parameter.

9. The method for video description generation based on key semantics and dynamic details perception according to claim 1, characterized in that, During the training process, the total loss function is used as the target for training, and the total loss function is: wherein λ obj , λ act , λ xe are weights, is an object semantic mean squared error loss function, is an action semantic mean squared error loss function, is a contrastive learning loss function, is a cross-entropy loss function when the decoder generates a natural language description of the video. 10.A system for generating video description based on key semantic and dynamic details perception, characterized in that, Including: The data receiving unit, the data processing unit and the output unit; the data receiving unit is used for inputting video, and the data receiving unit and the data processing unit are connected and communicate with each other; the data processing unit carries out video description on the video input by the data receiving unit according to the method for generating video description based on key semantic and dynamic detail information perception according to any one of claims 1-9, and outputs the result through the output unit; the data processing unit includes: The video frame extraction module is used for video frame sampling of the video; The initial image frame feature extraction module is used for initial image frame feature extraction of the sampled video frames, and each video obtains a feature sequence; The key semantic extraction module is used for encoding the extracted feature sequence to capture the action key semantic features and object key semantic features of the sampled video frame level; The differential detail perception module is used for encoding the extracted feature sequence to capture the dynamic differential detail features between the sampled video frames; The global basic encoder is used for encoding the extracted feature sequence to obtain the global basic features; The fusion decoding module is used for fusing the captured action key semantic features, object key semantic features, differential detail features and global basic features, and using the fused information to output the probability distribution of the vocabulary, so as to continuously predict the vocabulary of the next time step until the terminal vocabulary appears.

Images