Video description method and system based on video point trajectory constraints

Abstract

This paper proposes a video description method and system based on video point trajectory constraints. The method includes: sampling keyframe images and using a point tracking algorithm to obtain the spatiotemporal trajectories of pixel points between consecutive frames; performing an average pooling operation on the visual features of frames corresponding to the same trajectory segment; semantically aligning text features, visual features, and trajectory features before performing multi-head attention feature fusion; calculating semantic relevance scores for the visual areas corresponding to the trajectory segments and sorting them in descending order by relevance score, accumulating the relevance scores and setting a threshold; jointly optimizing the video point tracking model using a language generation loss and a focus loss; and decoding the multi-source features after focus optimization to obtain the final video description result. By introducing a video point trajectory aggregation strategy, the present invention explicitly models the dynamic characteristics of the target in the spatiotemporal dimension, preserving the spatial appearance and temporal coherence of the object, and effectively solving the problems of semantic discontinuity and description fragmentation in complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and video processing, and in particular to a video description method and system based on video point trajectory constraints. Background Art

[0002] Video content description technology aims to automatically generate semantically sound and linguistically coherent natural language sentences for input videos. It is a core task at the intersection of computer vision and natural language processing, and is widely used in scenarios such as video retrieval, intelligent surveillance, and assistance for the visually impaired. Existing methods typically use frame-level visual feature extraction and sequence modeling techniques, combined with attention mechanisms for multimodal fusion and text generation. However, in complex dynamic scenes such as occlusion, multi-target interaction, or rapid motion, existing models have difficulty continuously tracking target semantics, resulting in semantic incoherence, fragmentation, or redundancy in the description. In addition, the cross-modal attention mechanism lacks precise guidance, often causing the model to focus on irrelevant areas and insufficient semantic alignment accuracy. Summary of the Invention

[0003] In view of the above situation, the main purpose of the present invention is to propose a video description method and system based on video point trajectory constraints to solve the above technical problems.

[0004] The present invention proposes a video description method based on video point trajectory constraints, the method comprising the following steps:

[0005] Step 1: Sample the key frame images in the video and use the point tracking algorithm to obtain the spatiotemporal trajectory of the pixel points between consecutive frames to construct a set of trajectory segments of the video target;

[0006] Step 2: Based on the trajectory segment set of the video target, the visual features of the frame corresponding to the trajectory segment are extracted from the trajectory segment, and the visual features of the frame corresponding to the same trajectory segment are average pooled to obtain the trajectory features;

[0007] Step 3: Encode the text description to obtain text features;

[0008] Step 4: Semantically align text features, visual features, and trajectory features before performing multi-head attention feature fusion to obtain the attention allocation weight vector and the corresponding output features after multi-head attention.

[0009] Step 5: Use the attention allocation weight vector to calculate the semantic relevance score of the visual area corresponding to the trajectory segment to obtain the comprehensive trajectory relevance score, and sort them in descending order of relevance score. Accumulate the relevance score and set the threshold to obtain the key trajectory;

[0010] Step 6: Construct a language generation loss based on the corresponding output features after multi-head attention, construct a focus loss based on the comprehensive trajectory correlation score and key trajectories, and use the language generation loss and focus loss to jointly optimize the video point tracking model to obtain the optimized video point tracking model;

[0011] Utilize the optimized video point tracking model to obtain the multi-source features after focus optimization;

[0012] Step 7: Decode the multi-source features after focus optimization to obtain the final video description result.

[0013] The present invention also proposes a video description system based on video point trajectory constraints, the system comprising:

[0014] Feature encoding module, used to:

[0015] The key frame images in the video are sampled, and the point tracking algorithm is used to obtain the spatiotemporal trajectory of the pixel points between consecutive frames to construct a set of trajectory segments of the video target;

[0016] Based on the trajectory segment set of the video target, the visual features of the frame corresponding to the trajectory segment are extracted from the trajectory segment, and the visual features of the frame corresponding to the same trajectory segment are average pooled to obtain the trajectory features;

[0017] Encode the text description to obtain text features;

[0018] Multimodal fusion module, used for:

[0019] For text features, visual features, and trajectory features, semantic alignment is first performed and then multi-head attention feature fusion is performed to obtain the attention allocation weight vector and the corresponding output features after multi-head attention.

[0020] The attention allocation weight vector is used to calculate the semantic relevance score of the visual area corresponding to the trajectory segment to obtain the comprehensive trajectory relevance score. The scores are sorted in descending order by relevance score, and the relevance scores are accumulated and a threshold is set to obtain the key trajectory.

[0021] The language generation loss is constructed based on the corresponding output features after multi-head attention, and the focus loss is constructed based on the comprehensive trajectory correlation score and key trajectory. The language generation loss and focus loss are used to jointly optimize the video point tracking model to obtain the optimized video point tracking model.

[0022] Utilize the optimized video point tracking model to obtain the multi-source features after focus optimization;

[0023] Video description generation module, used to:

[0024] The multi-source features after focus optimization are decoded to obtain the final video description result.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. By introducing a video point trajectory aggregation strategy, this paper explicitly models the dynamic characteristics of the target in the spatiotemporal dimension, preserves the spatial appearance and temporal coherence of the object, and effectively solves the problems of semantic discontinuity and description fragmentation in complex scenes.

[0027] 2. This paper proposes an unsupervised key trajectory focus learning method, which uses the attention weight distribution between trajectories and text to screen key trajectories, guides the model to focus on semantically relevant areas, suppresses background interference, and significantly improves the semantic matching between video and text modalities.

[0028] 3. The key trajectory mining and focusing mechanisms proposed in this invention are based on the internal features of the model and the attention structure, without relying on manually labeled attention tags. They have stronger adaptability and practicality and are suitable for deployment in actual application systems.

[0029] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flowchart of the steps of the video description method based on video point trajectory constraints proposed by the present invention.

[0031] Figure 2 This is the overall architecture diagram of the video description method based on video point trajectory constraints proposed in the present invention.

[0032] Figure 3 This is a structural diagram of the video description system based on video point trajectory constraints proposed by the present invention. DETAILED DESCRIPTION

[0033] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0034] These and other aspects of the embodiments of the present invention will become clear with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to provide some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0035] See also Figure 1 The embodiment of the present invention proposes a video description method based on video point trajectory constraints, which includes the following steps:

[0036] Step 1: Sample the key frame images in the video and use the point tracking algorithm to obtain the spatiotemporal trajectory of pixel points between consecutive frames to construct a set of trajectory segments of the video target.

[0037] See also Figure 2 In step 1, the key frame images in the video are sampled, and the point tracking algorithm is used to obtain the spatiotemporal trajectory of the pixel points between consecutive frames to construct a set of trajectory segments of the video target. The corresponding process has the following relationship:

[0038] ;

[0039] in, represents the first space-time trajectory segment in the space-time trajectory segment set, Represents the first A fragment of space-time trajectory, represents the video point tracking model, Represents a collection of frames uniformly sampled from a video.

[0040] It should be noted that in Figure 2 In this paper, a video point trajectory estimator and a video encoder are used to sample the video key frame image to obtain the position change and motion trajectory of each pixel in the image in the video frame.

[0041] Step 2: Based on the trajectory segment set of the video target, the visual features of the frame corresponding to the trajectory segment are extracted from the trajectory segment, and the visual features of the frame corresponding to the same trajectory segment are average pooled to obtain the trajectory features.

[0042] In step 2, based on the trajectory segment set of the video target, the visual features of the frame corresponding to the trajectory segment are extracted from the trajectory segment, and the visual features of the frame corresponding to the same trajectory segment are average pooled to obtain the trajectory features. The specific steps include:

[0043] Based on the trajectory segment set of the video target, spatial perception calculation is performed on the frames uniformly sampled from the video to obtain the grid feature vector of each frame;

[0044] The grid feature vectors of each frame are spliced ​​together to obtain the visual feature vector;

[0045] The visual feature vectors on the same trajectory segment are average pooled to obtain the trajectory features.

[0046] Perform spatial perception calculations on frames uniformly sampled from the video to obtain the grid feature vector of each frame. The corresponding relationship is as follows:

[0047] ;

[0048] in, Represents each frame Grid feature vector, represents the grid feature vector of the first frame, Indicates the Frame grid feature vector;

[0049] It should be noted that in Figure 2 In , a video encoder is used to encode frames uniformly sampled from the video.

[0050] In the step of splicing the grid feature vectors of each frame to obtain the visual feature vector, the corresponding process has the following relationship:

[0051] ;

[0052] in, represents the visual feature vector, represents the first grid feature vector, represents the total number of visual grid vectors extracted from the entire video, Represents a video frame, represents the total number of grid feature vectors;

[0053] In the step of performing average pooling on the visual feature vectors on the same trajectory segment to obtain the trajectory features, the corresponding process has the following relationship:

[0054] ;

[0055] in, Represents the trajectory features corresponding to all spatiotemporal trajectory segments of the video, represents the first visual feature vector, represents the total number of visual feature vectors, Indicates the visual feature vectors, Represents the current spatiotemporal trajectory segment The trajectory length, Represents a spatiotemporal trajectory segment in a trajectory set On the A visual feature.

[0056] Step 3: Encode the text description to obtain text features.

[0057] In step 3, a text encoder is used to encode the text description.

[0058] Step 4: Semantically align the text features, visual features, and trajectory features before performing multi-head attention feature fusion to obtain the attention allocation weight vector and the corresponding output features after multi-head attention.

[0059] In step 4, the text features, visual features, and trajectory features are first semantically aligned and then multi-head attention feature fusion is performed to obtain the attention allocation weight vector and the corresponding output features after multi-head attention. The specific steps include the following:

[0060] The text features, visual features and trajectory features are spliced ​​together to obtain a comprehensive representation of the multimodal input;

[0061] Multi-head self-attention feature fusion is performed on the comprehensive representation of multimodal input to obtain the attention allocation weight vector and the corresponding output features after multi-head attention.

[0062] Multi-head self-attention feature fusion is performed on the comprehensive representation of multimodal input to obtain the attention allocation weight vector and the corresponding output feature after multi-head attention. The relationship between the corresponding process is as follows:

[0063] ;

[0064] in, represents the comprehensive representation of multimodal inputs in the fusion process, Indicates time, represents the attention weight vector The first attention weight vector in , represents the attention weight vector The The attention weight value, represents the normalization function, represents the initial encoding vector of the predicted word, represents the first embedding matrix, represents the comprehensive representation matrix of multimodal input, represents the second embedding matrix, represents transpose, Represents the dimension of the feature, Represents the corresponding output features after multi-head attention, represents the attention weight vector The The attention weight value, Indicates the A comprehensive representation vector of multimodal inputs.

[0065] Step 5: Use the attention allocation weight vector to calculate the semantic relevance score of the visual area corresponding to the trajectory segment to obtain the comprehensive trajectory relevance score, and arrange it in descending order of relevance score. Accumulate the relevance score and set the threshold to obtain the key trajectory.

[0066] In step 5, the attention allocation weight vector is used to calculate the semantic relevance score of the visual area corresponding to the trajectory segment to obtain the comprehensive trajectory relevance score. The scores are sorted in descending order by relevance score, and the relevance scores are accumulated and a threshold is set to obtain the key trajectory. The specific steps include the following:

[0067] Merge visual features and trajectory features to obtain multi-source video features;

[0068] The attention allocation weight vector is used to calculate the semantic relevance score of multi-source video features and text features, and the relevance score vector of multi-source video features and text features is obtained;

[0069] Normalizing the correlation score vectors of the multi-source video features and the text features to obtain the normalized correlation score vectors of the multi-source video features and the text features;

[0070] Perform comprehensive trajectory correlation calculation on the normalized correlation score vectors of multi-source video features and text features to obtain a comprehensive trajectory correlation score;

[0071] The comprehensive trajectory correlation scores are normalized and sorted in descending order to obtain the correlation scores corresponding to the sorted trajectories;

[0072] The correlation scores corresponding to the sorted trajectories are defined as a set to obtain the trajectory accumulation set;

[0073] Based on the trajectory accumulation set, the correlation cumulative score corresponding to the sorted trajectories is calculated to obtain the trajectory segment correlation cumulative score vector;

[0074] The cumulative score threshold is set, and the trajectory segment correlation cumulative score vector is screened to obtain the key trajectory.

[0075] The attention allocation weight vector is used to calculate the semantic relevance score of multi-source video features and text features, and the relevance score vector of multi-source video features and text features is obtained. The relationship between the corresponding process is as follows:

[0076] ;

[0077] in, Represents the correlation score vector of multi-source video features and text features The correlation scores, represents the total number of relevance scores;

[0078] In the step of normalizing the correlation score vectors of the multi-source video features and the text features to obtain the normalized correlation score vectors of the multi-source video features and the text features, the corresponding process has the following relationship:

[0079] ;

[0080] in, Represents the normalized correlation score vector of multi-source video features and text features, represents the normalized correlation score of the first visual feature, Indicates the The normalized correlation score of the visual features, represents the normalized correlation score of the first trajectory feature, Indicates the The correlation score of the normalized trajectory features;

[0081] In the step of calculating the comprehensive trajectory correlation of the correlation score vector after normalization of multi-source video features and text features to obtain the comprehensive trajectory correlation score, the corresponding process has the following relationship:

[0082] ;

[0083] in, Indicates the The comprehensive trajectory correlation score of the trajectories, represents the comprehensive trajectory correlation score function, Indicates the tracks;

[0084] In the step of defining the set of correlation scores corresponding to the sorted trajectories and obtaining the trajectory accumulation set, the corresponding process has the following relationship:

[0085] ;

[0086] in, represents the trajectory accumulation set, Indicates the first Tracks, represents the set of positive integers;

[0087] In the step of calculating the correlation cumulative score of the correlation scores corresponding to the sorted trajectories to obtain the trajectory segment correlation cumulative score vector, the corresponding process has the following relationship:

[0088] ;

[0089] in, represents the cumulative score vector of trajectory segment correlation, represents the cumulative score of the first trajectory segment relevance, Indicates the The cumulative correlation score of trajectory segments, Indicates the Trajectory accumulation set Middle Track segments, Represents a trajectory segment the corresponding integrated trajectory correlation;

[0090] In the steps of setting the cumulative score threshold and screening the trajectory segment correlation cumulative score vector to obtain the key trajectory, the corresponding process has the following relationship:

[0091] ;

[0092] in, represents the critical trajectory, Represents a set function that obtains the maximum value of the parameter. Indicates the cumulative score threshold.

[0093] It should be noted that setting a cumulative score threshold can filter out key tracks with greater semantic contribution.

[0094] Step 6: Construct a language generation loss based on the corresponding output features after multi-head attention, construct a focus loss based on the comprehensive trajectory correlation score and key trajectories, and use the language generation loss and focus loss to jointly optimize the video point tracking model to obtain the optimized video point tracking model;

[0095] The optimized video point tracking model is used to obtain focus-optimized multi-source features.

[0096] In step 6, the language generation loss is constructed based on the corresponding output features after multi-head attention. The relationship between the corresponding process is as follows:

[0097] ;

[0098] in, Represents the generated word, represents the probability distribution of the output words in the vocabulary, represents the fully connected layer, represents the language generation loss, Indicates the number of generated words, Indicates the generated words, Indicates the first 1 to word;

[0099] It should be noted that It is from The word with the highest predicted probability in the vocabulary is obtained by sampling.

[0100] In the step of constructing the focus loss based on the comprehensive trajectory correlation score and the key trajectory, the relationship between the corresponding process is as follows:

[0101] ;

[0102] in, represents the focus loss, represents the logarithmic function, Indicates the sequence number, represents the sum of attention weights of key track areas, Indicates that the cumulative score threshold is met Key Trajectory , represents the coefficient as the weight of the non-critical trajectory area, represents the adjustment factor.

[0103] It should be noted that the focus loss function is used to strengthen the attention distribution of the video point tracking model in the key trajectory area, guide the attention mechanism to focus on the semantically salient areas, and suppress attention to non-key trajectory areas to improve the accuracy of semantic alignment.

[0104] Furthermore, there is a total loss when jointly optimizing the video point tracking model using language generation loss and focus loss. The corresponding relationship is as follows:

[0105] ;

[0106] in, represents the total loss, represents a hyperparameter.

[0107] Step 7: Decode the multi-source features after focus optimization to obtain the final video description result.

[0108] Furthermore, the focused optimized multi-source features are input into a text encoder to generate a natural language description of the video content word by word.

[0109] Please refer to Figure 3 The present invention also proposes a video description system based on video point trajectory constraints, the system comprising:

[0110] Feature encoding module, used to:

[0111] The key frame images in the video are sampled, and the point tracking algorithm is used to obtain the spatiotemporal trajectory of the pixel points between consecutive frames to construct a set of trajectory segments of the video target;

[0112] Based on the trajectory segment set of the video target, the visual features of the frame corresponding to the trajectory segment are extracted from the trajectory segment, and the visual features of the frame corresponding to the same trajectory segment are average pooled to obtain the trajectory features;

[0113] Encode the text description to obtain text features;

[0114] Multimodal fusion module, used for:

[0115] For text features, visual features, and trajectory features, semantic alignment is first performed and then multi-head attention feature fusion is performed to obtain the attention allocation weight vector and the corresponding output features after multi-head attention.

[0116] The attention allocation weight vector is used to calculate the semantic relevance score of the visual area corresponding to the trajectory segment to obtain the comprehensive trajectory relevance score. The scores are sorted in descending order by relevance score, and the relevance scores are accumulated and a threshold is set to obtain the key trajectory.

[0117] The language generation loss is constructed based on the corresponding output features after multi-head attention, and the focus loss is constructed based on the comprehensive trajectory correlation score and key trajectory. The language generation loss and focus loss are used to jointly optimize the video point tracking model to obtain the optimized video point tracking model.

[0118] Utilize the optimized video point tracking model to obtain the multi-source features after focus optimization;

[0119] Video description generation module, used to:

[0120] The multi-source features after focus optimization are decoded to obtain the final video description result.

[0121] It should be understood that various components of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0122] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0123] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A video description method based on video point trajectory constraints, characterized in that: The method comprises the following steps: Step 1: Sample the key frame images in the video and use the point tracking algorithm to obtain the spatiotemporal trajectory of the pixel points between consecutive frames to construct a set of trajectory segments of the video target; Step 2: Based on the trajectory segment set of the video target, the visual features of the frame corresponding to the trajectory segment are extracted from the trajectory segment, and the visual features of the frame corresponding to the same trajectory segment are average pooled to obtain the trajectory features; Step 3: Encode the text description to obtain text features; Step 4: Semantically align text features, visual features, and trajectory features before performing multi-head attention feature fusion to obtain the attention allocation weight vector and the corresponding output features after multi-head attention. Step 5: Use the attention allocation weight vector to calculate the semantic relevance score of the visual area corresponding to the trajectory segment to obtain the comprehensive trajectory relevance score, and sort them in descending order of relevance score. Accumulate the relevance score and set the threshold to obtain the key trajectory; Step 6: Construct a language generation loss based on the corresponding output features after multi-head attention, construct a focus loss based on the comprehensive trajectory correlation score and key trajectories, and use the language generation loss and focus loss to jointly optimize the video point tracking model to obtain the optimized video point tracking model; Utilize the optimized video point tracking model to obtain the multi-source features after focus optimization; Step 7: Decode the multi-source features after focus optimization to obtain the final video description result; In step 5, the attention allocation weight vector is used to calculate the semantic relevance score of the visual area corresponding to the trajectory segment to obtain the comprehensive trajectory relevance score, and the scores are sorted in descending order by the relevance score. The relevance scores are accumulated and a threshold is set to obtain the key trajectory. The specific steps include the following: Merge visual features and trajectory features to obtain multi-source video features; The attention allocation weight vector is used to calculate the semantic relevance score of multi-source video features and text features, and the relevance score vector of multi-source video features and text features is obtained; Normalizing the correlation score vectors of the multi-source video features and the text features to obtain the normalized correlation score vectors of the multi-source video features and the text features; Perform comprehensive trajectory correlation calculation on the normalized correlation score vectors of multi-source video features and text features to obtain a comprehensive trajectory correlation score; The comprehensive trajectory correlation scores are normalized and sorted in descending order to obtain the correlation scores corresponding to the sorted trajectories; The correlation scores corresponding to the sorted trajectories are defined as a set to obtain the trajectory accumulation set; Based on the trajectory accumulation set, the correlation cumulative score corresponding to the sorted trajectories is calculated to obtain the trajectory segment correlation cumulative score vector; Set the cumulative score threshold and filter the trajectory segment correlation cumulative score vector to obtain the key trajectory; The attention allocation weight vector is used to calculate the semantic relevance score of multi-source video features and text features, and the relevance score vector of multi-source video features and text features is obtained. The relationship between the corresponding process is as follows: ； in, Represents the correlation score vector of multi-source video features and text features The correlation scores, represents the total number of relevance scores; In the step of normalizing the correlation score vectors of the multi-source video features and the text features to obtain the normalized correlation score vectors of the multi-source video features and the text features, the corresponding process has the following relationship: ； in, Represents the normalized correlation score vector of multi-source video features and text features, represents the normalized correlation score of the first visual feature, Indicates the The normalized correlation score of the visual features, represents the normalized correlation score of the first trajectory feature, Indicates the The correlation score of the normalized trajectory features; In the step of calculating the comprehensive trajectory correlation of the correlation score vector after normalization of multi-source video features and text features to obtain the comprehensive trajectory correlation score, the corresponding process has the following relationship: ； in, Indicates the The comprehensive trajectory correlation score of the trajectories, represents the comprehensive trajectory correlation score function, Indicates the tracks; In the step of defining the set of correlation scores corresponding to the sorted trajectories and obtaining the trajectory accumulation set, the corresponding process has the following relationship: ； in, represents the trajectory accumulation set, Indicates the first Tracks, represents the set of positive integers; In the step of calculating the correlation cumulative score of the correlation scores corresponding to the sorted trajectories to obtain the trajectory segment correlation cumulative score vector, the corresponding process has the following relationship: ； in, represents the cumulative score vector of trajectory segment correlation, represents the cumulative score of the first trajectory segment relevance, Indicates the The cumulative correlation score of trajectory segments, Indicates the Trajectory accumulation set Middle Track segments, Represents a trajectory segment the corresponding integrated trajectory correlation; In the steps of setting the cumulative score threshold and screening the trajectory segment correlation cumulative score vector to obtain the key trajectory, the corresponding process has the following relationship: ； in, represents the critical trajectory, Represents a set function that obtains the maximum value of the parameter. Indicates the cumulative score threshold.

2. The video description method based on video point trajectory constraints according to claim 1, characterized in that: In step 1, the key frame images in the video are sampled, and the point tracking algorithm is used to obtain the spatiotemporal trajectory of the pixel points between consecutive frames to construct a set of trajectory segments of the video target. The corresponding process has the following relationship: ； in, Represents the first space-time trajectory segment in the space-time trajectory segment set, Represents the first A fragment of space-time trajectory, represents the video point tracking model, Represents a collection of frames uniformly sampled from a video.

3. The video description method based on video point trajectory constraints according to claim 2, characterized in that: In step 2, based on the trajectory segment set of the video target, visual features of the frames corresponding to the trajectory segments are extracted from the trajectory segments, and the visual features of the frames corresponding to the same trajectory segments are average pooled to obtain trajectory features, which specifically includes the following steps: Based on the trajectory segment set of the video target, spatial perception calculation is performed on the frames uniformly sampled from the video to obtain the grid feature vector of each frame; The grid feature vectors of each frame are spliced ​​together to obtain the visual feature vector; The visual feature vectors on the same trajectory segment are average pooled to obtain the trajectory features.

4. The video description method based on video point trajectory constraints according to claim 3, characterized in that: Perform spatial perception calculations on frames uniformly sampled from the video to obtain the grid feature vector of each frame. The corresponding relationship is as follows: ； in, Represents each frame Grid feature vector, represents the grid feature vector of the first frame, Indicates the Frame grid feature vector; In the step of splicing the grid feature vectors of each frame to obtain the visual feature vector, the corresponding process has the following relationship: ； in, represents the visual feature vector, represents the first grid feature vector, represents the total number of visual grid vectors extracted from the entire video, Represents a video frame, represents the total number of grid feature vectors; In the step of performing average pooling on the visual feature vectors on the same trajectory segment to obtain the trajectory features, the corresponding process has the following relationship: ； in, Represents the trajectory features corresponding to all spatiotemporal trajectory segments of the video, represents the first visual feature vector, represents the total number of visual feature vectors, Indicates the visual feature vectors, Represents the current spatiotemporal trajectory segment The trajectory length, Represents a spatiotemporal trajectory segment in a trajectory set On the A visual feature.

5. The video description method based on video point trajectory constraints according to claim 4 is characterized in that: In step 4, the text features, visual features, and trajectory features are first semantically aligned and then multi-head attention feature fusion is performed to obtain the attention allocation weight vector and the corresponding output features after multi-head attention, which specifically includes the following steps: The text features, visual features and trajectory features are spliced ​​together to obtain a comprehensive representation of the multimodal input; Multi-head self-attention feature fusion is performed on the comprehensive representation of multimodal input to obtain the attention allocation weight vector and the corresponding output features after multi-head attention.

6. The video description method based on video point trajectory constraints according to claim 5, characterized in that: Multi-head self-attention feature fusion is performed on the comprehensive representation of multimodal input to obtain the attention allocation weight vector and the corresponding output feature after multi-head attention. The relationship between the corresponding process is as follows: ； in, represents the comprehensive representation of multimodal inputs and the attention weight vector in the fusion process, Indicates time, represents the attention weight vector The first attention weight vector in , represents the attention weight vector The The attention weight value, represents the normalization function, represents the initial encoding vector of the predicted word, represents the first embedding matrix, represents the comprehensive representation matrix of multimodal input, represents the second embedding matrix, represents transpose, Represents the dimension of the feature, Represents the corresponding output features after multi-head attention, represents the attention weight vector The The attention weight value, Indicates the A comprehensive representation vector of multimodal inputs.

7. The video description method based on video point trajectory constraints according to claim 1, characterized in that: In step 6, the language generation loss is constructed based on the corresponding output features after multi-head attention. The relationship between the corresponding process is as follows: ； in, Represents the generated word, represents the probability distribution of the output words in the vocabulary, represents the fully connected layer, represents the language generation loss, Indicates the number of generated words, Indicates the generated words, Indicates the first 1 to word; In the step of constructing the focus loss based on the comprehensive trajectory correlation score and the key trajectory, the relationship between the corresponding process is as follows: ； in, represents the focus loss, represents the logarithmic function, Indicates the sequence number, represents the sum of attention weights of key track areas, Indicates that the cumulative score threshold is met Key Trajectory , represents the coefficient as the weight of the non-critical trajectory area, represents the adjustment factor.

8. A video description system based on video point trajectory constraints, characterized in that: The system applies the video description method based on video point trajectory constraints as described in any one of claims 1 to 7 above, and the system includes: Feature encoding module, used to: The key frame images in the video are sampled, and the point tracking algorithm is used to obtain the spatiotemporal trajectory of the pixel points between consecutive frames to construct a set of trajectory segments of the video target; Based on the trajectory segment set of the video target, the visual features of the frame corresponding to the trajectory segment are extracted from the trajectory segment, and the visual features of the frame corresponding to the same trajectory segment are average pooled to obtain the trajectory features; Encode the text description to obtain text features; Multimodal fusion module, used for: For text features, visual features, and trajectory features, semantic alignment is first performed and then multi-head attention feature fusion is performed to obtain the attention allocation weight vector and the corresponding output features after multi-head attention. The attention allocation weight vector is used to calculate the semantic relevance score of the visual area corresponding to the trajectory segment to obtain the comprehensive trajectory relevance score. The scores are sorted in descending order by relevance score, and the relevance scores are accumulated and a threshold is set to obtain the key trajectory. The language generation loss is constructed based on the corresponding output features after multi-head attention, and the focus loss is constructed based on the comprehensive trajectory correlation score and key trajectory. The language generation loss and focus loss are used to jointly optimize the video point tracking model to obtain the optimized video point tracking model. Utilize the optimized video point tracking model to obtain the multi-source features after focus optimization; Video description generation module, used to: The multi-source features after focus optimization are decoded to obtain the final video description result.

Images