Video description method and system based on video point trajectory constraint

Abstract

The invention provides a video point trajectory constraint-based video description method and system. The method comprises the following steps of: sampling a key frame image and acquiring a space-time trajectory of continuous inter-frame pixel points by using a point tracking algorithm; performing average pooling operation on the visual features of the corresponding frames of the same track fragment; performing semantic alignment on the text features, the visual features and the track features and then performing multi-head attention feature fusion; semantic correlation score calculation is carried out on the visual areas corresponding to the track fragments, correlation scores are arranged in a descending order, the correlation scores are accumulated, and a threshold value is set; jointly optimizing a video point tracking model by using language generation loss and focusing loss; and decoding the multi-source features after focusing optimization to obtain a final video description result. The method introduces a video point trajectory aggregation strategy, explicitly models the dynamic characteristics of a target in a space-time dimension, retains the spatial appearance and time coherence of an object, and effectively solves the problems of semantic fracture and description fragmentation in a complex scene.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and video processing, and particularly relates to a video description method and system based on video point trajectory constraints. Background Art

[0002] Video content description technology aims to automatically generate semantically reasonable and linguistically coherent natural language sentences for input videos, which is a core task in the cross - field of computer vision and natural language processing and is widely used in scenarios such as video retrieval, intelligent monitoring, and visual impairment assistance. Existing methods usually adopt frame - level visual feature extraction and sequence modeling techniques, combined with attention mechanisms for multi - modal fusion and text generation. However, in complex dynamic scenarios such as occlusion, multi - object interaction, or fast movement, existing models are difficult to continuously track target semantics, resulting in problems such as semantic incoherence, fragmentation, or redundancy in the description; in addition, the cross - modal attention mechanism lacks precise guidance, often causing the model to focus on irrelevant regions and having insufficient semantic alignment accuracy. Summary of the Invention

[0003] In view of the above situation, the main object of the present invention is to propose a video description method and system based on video point trajectory constraints to solve the above - mentioned technical problems.

[0004] The present invention proposes a video description method based on video point trajectory constraints, and the method includes the following steps: Step 1: Sample key - frame images in the video, and use a point - tracking algorithm to obtain the spatio - temporal trajectories of pixel points between consecutive frames, so as to construct a set of trajectory segments of video targets; Step 2: Based on the set of trajectory segments of video targets, extract the visual features of the frames corresponding to the trajectory segments for the trajectory segments, and perform average pooling operations on the visual features of the frames corresponding to the same trajectory segment to obtain trajectory features; Step 3: Encode the text description to obtain text features; Step 4: For the text features, visual features, and trajectory features, first perform semantic alignment and then perform multi - head attention feature fusion to obtain an attention allocation weight vector and the corresponding output features after multi - head attention; Step 5: Use the attention allocation weight vector to calculate the semantic correlation scores for the visual regions corresponding to the trajectory segments, obtain the comprehensive trajectory correlation scores, sort the comprehensive trajectory correlation scores in descending order, accumulate the correlation scores and set a threshold to obtain key trajectories; Step 6: Construct a language generation loss based on the output features corresponding to the multi - head attention, construct a focus loss based on the comprehensive trajectory correlation scores and key trajectories, and jointly optimize the video point - tracking model using the language generation loss and the focus loss to obtain an optimized video point - tracking model; Use the optimized video point tracking model to obtain multi-source features after focus optimization; Step 7: Decode the multi-source features after focus optimization to obtain the final video description result.

[0005] The present invention also proposes a video description system based on video point trajectory constraints, and the system includes: A feature encoding module for: Sample key frame images in the video, and use the point tracking algorithm to obtain the spatio-temporal trajectories of pixel points between consecutive frames, so as to construct a set of trajectory segments of video targets; Based on the set of trajectory segments of video targets, extract the visual features of the frames corresponding to the trajectory segments, and perform average pooling operations on the visual features of the frames corresponding to the same trajectory segment to obtain trajectory features; Encode the text description to obtain text features; A multi-modal fusion module for: For text features, visual features, and trajectory features, first perform semantic alignment and then perform multi-head attention feature fusion to obtain an attention distribution weight vector and the corresponding output features after multi-head attention; Use the attention distribution weight vector to calculate the semantic correlation score for the visual regions corresponding to the trajectory segments, obtain the comprehensive trajectory correlation score, sort the correlation scores in descending order, accumulate the correlation scores and set a threshold to obtain key trajectories; Construct a language generation loss based on the output features corresponding to the multi-head attention, construct a focus loss based on the comprehensive trajectory correlation score and the key trajectories, and jointly optimize the video point tracking model using the language generation loss and the focus loss to obtain an optimized video point tracking model; Use the optimized video point tracking model to obtain multi-source features after focus optimization; A video description generation module for: Decode the multi-source features after focus optimization to obtain the final video description result.

[0006] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By introducing a video point trajectory aggregation strategy, the present invention explicitly models the dynamic features of the target in the spatio-temporal dimension, retains the spatial appearance and temporal coherence of the object, and effectively solves the problems of semantic break and description fragmentation in complex scenes.

[0007] 2. The present invention proposes an unsupervised key trajectory focus learning method, uses the attention weight distribution between the trajectory and the text to screen key trajectories, guides the model to focus on semantically relevant regions, suppresses background interference, and significantly improves the semantic matching degree between the video and the text modalities.

[0008] 3. The key trajectory mining and focusing mechanism proposed by the present invention are both realized based on the internal features and attention structure of the model, without relying on manually labeled attention tags, and have stronger adaptability and practicability, making them suitable for deployment in actual application systems.

[0009] Additional aspects and advantages of the present invention will be given in part in the following description, will become apparent in part from the following description, or will be understood through the embodiments of the present invention. Brief Description of the Drawings

[0010] Figure 1 It is a flowchart of the steps of the video description method based on video point trajectory constraints proposed by the present invention.

[0011] Figure 2 It is an overall architecture diagram of the video description method based on video point trajectory constraints proposed by the present invention.

[0012] Figure 3 It is a schematic structural diagram of the video description system based on video point trajectory constraints proposed by the present invention. Detailed Embodiments

[0013] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals are the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0014] Referring to the following description and drawings, these and other aspects of the embodiments of the present invention will become clear. In these descriptions and drawings, some specific embodiments of the embodiments of the present invention are specifically disclosed as some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited by this.

[0015] Please refer to Figure 1 , an embodiment of the present invention proposes a video description method based on video point trajectory constraints, and the method includes the following steps: Step 1. Sample the key frame images in the video, and use the point tracking algorithm to obtain the spatio-temporal trajectories of pixel points between consecutive frames, so as to construct a set of trajectory segments of the video target.

[0016] Please refer to Figure 2 , in Step 1, sample the key frame images in the video, and use the point tracking algorithm to obtain the spatio-temporal trajectories of pixel points between consecutive frames, so as to construct a set of trajectory segments of the video target. The corresponding relationships in the process are as follows: ; Among them, represents the first spatio-temporal trajectory segment in the set of spatio-temporal trajectory segments, represents the th spatio-temporal trajectory segment in the set of spatio-temporal trajectory segments, represents the video point tracking model, represents the set of frames uniformly sampled from the video.

[0017] It should be noted that in Figure 2 , a video point trajectory estimator and a video encoder are used to sample the key frame images of the video to obtain the position changes and motion trajectories of each pixel point in the video frame.

[0018] Step 2: Based on the set of trajectory segments of the video object, extract the visual features of the frames corresponding to the trajectory segments, and perform an average pooling operation on the visual features of the frames corresponding to the same trajectory segment to obtain the trajectory features.

[0019] In Step 2, based on the set of trajectory segments of the video object, extract the visual features of the frames corresponding to the trajectory segments, and perform an average pooling operation on the visual features of the frames corresponding to the same trajectory segment to obtain the trajectory features, which specifically includes the following steps: Based on the set of trajectory segments of the video object, perform spatial perception calculation on the frames uniformly sampled from the video to obtain a grid feature vector for each frame; Perform a concatenation process on the grid feature vectors of each frame to obtain a visual feature vector; Perform an average pooling process on the visual feature vectors on the same trajectory segment to obtain the trajectory features.

[0020] Perform spatial perception calculation on the frames uniformly sampled from the video to obtain a grid feature vector for each frame, and the relational formula existing in the corresponding process is as follows: ; where represents the grid feature vector of each frame , represents the grid feature vector of the first frame, represents the th grid feature vector of the frame; It should be noted that in Figure 2 , a video encoder is used to encode the frames uniformly sampled from the video.

[0021] In the step of performing a concatenation process on the grid feature vectors of each frame to obtain a visual feature vector, the relational formula existing in the corresponding process is as follows: ; where represents the visual feature vector, Represents the first grid feature vector, Represents the total number of visual grid vectors extracted from the entire video, Represents the video frame, Represents the total number of grid feature vectors; In the step of performing average pooling on the visual feature vectors on the same trajectory segment to obtain the trajectory feature, the relational expressions in the corresponding process are as follows: ; Among them, Represents the trajectory features corresponding to all spatio-temporal trajectory segments of the video, Represents the first visual feature vector, Represents the total number of visual feature vectors, Represents the th visual feature vector, Represents the current spatio-temporal trajectory segment of the trajectory length, Represents the spatio-temporal trajectory segment in the trajectory set on the th visual feature.

[0022] Step 3: Encode the text description to obtain text features.

[0023] In step 3, a text encoder is used to encode the text description.

[0024] Step 4: For the text features, visual features, and trajectory features, first perform semantic alignment and then perform multi-head attention feature fusion to obtain the attention distribution weight vector and the corresponding output features after multi-head attention.

[0025] In step 4, for the text features, visual features, and trajectory features, first perform semantic alignment and then perform multi-head attention feature fusion to obtain the attention distribution weight vector and the corresponding output features after multi-head attention, which specifically includes the following steps: Perform splicing processing on the text features, visual features, and trajectory features to obtain a comprehensive representation of the multi-modal input; Perform multi-head self-attention feature fusion on the comprehensive representation of the multi-modal input to obtain the attention distribution weight vector and the corresponding output features after multi-head attention.

[0026] Perform multi-head self-attention feature fusion on the comprehensive representation of the multi-modal input to obtain the attention distribution weight vector and the corresponding output features after multi-head attention, and the relational expressions in the corresponding process are as follows: ; Among them, Represents the attention weight vector in the comprehensive representation of multimodal input during the fusion process, Represents time, Represents the attention weight vector The first attention weight vector in, Represents the attention weight vector In the th attention weight value, Represents the normalization function, Represents the initial encoding vector of the predicted word, Represents the first embedding matrix, Represents the comprehensive representation matrix of multimodal input, Represents the second embedding matrix, Represents the transpose, Represents the dimension of the feature, Represents the output feature corresponding to the multi-head attention, Represents the attention weight vector In the th attention weight value, Represents the th comprehensive representation vector of multimodal input.

[0027] Step 5: Use the attention distribution weight vector to calculate the semantic correlation score for the visual region corresponding to the trajectory segment, obtain the comprehensive trajectory correlation score, sort it in descending order of the correlation score, accumulate the correlation score and set a threshold to obtain the key trajectory.

[0028] In step 5, use the attention distribution weight vector to calculate the semantic correlation score for the visual region corresponding to the trajectory segment, obtain the comprehensive trajectory correlation score, sort it in descending order of the correlation score, accumulate the correlation score and set a threshold to obtain the key trajectory, which specifically includes the following steps: Merge the visual feature and the trajectory feature to obtain the multi-source video feature; Use the attention distribution weight vector to calculate the semantic correlation score for the multi-source video feature and the text feature to obtain the correlation score vector of the multi-source video feature and the text feature; Normalize the correlation score vector of the multi-source video feature and the text feature to obtain the normalized correlation score vector of the multi-source video feature and the text feature; Perform comprehensive trajectory correlation calculation on the normalized correlation score vector of the multi-source video feature and the text feature to obtain the comprehensive trajectory correlation score; Perform overall normalization processing and descending order sorting processing on the comprehensive trajectory correlation score in sequence to obtain the correlation score corresponding to the sorted trajectory; Define a set for the correlation scores corresponding to the sorted trajectories to obtain a trajectory cumulative set; Based on the trajectory cumulative set, calculate the cumulative correlation scores for the correlation scores corresponding to the sorted trajectories to obtain a trajectory segment cumulative correlation score vector; Set a cumulative score threshold and filter the trajectory segment cumulative correlation score vector to obtain key trajectories.

[0029] Calculate the semantic correlation scores for the multi-source video features and text features using the attention distribution weight vector to obtain a correlation score vector for the multi-source video features and text features. The corresponding relationship in the process is as follows: ; Among them, represents the correlation score vector for the multi-source video features and text features the th correlation score in represents the total number of correlation scores; In the step of normalizing the correlation score vector for the multi-source video features and text features to obtain a normalized correlation score vector for the multi-source video features and text features, the corresponding relationship in the process is as follows: ; Among them, represents the normalized correlation score vector for the multi-source video features and text features, represents the normalized correlation score of the first visual feature, represents the th normalized correlation score of the visual feature, represents the normalized correlation score of the first trajectory feature, represents the th normalized correlation score of the trajectory feature; In the step of calculating the comprehensive trajectory correlation for the normalized correlation score vector of the multi-source video features and text features to obtain a comprehensive trajectory correlation score, the corresponding relationship in the process is as follows: ; Among them, represents the comprehensive trajectory correlation score of the th trajectory, represents the comprehensive trajectory correlation score function, represents the th trajectory; In the step of defining a set for the correlation scores corresponding to the sorted trajectories to obtain a trajectory cumulative set, the corresponding relationship in the process is as follows: ; Among them, represents the trajectory cumulative set, represents the th trajectory after sorting, represents the set of positive integers; In the step of calculating the correlation cumulative score of the sorted trajectories to obtain the correlation cumulative score vector of the trajectory segments, the relational expressions existing in the corresponding process are as follows: ; Among them, represents the correlation cumulative score vector of the trajectory segments, represents the first correlation cumulative score of the trajectory segments, represents the th correlation cumulative score of the trajectory segments, represents the th trajectory cumulative set in the th trajectory segment, represents the comprehensive trajectory correlation corresponding to the trajectory segment ; In the step of setting the cumulative score threshold and screening the correlation cumulative score vector of the trajectory segments to obtain the key trajectories, the relational expressions existing in the corresponding process are as follows: ; Among them, represents the key trajectory, represents the set function of obtaining the maximum value parameter, represents the cumulative score threshold.

[0030] It should be noted that setting the cumulative score threshold can screen and retain the key trajectories with greater semantic contributions.

[0031] Step 6: Construct a language generation loss based on the output features corresponding to the multi-head attention, construct a focus loss based on the comprehensive trajectory correlation score and the key trajectories, and jointly optimize the video point tracking model using the language generation loss and the focus loss to obtain an optimized video point tracking model; Use the optimized video point tracking model to obtain the multi-source features with optimized focus.

[0032] In Step 6, when constructing a language generation loss based on the output features corresponding to the multi-head attention, the relational expressions existing in the corresponding process are as follows: ; Among them, represents the generated word, represents the probability distribution of the output word on the vocabulary, Indicates a fully connected layer, Indicates the language generation loss, Indicates the number of generated words, Indicates the th generated word, Indicates the first 1 to words; It should be noted that, is sampled from i.e., the word with the highest predicted probability on the vocabulary list.

[0033] In the step of constructing the focal loss based on the comprehensive trajectory correlation score and the key trajectory, the relational expressions existing in the corresponding process are as follows: ; Among them, Indicates the focal loss, Indicates the logarithmic function, Indicates the number of sequences, Indicates the sum of the attention weights in the key trajectory region, Indicates the key trajectory that satisfies the cumulative score threshold , Indicates the coefficient used as the weight of the non-key trajectory region, Indicates the adjustment factor.

[0034] It should be noted that the focal loss function is used to strengthen the attention distribution of the video point tracking model in the key trajectory region, guide the attention mechanism to focus on the semantically significant region, and at the same time suppress the attention to the non-key trajectory region to improve the semantic alignment accuracy.

[0035] Furthermore, when jointly optimizing the video point tracking model using the language generation loss and the focal loss, there is also a total loss, and the relational expressions existing in the corresponding process are as follows: ; Among them, Indicates the total loss, Indicates the hyperparameter.

[0036] Step 7, Decode the multi-source features after focal optimization to obtain the final video description result.

[0037] Furthermore, the multi-source features after focal optimization are input into the text encoder to generate a natural language description of the video content word by word.

[0038] Please refer to Figure 3 , The present invention also proposes a video description system based on video point trajectory constraints, and the system includes: A feature encoding module, used for:​ Sample the key-frame images in the video, and use the point tracking algorithm to obtain the spatio-temporal trajectories of pixel points between consecutive frames, so as to construct a set of trajectory segments of the video target; Based on the set of trajectory segments of the video target, extract the visual features of the frames corresponding to the trajectory segments for the trajectory segments, and perform average pooling operations on the visual features of the frames corresponding to the same trajectory segment to obtain trajectory features; Encode the text description to obtain text features; A multi-modal fusion module, which is used for: For the text features, visual features, and trajectory features, first perform semantic alignment and then perform multi-head attention feature fusion to obtain an attention allocation weight vector and the corresponding output features after multi-head attention; Use the attention allocation weight vector to calculate the semantic correlation score for the visual regions corresponding to the trajectory segments, obtain the comprehensive trajectory correlation score, sort the scores in descending order of the correlation score, accumulate the correlation scores and set a threshold to obtain the key trajectories; Construct a language generation loss based on the output features corresponding to the multi-head attention, construct a focus loss based on the comprehensive trajectory correlation score and the key trajectories, and jointly optimize the video point tracking model using the language generation loss and the focus loss to obtain an optimized video point tracking model; Use the optimized video point tracking model to obtain multi-source features with optimized focus; A video description generation module, which is used for: Decode the multi-source features with optimized focus to obtain the final video description result.

[0039] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following well-known technologies in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0040] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0041] The above-described embodiments merely represent several implementation manners of the present invention. The descriptions are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims

1. A video description method based on video point trajectory constraints, characterized in that The method includes the following steps: Step 1: Sample key-frame images in the video, and use the point-tracking algorithm to obtain the spatio-temporal trajectories of pixel points between consecutive frames, so as to construct a set of trajectory segments of the video target; Step 2: Based on the set of trajectory segments of the video target, extract the visual features of the frames corresponding to the trajectory segments, and perform average pooling operations on the visual features of the frames corresponding to the same trajectory segment to obtain trajectory features; Step 3: Encode the text description to obtain text features; Step 4: For the text features, visual features, and trajectory features, first perform semantic alignment and then perform multi-head attention feature fusion to obtain an attention allocation weight vector and the corresponding output features after multi-head attention; Step 5: Use the attention allocation weight vector to calculate the semantic correlation score for the visual regions corresponding to the trajectory segments, obtain the comprehensive trajectory correlation score, sort the scores in descending order of correlation, accumulate the correlation scores and set a threshold to obtain the key trajectories; Step 6: Construct a language generation loss based on the output features corresponding to the multi-head attention, construct a focus loss based on the comprehensive trajectory correlation score and the key trajectories, and jointly optimize the video point-tracking model using the language generation loss and the focus loss to obtain an optimized video point-tracking model; Use the optimized video point-tracking model to obtain multi-source features with optimized focus; Step 7: Decode the multi-source features with optimized focus to obtain the final video description result.

2. The video description method based on video point trajectory constraint according to claim 1, wherein In Step 1, when sampling key-frame images in the video and using the point-tracking algorithm to obtain the spatio-temporal trajectories of pixel points between consecutive frames to construct a set of trajectory segments of the video target, the relational expressions for the corresponding process are as follows: ； Among them, represents the first spatio-temporal trajectory segment in the set of spatio-temporal trajectory segments, represents the th spatio-temporal trajectory segment in the set of spatio-temporal trajectory segments, represents the video point tracking model, represents the set of frames evenly sampled from the video.

3. The video description method based on video point trajectory constraints according to claim 2, wherein In Step 2, based on the set of trajectory segments of the video target, when extracting the visual features of the frames corresponding to the trajectory segments and performing average pooling operations on the visual features of the frames corresponding to the same trajectory segment to obtain trajectory features, it specifically includes the following steps: Based on the set of trajectory segments of the video target, perform spatial perception calculations on the frames uniformly sampled from the video to obtain a grid feature vector for each frame; Perform splicing processing on the grid feature vectors of each frame to obtain a visual feature vector; Perform average pooling processing on the visual feature vectors on the same trajectory segment to obtain trajectory features.

4. The video description method based on video point trajectory constraint according to claim 3, characterized in that, When performing spatial perception calculations on the frames uniformly sampled from the video to obtain a grid feature vector for each frame, the relational expressions for the corresponding process are as follows: ； Among them, represents each frame grid feature vector, represents the grid feature vector of the first frame, represents the grid feature vector of the frame; In the step of performing splicing processing on the grid feature vectors of each frame to obtain a visual feature vector, the relational expressions for the corresponding process are as follows: ； Among them, represents the visual feature vector, represents the first grid feature vector, represents the total number of visual grid vectors extracted from the entire video, represents the video frame, represents the total number of grid feature vectors; In the step of performing average pooling processing on the visual feature vectors on the same trajectory segment to obtain trajectory features, the relational expressions for the corresponding process are as follows: ； Among them, represents the trajectory features corresponding to all spatio-temporal trajectory segments of the video, represents the first visual feature vector, represents the total number of visual feature vectors, represents the th visual feature vector, represents the current spatio-temporal trajectory segment 's trajectory length, represents the spatio-temporal trajectory segment in the trajectory set on the th visual feature.

5. The video description method based on video point trajectory constraint according to claim 4, characterized in that, In Step 4, when performing semantic alignment and then multi-head attention feature fusion on the text features, visual features, and trajectory features to obtain an attention allocation weight vector and the corresponding output features after multi-head attention, it specifically includes the following steps: Perform splicing processing on the text features, visual features, and trajectory features to obtain a comprehensive representation of the multi-modal input; Perform multi-head self-attention feature fusion on the comprehensive representation of multi-modal input to obtain the attention allocation weight vector and the corresponding output features after multi-head attention.

6. The video description method based on video point trajectory constraint according to claim 5, wherein, Perform multi-head self-attention feature fusion on the comprehensive representation of multi-modal input to obtain the attention allocation weight vector and the corresponding output features after multi-head attention. The relational expressions existing in the corresponding process are as follows: ； Among them, represents the attention weight vector in the comprehensive representation of multimodal input during the fusion process, represents time, represents the attention weight vector the first attention weight vector in, represents the attention weight vector in the represents the normalization function, represents the initial encoding vector of the predicted word, represents the first embedding matrix, represents the comprehensive representation matrix of multimodal input, represents the second embedding matrix, represents the transpose, represents the dimension of the feature, represents the output feature corresponding to the multi-head attention, represents the attention weight vector in the represents the th comprehensive representation vector of multimodal input.

7. The video description method based on video point trajectory constraint according to claim 6, wherein In step 5, use the attention allocation weight vector to calculate the semantic correlation score for the visual region corresponding to the trajectory segment, obtain the comprehensive trajectory correlation score, sort it in descending order of the correlation score, accumulate the correlation score, and set a threshold to obtain the key trajectory. The specific steps are as follows: Merge the visual features and trajectory features to obtain multi-source video features; Use the attention allocation weight vector to calculate the semantic correlation score for the multi-source video features and text features to obtain the correlation score vector of the multi-source video features and text features; Perform normalization processing on the correlation score vector of the multi-source video features and text features to obtain the normalized correlation score vector of the multi-source video features and text features; Perform comprehensive trajectory correlation calculation on the normalized correlation score vector of the multi-source video features and text features to obtain the comprehensive trajectory correlation score; Perform overall normalization processing and descending order sorting processing on the comprehensive trajectory correlation score in sequence to obtain the correlation score corresponding to the sorted trajectory; Define a set for the correlation score corresponding to the sorted trajectory to obtain a trajectory cumulative set; Based on the trajectory cumulative set, perform correlation cumulative score calculation on the correlation score corresponding to the sorted trajectory to obtain the trajectory segment correlation cumulative score vector; Set a cumulative score threshold and filter the trajectory segment correlation cumulative score vector to obtain the key trajectory.

8. The video description method based on video point trajectory constraints according to claim 7, wherein Use the attention allocation weight vector to calculate the semantic correlation score for the multi-source video features and text features to obtain the correlation score vector of the multi-source video features and text features. The relational expressions existing in the corresponding process are as follows: ； Among them, represents the correlation score vector of multi-source video features and text features in the th correlation score, indicating the total number of correlation scores; In the step of performing normalization processing on the correlation score vector of the multi-source video features and text features to obtain the normalized correlation score vector of the multi-source video features and text features, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the correlation score vector after normalization of multi-source video features and text features, represents the correlation score after normalization of the first visual feature, represents the correlation score after normalization of the th visual feature, represents the correlation score after normalization of the first trajectory feature, represents the correlation score after normalization of the th trajectory feature; In the step of performing comprehensive trajectory correlation calculation on the normalized correlation score vector of the multi-source video features and text features to obtain the comprehensive trajectory correlation score, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the comprehensive trajectory correlation score of the th trajectory, represents the th trajectory; In the step of defining a set for the correlation score corresponding to the sorted trajectory to obtain a trajectory cumulative set, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the trajectory cumulative set, represents the th trajectory after sorting, represents the set of positive integers; In the step of performing correlation cumulative score calculation on the correlation score corresponding to the sorted trajectory to obtain the trajectory segment correlation cumulative score vector, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the cumulative score vector of track segment correlation, represents the cumulative score of the correlation of the first track segment, represents the cumulative score of the correlation of the th track segment, represents the th track cumulative set the th track segment in corresponding to the comprehensive track correlation; In the step of setting a cumulative score threshold and filtering the trajectory segment correlation cumulative score vector to obtain the key trajectory, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the key trajectory, represents the set function for obtaining the maximum value parameter, represents the cumulative score threshold.

9. The video description method based on video point trajectory constraint according to claim 8, wherein In step 6, construct a language generation loss based on the corresponding output features after multi-head attention. The relational expressions existing in the corresponding process are as follows: ； Among them, represents the generated word, represents the probability distribution of the output word on the vocabulary, represents the fully connected layer, represents the language generation loss, represents the number of generated words, represents the th generated word, represents the first 1 to words; In the step of constructing the focal loss based on the comprehensive trajectory correlation score and the key trajectory, the relational expressions existing in the corresponding process are as follows: ； Among them, represents the focal loss, represents the logarithmic function, represents the sequence number, represents the sum of the attention weights of the key trajectory region, represents satisfying the cumulative score threshold of the key trajectory , represents the coefficient as the weight of the non-key trajectory region, represents the adjustment factor.

10. A video description system based on video point trajectory constraints, characterized in that, The system applies the video description method based on video point trajectory constraints according to any one of the above claims 1 to 9. The system includes: A feature encoding module, configured to: Sample key frame images in the video, and use a point tracking algorithm to obtain the spatio-temporal trajectories of pixel points between consecutive frames, so as to construct a set of trajectory segments of the video target; Based on the set of trajectory segments of the video target, extract the visual features of the frames corresponding to the trajectory segments for the trajectory segments, and perform average pooling operations on the visual features of the frames corresponding to the same trajectory segment to obtain trajectory features; Encode the text description to obtain text features; A multi-modal fusion module, configured to: Perform semantic alignment on the text features, visual features, and trajectory features, and then perform multi-head attention feature fusion to obtain an attention allocation weight vector and the corresponding output features after multi-head attention; Use the attention allocation weight vector to calculate the semantic correlation score for the visual regions corresponding to the trajectory segments, obtain the comprehensive trajectory correlation score, sort the correlation scores in descending order, accumulate the correlation scores and set a threshold to obtain the key trajectory; Construct a language generation loss based on the output features corresponding to the multi-head attention, construct a focal loss based on the comprehensive trajectory correlation score and the key trajectory, and jointly optimize the video point tracking model using the language generation loss and the focal loss to obtain an optimized video point tracking model; Use the optimized video point tracking model to obtain multi-source features optimized by focusing; A video description generation module, configured to: Decode the multi-source features optimized by focusing to obtain the final video description result.

Images