Video description generation method and system based on self-service auxiliary task staggered fine tuning
By employing a self-help and assisted task interleaving fine-tuning method, and utilizing automated tools to construct a self-help and assisted task dataset, the pre-trained image visual language model is interleaved and fine-tuned. This solves the problems of temporal relationships, fine-grained target interactions, and static/dynamic information differentiation in video description generation, resulting in more accurate and richer video descriptions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-09
- Publication Date
- 2026-04-10
AI Technical Summary
Existing video description generation methods have shortcomings in terms of insufficient modeling of temporal relationships, inadequate depiction of fine-grained object interactions, unclear distinction between static and dynamic information, and reliance on additional video-text data or manual annotation, resulting in a lack of causal information and fine-grained interaction details in the generated descriptions.
By employing a self-help and assisted task interleaving fine-tuning method, an automated tool is used to construct a self-help and assisted task dataset, including temporal question answering, multi-frame multi-object tracking, and information decomposition dialogue tasks. The image visual language pre-trained model is interleaved and fine-tuned to generate a video description generation model.
Without increasing the cost of additional data or manual annotation, it significantly improves the temporal logic modeling capability, detail richness, and logicality of video descriptions, generating more accurate and temporally logical descriptive text.
Smart Images
Figure CN121842471A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a video description generation method and system based on self-service and assisted task interleaving and fine-tuning. Background Technology
[0002] With the explosive growth of internet video content, how to efficiently and accurately understand and describe massive amounts of video has become a core issue in applications such as intelligent retrieval, content moderation, human-computer interaction, and assisted accessible reading. Video description generation aims to automatically generate natural language text from input videos, providing a complete description of the main objects, actions, and scene states within the video; it is a typical visual-language multimodal understanding task. Existing video description generation methods can be broadly categorized into two types: one is an end-to-end approach based on video feature encoders and language decoders, typically training the model directly on video-text datasets; the other involves pre-training an image-visual-language pre-trained model on large-scale image-text data and then transferring it to the video description task. The former is often limited by the limited scale of video-text pairing data, high annotation costs, and insufficient generalization; while the latter, although possessing strong cross-modal alignment capabilities, suffers from limited temporal modeling capabilities due to the image-visual-language pre-trained model's native input being a single image and text. Its direct application to video description tasks usually requires the design of a separate temporal modeling module or reliance on expensive large-scale video-text pre-training.
[0003] On the other hand, video description tasks not only require a macroscopic understanding of the scene, but also fine-grained modeling of the sequential relationships between different time segments in the video, the interaction relationships between multiple targets, and the combination relationships between static scene information and dynamic action information. Existing methods still have shortcomings in the following aspects: 1) Insufficient modeling of temporal relationships: Many methods only model using simple time pooling or lightweight time attention modules, which makes it difficult to fully capture the temporal logic such as the order of events and the direction of playback.
[0004] 2) Insufficient fine-grained characterization of object interactions: Although some methods introduce object detection or region features, they are insufficient in continuously tracking objects and mining interaction relationships across multiple frames, resulting in a lack of detailed information in the generated description.
[0005] 3) Inability to distinguish between static and dynamic information: Traditional methods usually generate a complete description directly from the overall video features, lacking explicit decomposition and reorganization of static entities (such as people, objects, and scenes) and dynamic behaviors (such as actions and events).
[0006] 4) Reliance on additional video-text data or manual annotation: To compensate for the above shortcomings, some methods introduce additional video-text pre-training data or manually constructed task annotations, which significantly increases data costs and development costs.
[0007] For example, patent application CN118628963A discloses a video description method and system based on text content feature alignment. This method constructs a text-video correlation map through a pre-trained visual language model and performs temporal-dimensional pooling weighted fusion of video spatiotemporal features and action features to generate descriptive text. This method can only compress temporal information through temporal pooling operations, but cannot effectively preserve key temporal logic such as the order of frames, the cross-frame trajectory of the target, and the dynamic evolution of actions, resulting in a lack of causality and fine-grained interaction details in the video description.
[0008] Therefore, how to effectively improve the model's comprehensive understanding of video temporal relationships, fine-grained target interactions, and static / dynamic information without modifying the network structure of the image visual language pre-trained model, or increasing the cost of additional video-text pairs and manual annotation, and thus generate more accurate, richer, and temporally logical video description text, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0009] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a video description generation method and system based on self-service and assisted task interleaving and fine-tuning, which significantly reduces the cost of technology migration and deployment, achieves accurate modeling of video temporal logic, and effectively improves the detail richness, logic and completeness of the generated description.
[0010] The objective of this invention can be achieved through the following technical solutions: A video description generation method based on self-service assisted task interleaving and fine-tuning includes the following steps: The original video is sampled frame by frame to obtain multiple video images, which are then stitched together to form a grid image. Generate corresponding coordinate text descriptions for each frame of video image based on its grid position, and obtain the original video description dataset; A self-help task dataset is constructed based on the original video description dataset using an automated construction tool, which includes a preset parser, a part-of-speech tagger, and an image visual language pre-trained model. The image visual language pre-training model is fine-tuned by interleaving the original video description dataset and the self-help task dataset to obtain a video description generation model. The video description generation model has the same structure as the image visual language pre-training model. The original video is input into the video description generation model to obtain the video description of the original video.
[0011] Furthermore, the specific steps of performing frame sampling on the original video to obtain multiple video images and stitching them together into a grid image include: Multiple video frames are obtained by uniformly sampling along the time axis from the original video based on the number of rows and columns of the grid. The vertical scaling ratio and horizontal scaling ratio are calculated based on the original resolution of the video and the target image size, wherein the target image size is matched with the configuration of the visual language pre-trained language model; Each frame of video image is scaled according to the vertical and horizontal scaling ratios, and the scaled frames are filled into the corresponding grid cells in row priority order to obtain a grid image.
[0012] Furthermore, the specific steps for generating corresponding coordinate text descriptions based on the grid positions of each video frame include: Determine the row and column indices of each video frame in the grid; The normalized coordinates of the top left and bottom right corners of each video frame are calculated based on the row and column indices of each frame in the grid and the grid size. The normalized coordinates of the top left and bottom right corners of each video frame are converted into standardized coordinate text descriptions.
[0013] Furthermore, the self-assisted task dataset is obtained by formally defining the original video description dataset, including temporal question answering tasks, multi-frame multi-target tracking tasks, and information decomposition dialogue tasks.
[0014] Furthermore, the specific steps for the formal definition of the temporal question-answering task include: The video images in the grid image are divided into a first frame group and a second frame group. The first frame group corresponds to the frame sequence of the first half of the video, and the second frame group corresponds to the frame sequence of the second half of the video. The first frame group is randomly selected and arranged in the upper or lower half of the grid image by a preset parsing program, and the second frame group is arranged in the other half of the area to construct two video playback sequences, namely forward playback and reverse playback. Generate question text related to the video playback sequence, and automatically generate corresponding video playback answer labels based on the frame group arrangement, thereby forming a time-series question-and-answer task.
[0015] Furthermore, the specific steps for the formal definition of the multi-frame multi-target tracking task include: The image visual language pre-trained model is input with prompts containing detection instructions for each frame of video image, and the model outputs text. The target category and bounding box coordinates of each target within the frame are extracted from the output text of the model using a preset parsing program. Based on the row and column positions, scaling ratios, and bounding box coordinates of each video frame in the grid image, the single-frame coordinates are converted into global coordinates in the grid image coordinate system, and the target category and global coordinates are organized into target description text. The target description text of each video frame is concatenated in chronological order to form a multi-frame multi-target tracking task.
[0016] Furthermore, the specific steps for the formal definition of the information decomposition dialogue task include: Part-of-speech tagging was used to perform part-of-speech tagging on the original video description dataset, extracting nouns and adjectives to form a static information word set, and extracting verbs to form a dynamic information word set; The process involves three rounds of dialogue to generate three rounds of question-and-answer sessions. In the first round, the system requests a list of static information related to the video content, and the set of static information words is used as the output annotation for the first round. In the second round, the system requests a list of dynamic information related to the video content, and the set of dynamic information words is used as the output annotation for the second round. In the third round, the system requests the generation of a complete video description based on the static and dynamic information, and the original video description dataset is used as the output annotation for the third round. The three rounds of question-and-answer sessions are organized into a unified training sample format to form an information decomposition dialogue task.
[0017] Furthermore, when fine-tuning the image visual language pre-trained model, samples of temporal question-and-answer tasks, multi-frame multi-target tracking tasks, information decomposition dialogue tasks, and original video description data are mixed and sampled in each training batch according to a preset ratio, so that they appear alternately during training. The model parameters are optimized by maximizing the likelihood of the model generating standard answer text under given grid images and text prompts, based on a unified generative loss function.
[0018] Furthermore, the generative loss function is: In the formula, For generative loss function, For model parameters, This represents the total number of lexical units in the video description text. For conditional probability distribution, For the first part of the video description text Each word element, For the video description text in the first... All lexical units preceding the given lexical unit, For grid images, This is a text prompt.
[0019] According to another aspect of the present invention, a video description generation system based on self-service assisted task interleaving and fine-tuning is provided, comprising: The grid image stitching module is used to perform frame sampling on the original video to obtain multiple video images, and then stitch them together into a grid image. The original video description dataset acquisition module is used to generate the corresponding coordinate text descriptions for each frame of video image based on the grid position, thus obtaining the original video description dataset; A self-help assistance task dataset construction module is used to construct a self-help assistance task dataset based on the original video description dataset using an automated construction tool. The automated construction tool includes a preset parsing program, a part-of-speech tagger, and an image visual language pre-trained model. The interleaved fine-tuning training module performs interleaved fine-tuning on the image visual language pre-training model based on the original video description dataset and the self-help task dataset to obtain a video description generation model with video analysis and temporal awareness capabilities. The video description generation model has the same structure as the image visual language pre-training model. The video description generation module is used to input the original video into the video description generation model to obtain the video description of the original video.
[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs a self-help task dataset from the original video description dataset using fully automated tools, and then performs interleaved fine-tuning of the image visual language pre-trained model based on the original video description dataset and the self-help task dataset to obtain a video description generation model. This achieves synergistic improvement of the various capabilities of the image visual language pre-trained model without introducing additional data or manual annotation, significantly reducing the cost of technology migration and deployment, and overcoming the dependence of existing advanced methods on expensive annotation data.
[0021] 2. This invention introduces a temporal question-and-answer task, arranges video frame groups in an adaptive grid image format, and automatically generates forward and reverse temporal question-and-answer pairs. Without relying on external annotations, it forces the model to explicitly learn the sequence of video events and the playback direction, thereby achieving accurate modeling of video temporal logic and overcoming the shortcomings of traditional methods that are limited in accuracy due to insufficient modeling of temporal relationships.
[0022] 3. This invention designs a multi-frame, multi-target tracking task, uses a visual language model to automatically detect targets and transform coordinates to construct cross-frame target trajectories, and achieves fine-grained, automated characterization of the motion and interaction relationships of multiple objects in a video. It solves the shortcomings of existing methods in continuous target tracking and relationship mining, and significantly improves the detail richness of the generated description.
[0023] 4. This invention constructs an information decomposition dialogue task, automatically separates static entity information and dynamic behavior information based on part-of-speech tagging, and guides the model to decompose and then synthesize in a multi-turn dialogue format, thereby realizing a structured understanding and description generation of video content, effectively improving the logic and completeness of the generated text, and overcoming the problem of unclear description in traditional end-to-end methods. Attached Figure Description
[0024] Figure 1 This is a flowchart illustrating a video description generation method based on self-service assisted task interleaving and fine-tuning proposed in this invention. Figure 2 A flowchart illustrating the formal definition of the information decomposition dialogue task. Detailed Implementation
[0025] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0026] Example 1 This embodiment provides a video description generation method based on self-service assisted task interleaving and fine-tuning, such as... Figure 1 As shown, it includes the following steps: S1. Perform frame sampling on the original video to obtain multiple video images, and then stitch them together into a grid image.
[0027] The specific steps for performing frame sampling on the original video to obtain multiple video images, and then stitching them together into a grid image, include: Multiple video frames are obtained by uniformly sampling along the time axis from the original video based on the number of rows and columns of the grid. To adapt to the characteristic that the image visual language pre-trained model only accepts a single image input, uniform or approximately uniform frame sampling is performed along the time axis for each video segment, and a fixed number of frame images are selected from them.
[0028] The vertical scaling ratio and horizontal scaling ratio are calculated based on the original resolution of the video and the target image size, and the target image size is matched with the configuration of the visual language pre-trained language model.
[0029] Assume the original video size is ,in, For frame number, and These are the frame height and frame width, respectively. Let the size of the input image for the model be... The grid size is It is necessary to select from the video. Frames form a grid image .
[0030] Next, frame sampling and scaling ratio calculations are performed, from... A frame set is obtained by sampling the frame at equal time intervals. In order to make The frame image can fill the entire input image, and the vertical and horizontal scaling ratios are defined as follows: Each frame of video image is scaled according to the vertical and horizontal scaling ratios, and then the scaled frames are filled into the corresponding grid cells in row-major order to obtain a grid image. This allows multiple video frames to appear in a regular grid layout within the same input image.
[0031] S2. Generate the corresponding coordinate text descriptions for each frame of video image based on its grid position, and obtain the original video description dataset.
[0032] The specific steps for generating the corresponding coordinate text descriptions based on the grid positions of each video frame include: Determine the row and column indices of each video frame within the grid. The row and column indices are calculated as follows: In the formula, For row index, For column indexes, The linear sequence number of the frame. For grid size, This is the modulo operator.
[0033] The normalized coordinates of the top-left and bottom-right corners of each video frame are calculated based on the row and column indices of each frame in the grid and the grid size.
[0034] In the normalized coordinate system, the normalized coordinates of the top-left corner of the video image are: The normalized coordinates of the bottom right corner of the video image are: In the formula, For the first Line number The top left corner of the video image Axis coordinates For the first Line number The top left corner of the video image Axis coordinates For the floor function, For row index, For column indexes, For grid size, For the first Line number The bottom right corner of the video image Axis coordinates For the first Line number The bottom right corner of the video image Axis coordinates.
[0035] The normalized coordinates of the top-left and bottom-right corners of each video frame are converted into standardized coordinate text descriptions. A generic coordinate description text is constructed by concatenating the coordinates of each video frame using a predefined template. This, along with the instruction text for subsequent tasks, serves as part of the model input. Ultimately, the visual input and general text corresponding to a video sample can be denoted as: .
[0036] S3. Construct a self-help task dataset based on the original video description dataset using automated construction tools. The automated construction tools include a preset parser, a part-of-speech tagger, and an image visual language pre-trained model.
[0037] This embodiment uses MiniGPT4-v2 as the image visual language pre-training model, denoted as [model name missing]. Its input is an image. With text prompts The output is a text sequence. Among them, text prompts From model instruction text General coordinate description Task-related text composition: .
[0038] The self-assisted task dataset is obtained by formally defining the original video description dataset, including temporal question answering tasks, multi-frame multi-object tracking tasks, and information decomposition dialogue tasks.
[0039] The specific steps for formally defining a time-series question-answering task include: The video images in the grid image are divided into a first frame group and a second frame group. The first frame group corresponds to the frame sequence of the first half of the video, and the second frame group corresponds to the frame sequence of the second half of the video. The first frame group is randomly selected and arranged in the upper or lower half of the grid image by a preset parsing program, and the second frame group is arranged in the other half of the area to construct two video playback sequences, namely forward playback or reverse playback. Generate question text related to the video playback sequence, and automatically generate corresponding video playback answer labels based on the frame group arrangement, thus forming a time-series question answering task.
[0040] For time-series question answering tasks, the frame set is divided into the following categories according to time order: and Let random variable Control the grid layout, when At that time, Place in the upper area Placed in the lower half of the area, labeled "playing forward"; when At that time, Place in the upper area Placed in the lower half of the area, with the label "Reverse Playback". The label format is: By constructing multi-turn dialogue samples, the model is first guided to list static and dynamic information respectively, and then guided to synthesize a complete description based on this, thereby enabling the model to learn the ability to decompose and combine information.
[0041] The specific steps for formally defining a multi-frame, multi-object tracking task include: The image visual language pre-trained model (MiniGPT4-v2) takes a prompt containing detection instructions as input to each frame of video image and obtains the model output text. The target category and bounding box coordinates of each target in the frame are extracted from the model output text using a preset parsing program. Based on the row and column positions, scaling ratios, and bounding box coordinates of each video frame in the grid image, the single-frame coordinates are converted into global coordinates in the grid image coordinate system, and the target category and global coordinates are organized into target description text. The target description text of each video frame is concatenated in chronological order to form a multi-frame multi-target tracking task.
[0042] For multi-frame, multi-object tracking tasks, for a single frame Input detection command prompt At this point, the model output is 。 Extract the target set from it using the parsing program: ,in m This indicates the number of targets detected. This indicates the coordinates of the top-left corner of the detection box. This represents the coordinates of the bottom right corner of the detection box. Assume the frame... The position in the grid is The normalized coordinates of the top left corner of the video image are Then the global coordinates of the target in the grid image are: In the formula, and The coordinates of the top left corner of the target are... and The coordinates of the bottom right corner of the target.
[0043] The specific steps for formally defining the information decomposition dialogue task are as follows: Figure 2 As shown, it includes: Part-of-speech tagging was used to perform part-of-speech tagging on the original video description dataset, extracting nouns and adjectives to form a static information word set, and extracting verbs to form a dynamic information word set; The process involves three rounds of dialogue to generate three rounds of question-and-answer sessions. In the first round, the user is asked to list static information related to the video content, and the set of static information words is used as the output annotation for the first round. In the second round, the user is asked to list dynamic information related to the video content, and the set of dynamic information words is used as the output annotation for the second round. In the third round, the user is asked to generate a complete video description based on the static and dynamic information, and the original video description dataset is used as the output annotation for the third round. The three rounds of question-and-answer sessions are organized into a unified training sample format to form an information decomposition dialogue task.
[0044] For the information decomposition dialogue task, let the original video description text be... Use part-of-speech taggers to extract static information words (input parameters) and Sets and dynamic information words (input parameters) )gather: The true labels for the three rounds of dialogue are, in order: In the formula, It is a set of static information words. It is a dynamic information word set. This is a part-of-speech tagging function. The true label for the first round of dialogue. The true label for the second round of dialogue. This is the true label for the third round of dialogue.
[0045] The core of this stage, automating the construction of the self-help assisted task dataset, involves automatically executing the aforementioned process on the entire original dataset. For each video sample, multiple grid images of different grid sizes, general coordinate description text, corresponding temporal question-answering task samples, multi-frame multi-object tracking task samples, and information decomposition dialogue task samples are generated sequentially. Ultimately, a training / validation / test dataset jointly containing multiple tasks is constructed. All data is generated by automated programs and tools, requiring no manual annotation or additional video-text pairs, demonstrating high automation and scalability.
[0046] S4. Based on the original video description dataset and the self-help task dataset, the image visual language pre-trained model is fine-tuned by alternation to obtain the video description generation model. The video description generation model has the same structure as the image visual language pre-trained model.
[0047] This step builds upon the pre-trained image-visual language model. Without altering its visual encoder structure, it jointly fine-tunes the model by interleaving input samples from self-help and video description tasks. For the same video, at least two different grid sizes are constructed, and samples corresponding to these different grid sizes are used together for training to improve the robustness and generalization performance of the video description generation model across different spatiotemporal resolutions.
[0048] When fine-tuning the image-visual language pre-trained model, samples from temporal question-answering tasks, multi-frame multi-object tracking tasks, information decomposition dialogue tasks, and original video description data are mixed and sampled in a preset proportion in each training batch. These samples are then interleaved during training. The model parameters are optimized by maximizing the likelihood of generating standard answer text given a grid image and text prompts, based on a unified generative loss function. In this embodiment, MiniGPT4-v2 is used as the pre-trained model. Its visual encoder is frozen, and only a low-rank adaptation module is introduced into the language model, with a small number of projection layer parameters enabled for training.
[0049] The generative loss function is: In the formula, For generative loss function, For model parameters, This represents the total number of lexical units in the video description text. For conditional probability distribution, For the first part of the video description text Each word element, For the video description text in the first... All lexical units preceding the given lexical unit, For grid images, This is a text prompt.
[0050] S5. Input the original video into the video description generation model to obtain the video description of the original video.
[0051] This example demonstrates performance evaluation through comparative and ablation experiments. The method was tested on standard public datasets MSR-VTT, MSVD, and VATEX, and compared with several existing video description methods. The MSVD dataset contains approximately 2000 short videos, each with about 40 human descriptions; the MSR-VTT dataset contains approximately 10,000 videos across multiple categories, each with 20 descriptions; the VATEX dataset is even larger, containing over 40,000 videos and providing bilingual (Chinese and English) descriptions. Evaluation metrics include commonly used video description metrics such as BLEU@4, METEOR, ROUGE-L, and CIDEr.
[0052] Comparative experiments were conducted with similar methods, and test results on the MSR-VTT, MSVD, and VATEX datasets were statistically analyzed. Four metrics—BLEU@4, METEOR, ROUGE-L, and CIDEr—were used for comprehensive evaluation. The video description performance of our method on the MSR-VTT dataset without introducing additional video-text data is shown in the table below.
[0053] Table 1. Video description performance on the MSR-VTT dataset. The performance of the proposed method on the MSVD dataset without introducing additional video-text data is shown in the table below.
[0054] Table 2. Video Description Performance on the MSVD Dataset The performance of the proposed method on the VATEX dataset without introducing additional video-text data is shown in the table below.
[0055] Table 3. Video Description Performance on the VATEX Dataset As shown in Tables 1, 2, and 3, the method in this application achieves CIDEr scores of 69.2, 160.7, and 72.2 on the three major datasets without introducing additional video-text data, all of which outperform other methods that do not use additional video-text data. It also demonstrates a significant advantage in the METEOR metric, with scores of 65.8, 84.7, and 55.8 on the three datasets. The other two metrics, ROUGE-L and CIDEr, also show competitive results. Overall, this demonstrates the effectiveness of the self-help-assisted task-based interleaved fine-tuning of the image visual language pre-trained model.
[0056] To further analyze the contributions of each component module in the proposed method, two types of ablation experiments were designed: First, under fixed data and training configurations, different self-help assistance tasks (temporal question answering, multi-frame multi-object tracking, and information decomposition dialogue tasks) were removed or combined to form various task combination configurations, which were then independently trained and evaluated on the MSVD, MSR-VTT, and VATEX datasets. Second, for different grid size configurations in video grid construction, comparisons were made between using only 1×1 (single frame), only 4×4, only 16×16, a combination of 4×4 and 16×16, and even using multiple grid sizes simultaneously.
[0057] The ablation experiment results for different tasks on the MSR-VTT dataset are shown in the table below.
[0058] Table 4 Ablation experiment results for different tasks on the MSR-VTT dataset. TOQA, MFMOT, and IDC represent temporal question answering, multi-frame multi-target tracking, and information decomposition dialogue tasks, respectively.
[0059] The ablation experiment results for different tasks on the MSVD dataset are shown in the table below.
[0060] Table 5 Ablation experiment results for different tasks on the MSVD dataset. The ablation experiment results for different tasks on the VATEX dataset are shown in the table below.
[0061] Table 6. Ablation experiment results for different tasks on the VATEX dataset. As shown in Tables 4, 5, and 6, any single auxiliary task can bring a certain improvement compared to the pure video description task. However, the complete configuration that introduces all three tasks at the same time has the best effect. The combination of information decomposition dialogue task often has a particularly significant improvement on CIDEr and METEOR indicators.
[0062] The ablation test results for different grid number combinations on the MSR-VTT dataset are shown in the table below.
[0063] Table 7 Ablation test results for different grid number combinations on the MSR-VTT dataset The ablation experiment results for different grid number combinations on the MSVD dataset are shown in the table below.
[0064] Table 8 Ablation test results for different grid number combinations on the MSVD dataset. The ablation test results for different grid number combinations on the VATEX dataset are shown in the table below.
[0065] Table 9 Ablation test results for different grid number combinations on the VATEX dataset. According to the experimental results in Tables 7, 8 and 9, although a single grid size can achieve certain performance, its overall performance is not as good as the combined use of multiple grid sizes. In particular, the combination of 4×4 and 16×16 generally has better overall performance on the three major datasets. This indicates that the joint modeling of medium and high temporal granularity is beneficial for capturing short-term actions and long-term events in videos, thereby improving the accuracy and diversity of generated descriptions.
[0066] Example 2 This embodiment provides a video description generation system based on self-service assisted task interleaving and fine-tuning, including: The grid image stitching module is used to perform frame sampling on the original video to obtain multiple video images, and then stitch them together into a grid image. The original video description dataset acquisition module is used to generate the corresponding coordinate text descriptions for each frame of video image based on the grid position, thus obtaining the original video description dataset; The self-help assistance task dataset construction module is used to construct a self-help assistance task dataset based on the original video description dataset using an automated construction tool. The automated construction tool includes a preset parser, a part-of-speech tagger, and an image visual language pre-trained model. The interleaved fine-tuning training module performs interleaved fine-tuning on the image visual language pre-trained model based on the original video description dataset and the self-help task dataset, resulting in a video description generation model with video analysis and temporal awareness capabilities. The video description generation model has the same structure as the image visual language pre-trained model. The video description generation module is used to input the original video into the video description generation model to obtain the video description of the original video.
[0067] The rest is the same as in Example 1.
[0068] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A video description generation method based on self-service and assisted task interleaving and fine-tuning, characterized in that, Includes the following steps: The original video is sampled frame by frame to obtain multiple video images, which are then stitched together to form a grid image. Generate corresponding coordinate text descriptions for each frame of video image based on its grid position, and obtain the original video description dataset; A self-help task dataset is constructed based on the original video description dataset using an automated construction tool, which includes a preset parser, a part-of-speech tagger, and an image visual language pre-trained model. The image visual language pre-training model is fine-tuned by interleaving the original video description dataset and the self-help task dataset to obtain a video description generation model. The video description generation model has the same structure as the image visual language pre-training model. The original video is input into the video description generation model to obtain the video description of the original video.
2. The video description generation method based on self-service assisted task interleaving and fine-tuning according to claim 1, characterized in that, The specific steps for performing frame sampling on the original video to obtain multiple video images, and then stitching them together into a grid image, include: Multiple video frames are obtained by uniformly sampling along the time axis from the original video based on the number of rows and columns of the grid. The vertical scaling ratio and horizontal scaling ratio are calculated based on the original resolution of the video and the target image size, wherein the target image size is matched with the configuration of the visual language pre-trained language model; Each frame of video image is scaled according to the vertical and horizontal scaling ratios, and the scaled frames are filled into the corresponding grid cells in row priority order to obtain a grid image.
3. The video description generation method based on self-service assisted task interleaving and fine-tuning according to claim 1, characterized in that, The specific steps for generating the corresponding coordinate text descriptions based on the grid positions of each video frame include: Determine the row and column indices of each video frame in the grid; The normalized coordinates of the top left and bottom right corners of each video frame are calculated based on the row and column indices of each frame in the grid and the grid size. The normalized coordinates of the top left and bottom right corners of each video frame are converted into standardized coordinate text descriptions.
4. The video description generation method based on self-service assisted task interleaving and fine-tuning according to claim 1, characterized in that, The self-assisted task dataset is obtained by formally defining the original video description dataset, and includes temporal question answering tasks, multi-frame multi-object tracking tasks, and information decomposition dialogue tasks.
5. The video description generation method based on self-service assisted task interleaving and fine-tuning according to claim 4, characterized in that, The specific steps for formally defining the temporal question-answering task include: The video images in the grid image are divided into a first frame group and a second frame group. The first frame group corresponds to the frame sequence of the first half of the video, and the second frame group corresponds to the frame sequence of the second half of the video. The first frame group is randomly selected and arranged in the upper or lower half of the grid image by a preset parsing program, and the second frame group is arranged in the other half of the area to construct two video playback sequences, namely forward playback and reverse playback. Generate question text related to the video playback sequence, and automatically generate corresponding video playback answer labels based on the frame group arrangement, thereby forming a time-series question-and-answer task.
6. The video description generation method based on self-service assisted task interleaving and fine-tuning according to claim 4, characterized in that, The specific steps for the formal definition of the multi-frame multi-target tracking task include: The image visual language pre-trained model is input with prompts containing detection instructions for each frame of video image, and the model outputs text. The target category and bounding box coordinates of each target within the frame are extracted from the output text of the model using a preset parsing program. Based on the row and column positions, scaling ratios, and bounding box coordinates of each video frame in the grid image, the single-frame coordinates are converted into global coordinates in the grid image coordinate system, and the target category and global coordinates are organized into target description text. The target description text of each video frame is concatenated in chronological order to form a multi-frame multi-target tracking task.
7. The video description generation method based on self-service assisted task interleaving and fine-tuning according to claim 4, characterized in that, The specific steps for formally defining the information decomposition dialogue task include: Part-of-speech tagging was used to perform part-of-speech tagging on the original video description dataset, extracting nouns and adjectives to form a static information word set, and extracting verbs to form a dynamic information word set; The process involves three rounds of dialogue to generate three rounds of question-and-answer sessions. In the first round, the system requests a list of static information related to the video content, and the set of static information words is used as the output annotation for the first round. In the second round, the system requests a list of dynamic information related to the video content, and the set of dynamic information words is used as the output annotation for the second round. In the third round, the system requests the generation of a complete video description based on the static and dynamic information, and the original video description dataset is used as the output annotation for the third round. The three rounds of question-and-answer sessions are organized into a unified training sample format to form an information decomposition dialogue task.
8. The video description generation method based on self-service assisted task interleaving and fine-tuning according to claim 4, characterized in that, When fine-tuning the image visual language pre-trained model, samples of temporal question-and-answer tasks, multi-frame multi-object tracking tasks, information decomposition dialogue tasks, and original video description data are mixed and sampled in each training batch according to a preset ratio, so that they appear alternately during training. The model parameters are optimized by maximizing the likelihood of the model generating standard answer text under given grid images and text prompts, based on a unified generative loss function.
9. The video description generation method based on self-service assisted task interleaving and fine-tuning according to claim 8, characterized in that, The generative loss function is: In the formula, For generative loss function, For model parameters, This represents the total number of lexical units in the video description text. For conditional probability distribution, For the first part of the video description text Each word element, For the video description text in the first... All lexical units preceding the given lexical unit, For grid images, This is a text prompt.
10. A video description generation system based on self-service assisted task interleaving and fine-tuning, characterized in that, include: The grid image stitching module is used to perform frame sampling on the original video to obtain multiple video images, and then stitch them together into a grid image. The original video description dataset acquisition module is used to generate the corresponding coordinate text descriptions for each frame of video image based on the grid position, thus obtaining the original video description dataset; A self-help assistance task dataset construction module is used to construct a self-help assistance task dataset based on the original video description dataset using an automated construction tool. The automated construction tool includes a preset parsing program, a part-of-speech tagger, and an image visual language pre-trained model. The interleaved fine-tuning training module performs interleaved fine-tuning on the image visual language pre-training model based on the original video description dataset and the self-help task dataset to obtain a video description generation model with video analysis and temporal awareness capabilities. The video description generation model has the same structure as the image visual language pre-training model. The video description generation module is used to input the original video into the video description generation model to obtain the video description of the original video.