Video description large model training method, device and electronic equipment

Abstract

The application relates to a video description large model training method and device and electronic equipment. The method comprises: performing spatiotemporal feature analysis on a sample video to calculate the video complexity thereof and determining the optimal length of the text description corresponding to the sample video; performing entity detection and action recognition on the sample video to construct a video element ground truth library of the sample video; inputting the sample video into a video description large model to generate at least one candidate video description text; based on the optimal length of the text description and the video element ground truth library, calculating the video description score, the adaptive length score and the fine-grained alignment score of the candidate video description text, and obtaining the comprehensive score of the candidate video description text; and taking the comprehensive score as a reinforcement learning reward signal and iteratively updating the parameters of the video description large model by using a reinforcement learning strategy optimization algorithm. The application solves the technical problems of low information density, redundancy and lack of complexity adaptation of an existing video description model.

Description

Technical Field

[0001] This application relates to the field of multimedia intelligent processing technology, and in particular to a training method, apparatus, and electronic device for a large video description model. Background Technology

[0002] With the rapid development of AI-generated content (AIGC) technology, video understanding and description has become an important research direction at the intersection of computer vision and natural language processing. Its core goal is to enable computers to understand video content and generate accurate and fluent descriptions in natural language. Currently, most mainstream video description models adopt an encoder-decoder architecture. The encoder is responsible for extracting visual features from the video, while the decoder uses a large language model (LLM) to generate text. Training these models typically includes multimodal alignment pre-training, supervised fine-tuning (SFT), and reinforcement learning optimization in some cases to align visual and linguistic features and improve the quality of generated text. Although existing methods can generate grammatically correct sentences, they generally suffer from low information density and verbose repetition in practical applications, often resulting in "nonsense literature" or "safe mode." The main reasons for this include: maximum likelihood estimation training tends to select common high-frequency words to reduce prediction errors; traditional evaluation metrics such as BLEU and CIDEr overemphasize word overlap while neglecting accuracy and conciseness; and the lack of adaptive processing for video complexity leads to insufficient detail in complex video descriptions and verbose descriptions in simple video descriptions. Summary of the Invention

[0003] This application provides a training method, apparatus, and electronic device for a large video description model to address the technical problems of existing video description models, such as low information density, verbose and repetitive content, and lack of complexity adaptation.

[0004] Firstly, this application provides a training method for a large video description model, comprising: performing spatiotemporal feature analysis on sample videos to calculate their video complexity, and determining the optimal length of the text description corresponding to the sample videos based on the video complexity; performing entity detection and action recognition on the sample videos to extract the entity set and action set present in the sample videos, and constructing a video element truth database for the sample videos; inputting the sample videos into the large video description model to generate at least one candidate video description text based on the video content of the sample videos; calculating the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text based on the optimal length of the text description and the video element truth database, and performing a weighted summation of the video description score, the adaptive length score, and the fine-grained alignment score to obtain a comprehensive score for the candidate video description text; using the comprehensive score as a reinforcement learning reward signal, and employing a reinforcement learning strategy optimization algorithm to iteratively update the parameters of the large video description model to obtain an iteratively updated large video description model.

[0005] Secondly, this application provides a training apparatus for a large video description model, comprising: a first calculation module, used to perform spatiotemporal feature analysis on sample videos to calculate their video complexity, and determine the optimal length of the text description corresponding to the sample videos based on the video complexity; a recognition module, used to perform entity detection and action recognition on the sample videos to extract the entity set and action set present in the sample videos, and construct a video element truth database of the sample videos; a generation module, used to input the sample videos into the large video description model, so that the large video description model generates at least one candidate video description text based on the video content of the sample videos; a second calculation module, used to calculate the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text based on the optimal length of the text description and the video element truth database, and to perform a weighted summation of the video description score, the adaptive length score, and the fine-grained alignment score to obtain a comprehensive score for the candidate video description text; and an update module, used to use the comprehensive score as a reinforcement learning reward signal, and to iteratively update the parameters of the large video description model using a reinforcement learning strategy optimization algorithm to obtain an iteratively updated large video description model.

[0006] As an optional example, the first calculation module includes: a first calculation unit, used to extract the feature vector of each frame of the sample video and calculate the average cosine distance between the feature vectors of adjacent frames to calculate the temporal variability of the sample video; a second calculation unit, used to perform clustering processing on all frame feature vectors of the sample video to select a set of key frames, and calculate the average spatial information entropy of each key frame based on the set of key frames to calculate the spatial information entropy of the sample video; a third calculation unit, used to perform weighted fusion of the temporal variability and the spatial information entropy according to a preset weight coefficient to calculate the video complexity of the sample video; and a determination unit, used to determine the optimal length of the text description of the sample video based on the video complexity, wherein the optimal length of the text description is positively correlated with the video complexity.

[0007] As an optional example, the above-mentioned recognition module includes: a recognition unit, used to input the above-mentioned sample video into a preset detection model, so that the preset detection model performs entity detection and action recognition on the above-mentioned sample video, identifies all entity objects appearing in the above-mentioned sample video and all action behaviors occurring, and extracts the corresponding entity category information and action category information to generate an entity set and an action set; and a fusion unit, used to summarize and fuse the above-mentioned entity set and the above-mentioned action set to construct a video element truth library of the above-mentioned sample video.

[0008] As an optional example, the second calculation module includes: an acquisition unit for acquiring the manually annotated ground truth description text corresponding to the sample video; and an evaluation unit for inputting the candidate video description text and the ground truth description text into a preset video description scoring model, so that the preset video description scoring model compares and evaluates the candidate video description text from the perspectives of scene consistency, event matching, and reasonableness of character behavior, and outputs the video description score of the candidate video description text based on the comparison and evaluation results.

[0009] As an optional example, the second calculation module includes: a fourth calculation unit for calculating the actual length of the candidate video description text; and a fifth calculation unit for calculating an adaptive length score of the candidate video description text using a Gaussian distribution-formed length reward function based on the difference between the optimal length of the text description and the actual length.

[0010] As an optional example, the second calculation module includes: an extraction unit for extracting a set of nouns and a set of verbs from the candidate video description text; a sixth calculation unit for comparing the set of nouns and the set of verbs with the set of entities and the set of actions in the video element truth database to calculate coverage and precision; and a seventh calculation unit for harmonic averaging the coverage and precision to calculate a fine-grained alignment score for the candidate video description text.

[0011] As an optional example, the above-mentioned update module includes: an eighth calculation unit, used to input the above-mentioned comprehensive score as a reward signal into the above-mentioned reinforcement learning policy optimization algorithm, so that the above-mentioned reinforcement learning policy optimization algorithm calculates the policy gradient loss based on the above-mentioned reward signal; and an update unit, used to iteratively update the parameters of the above-mentioned video description large model using the above-mentioned policy gradient loss.

[0012] Thirdly, this application provides a storage medium storing a computer program, wherein the computer program is executed by a processor to perform the training method of the above-described large model of video description.

[0013] Fourthly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described training method for a large video description model through the computer program.

[0014] The technical solutions provided in this application have the following advantages compared with the prior art: This application employs spatiotemporal feature analysis of sample videos to calculate their video complexity, and determines the optimal length of the corresponding text description based on the video complexity; it performs entity detection and action recognition on the sample videos to extract the entity and action sets present in the sample videos, and constructs a video element truth database for the sample videos; it inputs the sample videos into a large-scale video description model, enabling the large-scale video description model to generate at least one candidate video description text based on the video content of the sample videos; based on the optimal length of the text description and the video element truth database, it calculates the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text, and performs a weighted sum of the video description score, adaptive length score, and fine-grained alignment score to obtain a comprehensive score for the candidate video description text; it uses the comprehensive score as a reinforcement learning reward signal, and employs reinforcement... The learning strategy optimization algorithm iteratively updates the parameters of the aforementioned video description model to obtain an iteratively updated video description model. In this method, spatiotemporal feature analysis is performed on sample videos to calculate video complexity and determine the optimal length of the text description. Simultaneously, entity detection and action recognition are performed on the videos to construct a video element truth database. Sample videos are input into the large model to generate candidate text. Then, based on the text length and the video element truth database, a video description score, an adaptive length score, and a fine-grained alignment score are calculated. These scores are weighted and summed to obtain a comprehensive score, which serves as a reinforcement learning reward signal. The model parameters are iteratively optimized through a policy gradient algorithm. This significantly improves the information density, accuracy of key details, and conciseness of the generated text, enhancing the accuracy and practicality of video understanding and automatic description. This solves the technical problems of existing video description models, such as low information density, verbose and repetitive content, and a lack of adaptive complexity. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 This is a flowchart of an optional training method for a large video description model according to an embodiment of this application; Figure 2 This is a system architecture diagram of an optional training method for a large video description model according to an embodiment of this application; Figure 3 This is a flowchart illustrating the specific implementation of an optional training method for a large video description model according to an embodiment of this application. Figure 4 This is a schematic diagram of the structure of an optional training device for a large video description model according to an embodiment of this application; Figure 5 This is a schematic diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The following disclosure provides many different embodiments or examples for implementing different structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0021] According to a first aspect of the embodiments of this application, a method for training a large video description model is provided, optionally, as follows: Figure 1 As shown, the above method includes: S102, Perform spatiotemporal feature analysis on the sample video to calculate its video complexity, and determine the optimal length of the text description corresponding to the sample video based on the video complexity. S104, Perform entity detection and action recognition on the sample video to extract the entity set and action set present in the sample video, and construct a video element truth library of the sample video; S106, Input the sample video into the video description big model so that the video description big model generates at least one candidate video description text based on the video content of the sample video; S108. Based on the optimal length of the text description and the truth database of video elements, calculate the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text, and perform a weighted summation of the video description score, adaptive length score, and fine-grained alignment score to obtain the comprehensive score of the candidate video description text. S110 uses the comprehensive score as a reward signal for reinforcement learning and employs a reinforcement learning strategy optimization algorithm to iteratively update the parameters of the large video description model to obtain the iteratively updated large video description model.

[0022] Optionally, in this embodiment, a video description large model training method based on video complexity adaptation and fine-grained semantic verification is proposed. Its core objective is to improve the accuracy, information density, and conciseness of the generated text.

[0023] like Figure 2 The system architecture diagram shown illustrates a closed-loop reinforcement learning system, primarily comprising three core modules: a video understanding and generation module (Agent / Policy), which acts as the optimized entity, receiving video input and generating natural language descriptions; this module is essentially a large-scale video description model. A multi-dimensional reward model module (Reward Model), serving as the core evaluation component, scores the generated text across three sub-units: video complexity assessment, fine-grained semantic verification, and video description scoring. Specifically, the video complexity assessment unit analyzes the video's spatiotemporal features and calculates the dynamically optimal target length to generate a length reward; the fine-grained semantic verification unit compares the video detection results with the text content to calculate the accuracy rewards for entities and actions; and the video description scoring unit uses a pre-trained model to score the text from the global scene and event dimensions. Finally, the policy optimization module (Optimizer) receives the comprehensive reward signal and iteratively updates the parameters of the video understanding and generation module using reinforcement learning algorithms (such as policy gradients), gradually optimizing the accuracy, information density, and conciseness of the text generated by the large-scale video description model.

[0024] like Figure 3 The flowchart shown first performs spatiotemporal feature analysis on the sample video v to calculate video complexity. Specifically, by extracting visual feature vectors from video frames, the temporal variability between adjacent frames is calculated to measure the degree of dynamic change in the video; simultaneously, spatial information entropy is calculated on keyframes to quantify the richness of detail in the video frame. The video complexity index is obtained by weighted fusion of the temporal variability and spatial information entropy. Based on this, the optimal length of the text description corresponding to the sample video (dynamic target length) is determined. This allows the generated text length to match the amount of video information, thereby achieving the adaptive goal of generating longer descriptions for complex videos and shorter descriptions for simple videos.

[0025] Simultaneously, entity detection and action recognition are performed on the sample video v to extract the set of entities present in the video. With action set A video element truth database is constructed. This database records key entities and actions that actually exist in the video, providing a basis for fine-grained semantic verification of the subsequently generated text. In the candidate text generation stage, the sample video is input into the video description model, which generates at least one candidate video description text T. Subsequently, multi-dimensional scoring is performed on the generated candidate text T, including a video description score. Adaptive length scoring and fine-grained alignment scoring Among them, video description rating. Using an offline pre-trained video description scoring model, candidate text T is compared with manually labeled ground truth values ​​based on dimensions such as scene consistency, event matching, and the reasonableness of character behavior, and a quality score is output; adaptive length scoring is also employed. Based on the difference between candidate text length and optimal target length, a Gaussian function is used to quantify the length matching degree, ensuring that the text length adapts to the video complexity; fine-grained alignment scoring. The accuracy and coverage are calculated by comparing the entities and actions mentioned in candidate text T with the truth database of video elements, and then comprehensively evaluated using an F1 score to suppress hallucinatory content and ensure the complete representation of key details. Finally, the three scores are weighted and summed to obtain the comprehensive score of the candidate text. .

[0026] During the model optimization phase, the comprehensive score of the candidate texts will be used. As a reward signal for reinforcement learning, it is input into the reinforcement learning policy optimization module. By calculating the policy gradient loss, the direction of policy parameter adjustment is quantified, and the parameters of the large video description model are iteratively updated to gradually increase the probability of generating text with high comprehensive scores. Algorithms such as Proximal Policy Optimization (PPO) and Generalized Reward Policy Optimization (GRPO) can be used to iteratively update the model parameters, thereby continuously improving the performance of the generated text in terms of accuracy, information density, and conciseness.

[0027] The above method adaptively adjusts the text length according to the video complexity, while performing semantic verification at the fine-grained entity and action levels, thereby generating accurate, information-rich, and concise natural language descriptions. Compared with existing technologies, this invention significantly alleviates the problems of low information density and verbose repetition, ensures the complete expression of key details, and improves the readability and application value of the generated text. It is applicable to various practical scenarios such as video understanding, content retrieval, and automatic summarization, and has broad industrial application prospects.

[0028] As an optional example, performing spatiotemporal feature analysis on sample videos to calculate their video complexity, and determining the optimal length of the corresponding text description based on the video complexity, includes: Extract the feature vector of each frame of the sample video and calculate the average cosine distance between the feature vectors of adjacent frames to obtain the temporal variability of the sample video. Clustering is performed on all frame feature vectors of the sample video to select a set of key frames, and the mean spatial information entropy of each key frame is calculated based on the set of key frames to obtain the spatial information entropy of the sample video. The temporal variability and spatial information entropy are weighted and fused according to preset weighting coefficients to calculate the video complexity of the sample video. The optimal length of the text description for a sample video is determined based on the video complexity, where the optimal length of the text description is positively correlated with the video complexity.

[0029] Optionally, in this embodiment, spatiotemporal feature analysis is performed on the sample video to calculate the video complexity, and the optimal length of the corresponding text description is determined based on the video complexity. Specifically, a pre-trained video encoder (such as CLIP-ViT) is first used to extract the feature vector of each frame of the sample video. The temporal variability of the video is quantified by calculating the average cosine distance between the feature vectors of adjacent frames. As shown in formula (1): (1) Temporal variability reflects the degree of dynamic change in video footage over time; a higher value indicates more drastic changes in video action or scene, requiring a larger vocabulary for the generated text. Subsequently, the feature vectors of all video frames are analyzed. Clustering (such as the K-Means algorithm) is performed to divide the frame features into... Each cluster is divided into clusters, and the frame closest to the cluster center in each cluster is selected as the keyframe set. This ensures that keyframes cover different scenes, actions, and perspectives in the video.

[0030] For each frame in the keyframe set The spatial entropy is calculated to quantify the richness of image detail. Specifically, the keyframe image is converted to grayscale, and the probability of each grayscale level j appearing in the grayscale histogram is calculated. And calculate the single-frame entropy. As shown in formula (2): (2) The spatial information entropy of the entire video is the average of the entropies of all keyframes. As shown in formula (3): (3) Next, according to the preset weighting coefficients and The temporal variability and spatial information entropy are weighted and fused to calculate the overall video complexity of the sample videos. As shown in formula (4): (4) Finally, the optimal length of the text description is dynamically set based on the video complexity. As shown in formula (5): (5) In formula (5), Based on length, This is a scaling factor that makes the text length positively correlated with the video complexity.

[0031] By following the steps above, the text length can be adaptively adjusted to the complexity of the video, thereby ensuring that the generated text covers key actions and entities in the video while avoiding redundant information, thus improving the accuracy, information density, and conciseness of the text generated by the large video description model.

[0032] As an optional example, entity detection and action recognition are performed on sample videos to extract the sets of entities and actions present in the sample videos, and a truth database of video elements for the sample videos is constructed, including: The sample video is input into the preset detection model so that the preset detection model can perform entity detection and action recognition on the sample video, identify all entity objects and all actions that occur in the sample video, and extract the corresponding entity category information and action category information to generate entity set and action set. The entity set and action set are aggregated and merged to construct a truth database of video elements from the sample videos.

[0033] Optionally, in this embodiment, entity detection and action recognition are performed on the sample video to extract the set of entities present in the video. With action set This process involves constructing a ground truth database for video elements. Specifically, sample videos are first input into a pre-defined detection model, which can be a powerful existing object detection and action recognition model, such as Faster R-CNN, I3D, or VideoMAE. The detection model analyzes video frames or segments to automatically identify all entities and their categories in the video, including people, animals, vehicles, and everyday objects. It also identifies actions and their categories, such as running, jumping, carrying, and interacting. The resulting entity set... and action set It can comprehensively and accurately cover the key elements in the video, providing a real and verifiable reference for subsequent text generation.

[0034] After completing entity and action extraction, the identified entity set is processed. With action set The data is aggregated and integrated to form a complete truth database of video elements. This database not only records the category information of each entity and action, but also annotates it with frequency of occurrence, temporal location, or keyframe information, thus providing multi-dimensional structured knowledge. In the subsequent video description generation process, this truth database is used to perform fine-grained semantic verification and accuracy scoring on candidate text, ensuring that the generated text content accurately reflects the real entities and actions in the video, effectively avoiding model-generated illusions or omission of important details.

[0035] Optionally, in this embodiment, a high-quality, structured truth library of video elements can be established to provide accurate training and evaluation basis for the large video description model, thereby significantly improving the performance of the generated text in terms of entity accuracy, action alignment and overall semantic integrity, and improving the reliability and practicality of video understanding and automatic description.

[0036] As an optional example, based on the optimal length of the text description and the truth database of video elements, the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text are calculated as follows: Obtain the manually annotated truth description text corresponding to the sample video; The candidate video description text and the ground truth description text are input into the preset video description scoring model. The preset video description scoring model compares and evaluates the candidate video description text from the perspectives of scene consistency, event matching and reasonableness of character behavior, and outputs the video description score of the candidate video description text based on the comparison and evaluation results.

[0037] Optionally, in this embodiment, based on the optimal length of the text description and the truth database of video elements, a multi-dimensional scoring calculation is performed on the candidate video description text, including a video description score. Adaptive length scoring and fine-grained alignment scoring First, manually annotated ground truth texts corresponding to the sample videos are obtained. These ground truth texts record key details of the scenes, events, and character behaviors in the videos, providing a reference for evaluating candidate texts. Then, the candidate video description texts and ground truth texts are input into a pre-defined video description scoring model. This model is a deep learning-based multimodal evaluation model capable of performing refined comparisons and scoring of candidate texts from dimensions such as scene consistency, event matching, and the reasonableness of character behavior. The calculation of the video description score not only considers whether the text contains key entities and actions from the video but also comprehensively evaluates the semantic coherence and logical reasonableness of the text, thereby outputting an evaluation score reflecting the overall quality of the candidate text.

[0038] As an optional example, based on the optimal length of the text description and the truth database of video elements, the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text are calculated as follows: Calculate the actual length of the candidate video description text; Based on the difference between the optimal and actual length of the text description, an adaptive length score for the candidate video description text is calculated using a length reward function in the form of a Gaussian distribution.

[0039] Optionally, in this embodiment, candidate video description texts are scored in multiple dimensions based on the optimal length of the text description and the truth database of video elements, including adaptive length scoring. Specifically, the actual length of the candidate video description text is first calculated. This length can be obtained by counting the number of words or tokens contained in the text. Then, this actual length is compared with the optimal length of the text description calculated based on the video complexity. A comparison was made to quantify the degree of match between text length and video content. To encourage generated text to be as close as possible to the optimal length, a Gaussian-distributed length reward function was designed to transform length differences into adaptive length scoring. As shown in formula (6): (6) In formula (6), The preset standard deviation parameter controls the tolerance for length deviation. The reward value is calculated when the candidate text length is close to the optimal length. A value close to 1 indicates a high degree of match between the text length and the video complexity; conversely, when the text is too short or too long, the reward value drops rapidly, thus suppressing the generation of inappropriately long texts during the reinforcement learning optimization process.

[0040] Through the adaptive length scoring mechanism, the text generation strategy can be dynamically adjusted, enabling the model to automatically adjust the length of the generated text when faced with videos of different complexities. This satisfies the need for more detailed descriptions in complex videos while avoiding the generation of redundant information in simple videos.

[0041] As an optional example, based on the optimal length of the text description and the truth database of video elements, the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text are calculated as follows: Extract noun and verb sets from candidate video description text; The noun and verb sets are compared with the entity and action sets in the video element truth database to calculate coverage and precision. The fine-grained alignment score of the candidate video description text is calculated by harmonic averaging of coverage and precision.

[0042] Optionally, in this embodiment, based on the optimal length of the text description and the truth database of video elements, a fine-grained alignment score is calculated for the candidate video description text to ensure that the generated text is highly consistent with the actual entities and actions in the video. Specifically, a set of nouns is first extracted from the candidate video description text T. and verb set This is used to represent entities and actions mentioned in the text. Simultaneously, the corresponding entity set is retrieved from the video element truth database. and action set This truth database is generated by pre-trained detection models (such as Faster R-CNN, I3D, etc.) and can accurately record key entities and actions present in videos.

[0043] Subsequently, the set of nouns and verbs extracted from the text is compared with the set of entities and actions in the video to calculate two metrics: recall and precision. Recall measures the degree to which key entities and actions in the video are mentioned in the text, as shown in formula (7): (7) A higher coverage value indicates that the generated text can cover more key elements in the video, which helps to ensure the completeness of the description. Precision is used to measure whether the entities and actions mentioned in the text actually exist in the video, and to suppress the generation of hallucinatory information by the model, as shown in formula (8): (8) Ultimately, the coverage rate and accuracy Harmonic averaging is performed to obtain fine-grained alignment scores for candidate video description texts. As shown in formula (9): (9) This scoring mechanism ensures that the entities and actions in the model-generated text are highly consistent with the real content of the video at a fine-grained level, effectively preventing the generation of redundant, erroneous, or illusory information. When used in conjunction with video description scoring and adaptive length scoring, fine-grained alignment scoring becomes an important component of the reinforcement learning reward signal, guiding the large video description model to optimize its generation strategy during training. This results in video description text generation with high information density, complete key details, and semantic accuracy, improving the accuracy and reliability of video understanding and automatic description.

[0044] Optionally, in this embodiment, candidate video description texts are scored in multiple dimensions, and these scores are weighted and summed to obtain a comprehensive score, which is used to guide the training of the large-scale video description model. Specifically, three independent scores are first calculated for the candidate texts: video description score... Adaptive length scoring and fine-grained alignment scoring Among them, video description rating. By comparing candidate texts with manually annotated ground truth texts, the overall quality of the texts is quantified from multiple dimensions, including scene consistency, event matching, and the reasonableness of character behavior; adaptive length scoring is employed. The optimal length is calculated by combining text length with video complexity. The comparison utilizes a Gaussian distribution function to quantify length deviations, thereby encouraging the generated text length to match the video content; fine-grained alignment scoring. Then, by comparing the set of entities mentioned in the candidate text... and action set The entity set corresponding to the video element truth database and action set Calculate coverage (Recall) and precision (Precision), and obtain the F1 score through harmonic averaging. .

[0045] After obtaining the three independent scores mentioned above, they are weighted and summed according to preset weights w1, w2, and w3 to form a comprehensive score for the candidate video description text. As shown in formula (10): (10) In formula (10), the weights can be adjusted according to the actual task requirements to balance the impact of overall text quality, length matching degree, and fine-grained accuracy on the comprehensive score. This weighted comprehensive score simultaneously constrains the performance of the generated text in terms of information completeness, semantic accuracy, and length reasonableness, providing a reliable reward signal for subsequent reinforcement learning strategy optimization. This ensures that the video description model gradually optimizes its generation strategy during training, enabling the model to generate text that accurately reflects the video content while avoiding redundancy and information loss, thus improving the overall effect of video understanding and automatic text generation.

[0046] As an optional example, the comprehensive score is used as a reinforcement learning reward signal. A reinforcement learning strategy optimization algorithm is employed to iteratively update the parameters of the large video description model, resulting in the iteratively updated large video description model, which includes: The comprehensive score is used as a reward signal and input into the reinforcement learning policy optimization algorithm so that the algorithm can calculate the policy gradient loss based on the reward signal. The parameters of the large video description model are iteratively updated using policy gradient loss.

[0047] Optionally, in this embodiment, the comprehensive score of the candidate video description text is used as the reinforcement learning reward signal. A reinforcement learning policy optimization algorithm is then used to iteratively update the parameters of the large video description model to obtain the updated model. Specifically, the comprehensive score (including the video description score, adaptive length score, and fine-grained alignment score) is first input as a reward signal to a preset reinforcement learning policy optimization algorithm, such as Policy Gradient, Proximal Policy Optimization (PPO), or Generalized Reward Policy Optimization (GRPO). The reinforcement learning policy optimization algorithm evaluates the model's performance in generating the current candidate text based on the reward signal, and guides the model to learn and generate a better text description strategy by maximizing the reward signal.

[0048] In this process, the reinforcement learning policy optimization algorithm calculates the policy gradient loss based on the reward signal. This loss function is used to quantify the deviation between the policy that generates candidate text and the ideal policy under the current model parameters.

[0049] By minimizing the policy gradient loss, the model parameters are iteratively updated, thereby improving the overall quality of the generated text. During the update process, the model gradually strengthens policies that perform well in covering key entities and actions in the video, maintaining reasonable text length, and ensuring overall semantic coherence, while suppressing policies that generate low-quality, verbose, or inaccurate text.

[0050] After multiple rounds of iterative training, the video description model is able to generate descriptive text with high information density, complete key details, concise language, and accurate semantics in complex video scenarios. Meanwhile, this reinforcement learning closed-loop system ensures that the model can adapt to different video complexities during the generation process, optimizing text length and content matching, improving the practical application value and interpretability of the generated text, and providing high-quality technical support for video understanding, automatic summarization, and multimodal content generation.

[0051] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0052] According to another aspect of the embodiments of this application, a training apparatus for video description of a large model is also provided, such as... Figure 4 As shown, it includes: The first calculation module 402 is used to perform spatiotemporal feature analysis on the sample video to calculate its video complexity, and determine the optimal length of the text description corresponding to the sample video based on the video complexity. The recognition module 404 is used to perform entity detection and action recognition on the sample video in order to extract the set of entities and the set of actions present in the sample video and to build a truth database of video elements of the sample video. The generation module 406 is used to input the sample video into the video description big model so that the video description big model can generate at least one candidate video description text based on the video content of the sample video. The second calculation module 408 is used to calculate the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text based on the optimal length of the text description and the truth library of video elements, and to perform a weighted summation of the video description score, adaptive length score, and fine-grained alignment score to obtain the comprehensive score of the candidate video description text. The update module 410 is used to use the comprehensive score as a reinforcement learning reward signal and to iteratively update the parameters of the video description model using a reinforcement learning strategy optimization algorithm to obtain the iteratively updated video description model.

[0053] It should be noted that the first calculation module 402 in this embodiment can be used to execute step S102 in this application embodiment, the identification module 404 in this embodiment can be used to execute step S104 in this application embodiment, the generation module 406 in this embodiment can be used to execute step S106 in this application embodiment, the second calculation module 408 in this embodiment can be used to execute step S108 in this application embodiment, and the update module 410 in this embodiment can be used to execute step S110 in this application embodiment.

[0054] As an optional example, the first computation module includes: The first calculation unit is used to extract the feature vector of each frame of the sample video and calculate the average cosine distance between the feature vectors of adjacent frames to calculate the temporal variability of the sample video. The second calculation unit is used to cluster the feature vectors of all frames of the sample video to select a set of key frames, and calculate the average spatial information entropy of each key frame based on the set of key frames to obtain the spatial information entropy of the sample video. The third calculation unit is used to calculate the video complexity of the sample video by weighted fusion of temporal variability and spatial information entropy according to preset weight coefficients. The determination unit is used to determine the optimal length of the text description for a sample video based on the video complexity, wherein the optimal length of the text description is positively correlated with the video complexity.

[0055] As an optional example, the recognition module includes: The recognition unit is used to input the sample video into the preset detection model so that the preset detection model can perform entity detection and action recognition on the sample video, identify all entity objects and all actions that occur in the sample video, and extract the corresponding entity category information and action category information to generate entity set and action set. The fusion unit is used to aggregate and merge the entity set and the action set to build a truth library of video elements from the sample video.

[0056] As an optional example, the second computation module includes: The acquisition unit is used to acquire the manually annotated truth description text corresponding to the sample video; The evaluation unit is used to input candidate video description text and ground truth description text into a preset video description scoring model, so that the preset video description scoring model can compare and evaluate the candidate video description text from the perspectives of scene consistency, event matching and reasonableness of character behavior, and output the video description score of the candidate video description text based on the comparison and evaluation results.

[0057] As an optional example, the second computation module includes: The fourth calculation unit is used to calculate the actual length of the candidate video description text; The fifth calculation unit is used to calculate the adaptive length score of the candidate video description text based on the difference between the optimal length and the actual length of the text description, using a length reward function in the form of a Gaussian distribution.

[0058] As an optional example, the second computation module includes: The extraction unit is used to extract the set of nouns and the set of verbs from the candidate video description text; The sixth calculation unit is used to compare the set of nouns and the set of verbs with the set of entities and the set of actions in the video element truth database to calculate coverage and accuracy. The seventh calculation unit is used to harmonic average the coverage and accuracy to calculate a fine-grained alignment score for the candidate video description text.

[0059] As an optional example, the update module includes: The eighth computational unit is used to input the comprehensive score as a reward signal into the reinforcement learning policy optimization algorithm, so that the reinforcement learning policy optimization algorithm can calculate the policy gradient loss based on the reward signal. The update unit is used to iteratively update the parameters of the large video description model using policy gradient loss.

[0060] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.

[0061] Figure 5 This is a schematic diagram of an optional electronic device according to an embodiment of this application, such as... Figure 5 As shown, it includes a processor 502, a communication interface 504, a memory 506, and a communication bus 508. The processor 502, communication interface 504, and memory 506 communicate with each other via the communication bus 508. Memory 506 is used to store computer programs; When processor 502 executes a computer program stored in memory 506, it performs the following steps: Spatiotemporal feature analysis is performed on the sample videos to calculate their video complexity, and the optimal length of the text description corresponding to the sample videos is determined based on the video complexity. Entity detection and action recognition are performed on the sample videos to extract the sets of entities and actions present in the sample videos, and a truth database of video elements of the sample videos is constructed. Input the sample video into the video description big model so that the video description big model can generate at least one candidate video description text based on the video content of the sample video; Based on the optimal length of the text description and the truth database of video elements, the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text are calculated. The video description score, adaptive length score, and fine-grained alignment score are then weighted and summed to obtain the comprehensive score of the candidate video description text. The comprehensive score is used as a reward signal for reinforcement learning. The parameters of the video description model are iteratively updated using a reinforcement learning strategy optimization algorithm to obtain the iteratively updated video description model.

[0062] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0063] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0064] As an example, the memory 506 described above may include, but is not limited to, the first calculation module 402, the recognition module 404, the generation module 406, the second calculation module 408, and the update module 410 from the training device for the large video description model described above. Furthermore, it may include, but is not limited to, other module units from the training device for the large video description model described above, which will not be elaborated upon in this example.

[0065] The processor mentioned above can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0066] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0067] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. The device that implements the training method of the large model described in the video above can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile Internet device (MID), PAD, etc. Figure 5 This does not limit the structure of the aforementioned electronic devices. For example, the electronic device may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0068] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0069] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which, when executed by a processor, performs the steps in the above-described training method for a large video description model.

[0070] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0071] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0072] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0073] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0075] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0077] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A training method for a large video description model, characterized in that, include: Spatiotemporal feature analysis is performed on the sample video to calculate its video complexity, and the optimal length of the text description corresponding to the sample video is determined based on the video complexity. Entity detection and action recognition are performed on the sample video to extract the entity set and action set present in the sample video, and a video element truth database of the sample video is constructed. The sample video is input into the video description model, so that the video description model generates at least one candidate video description text based on the video content of the sample video. Based on the optimal length of the text description and the truth library of video elements, the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text are calculated, and the video description score, the adaptive length score, and the fine-grained alignment score are weighted and summed to obtain the comprehensive score of the candidate video description text. The comprehensive score is used as a reinforcement learning reward signal, and the parameters of the video description model are iteratively updated using a reinforcement learning strategy optimization algorithm to obtain the iteratively updated video description model.

2. The method according to claim 1, characterized in that, Performing spatiotemporal feature analysis on sample videos to calculate their video complexity, and determining the optimal length of the text description corresponding to the sample video based on the video complexity, includes: Extract the feature vector of each frame of the sample video and calculate the average cosine distance between the feature vectors of adjacent frames to obtain the temporal variability of the sample video. Clustering is performed on all frame feature vectors of the sample video to select a set of key frames, and the average spatial information entropy of each key frame is calculated based on the set of key frames to obtain the spatial information entropy of the sample video. The temporal variability and spatial information entropy are weighted and fused according to preset weighting coefficients to calculate the video complexity of the sample video. The optimal length of the text description for the sample video is determined based on the video complexity, wherein the optimal length of the text description is positively correlated with the video complexity.

3. The method according to claim 1, characterized in that, Entity detection and action recognition are performed on the sample videos to extract the sets of entities and actions present in the sample videos, and a truth database of video elements for the sample videos is constructed, including: The sample video is input into a preset detection model so that the preset detection model performs entity detection and action recognition on the sample video, identifies all entity objects and all actions that occur in the sample video, and extracts the corresponding entity category information and action category information to generate an entity set and an action set. The entity set and the action set are aggregated and merged to construct a truth database of video elements for the sample video.

4. The method according to claim 1, characterized in that, Based on the optimal length of the text description and the truth database of video elements, the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text are calculated as follows: Obtain the manually annotated truth description text corresponding to the sample video; The candidate video description text and the truth description text are input into a preset video description scoring model, so that the preset video description scoring model compares and evaluates the candidate video description text from the perspectives of scene consistency, event matching and reasonableness of character behavior, and outputs the video description score of the candidate video description text based on the comparison and evaluation results.

5. The method according to claim 1, characterized in that, Based on the optimal length of the text description and the truth database of video elements, the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text are calculated as follows: Calculate the actual length of the candidate video description text; Based on the difference between the optimal length of the text description and the actual length, an adaptive length score for the candidate video description text is calculated using a length reward function in the form of a Gaussian distribution.

6. The method according to claim 1, characterized in that, Based on the optimal length of the text description and the truth database of video elements, the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text are calculated as follows: Extract the noun set and verb set from the candidate video description text; The noun set and the verb set are compared with the entity set and action set in the video element truth database to calculate coverage and accuracy. The fine-grained alignment score of the candidate video description text is calculated by harmonic averaging of the coverage and the accuracy.

7. The method according to any one of claims 1 to 6, characterized in that, The comprehensive score is used as a reinforcement learning reward signal. A reinforcement learning strategy optimization algorithm is employed to iteratively update the parameters of the large video description model, resulting in the iteratively updated large video description model, which includes: The comprehensive score is used as a reward signal and input into the reinforcement learning policy optimization algorithm so that the reinforcement learning policy optimization algorithm can calculate the policy gradient loss based on the reward signal. The parameters of the video description model are iteratively updated using the gradient loss of the aforementioned strategy.

8. A training device for video-described large models, characterized in that, include: The first calculation module is used to perform spatiotemporal feature analysis on the sample video to calculate its video complexity, and determine the optimal length of the text description corresponding to the sample video based on the video complexity. The recognition module is used to perform entity detection and action recognition on the sample video to extract the entity set and action set present in the sample video and construct a video element truth library of the sample video. The generation module is used to input the sample video into the video description big model, so that the video description big model generates at least one candidate video description text based on the video content of the sample video; The second calculation module is used to calculate the video description score, adaptive length score, and fine-grained alignment score of the candidate video description text based on the optimal length of the text description and the truth database of video elements, and to perform a weighted summation of the video description score, the adaptive length score, and the fine-grained alignment score to obtain the comprehensive score of the candidate video description text. The update module is used to use the comprehensive score as a reinforcement learning reward signal and to iteratively update the parameters of the video description model using a reinforcement learning strategy optimization algorithm to obtain the iteratively updated video description model.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images