A complexity adaptive video description method and system based on de-bias reference selection

Abstract

The application relates to a complexity self-adaptive video description method and system based on a de-bias reference selection, and relates to the technical field of computer vision and natural language processing crossing; the method comprises the following steps: step 1: after an input video and an artificial reference subtitle are obtained, real words are extracted, V2c semantic diversity is calculated, complexity buckets are obtained according to equal-frequency bucketing, and granularity self-adaptive degrees are calculated; step 2: bias metrics are calculated according to three instantiation schemes, and minimum-maximum normalization is performed; step 3: a selection rule of de-bias scoring is adopted to pick out supervision targets; step 4: for the selected single supervision target, parameter efficient fine-tuning is performed on a base model, and the trained base model is output; and step 5: after an input video to be described is preprocessed, the input video is input into the trained base model, and a video description is output. The application improves the overall quality of the video description.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and natural language processing, specifically to a complexity-adaptive video description method and system based on biased reference selection. Background Technology

[0002] Video description aims to automatically generate semantically correct natural language descriptions for a given video, placing it at the intersection of computer vision and natural language processing. In recent years, large-scale video models have significantly improved description quality on standard benchmarks by bridging visual encoders with large language model backbones. Representative works include Video-LLaVA for unified image-text understanding, VideoChat with a hierarchical visual-language interface, Qwen2-VL which extends visual-language pre-training with dynamic resolution, and mPLUG-2 trained on large-scale video-text data. However, existing research primarily focuses on aggregated scores and absolute rankings, while a more fundamental question has long been overlooked: does the granularity of the model's descriptions truly adaptively adjust to the complexity of the content being described?

[0003] Intuitively, a video dense with objects, actions, and events should receive a longer and more detailed description than a simpler video; this is precisely what human annotators do, with their reference subtitles being significantly longer on complex videos. A model that truly understands the content should exhibit the same tendency. However, engineers have found that the finely tuned large-scale video model outputs descriptions of almost constant length regardless of video complexity, losing the weak adaptive capabilities already present in its pre-trained foundation, and precisely collapsing on complex content that most requires longer descriptions.

[0004] There are many approaches to existing work on reference caption selection and data quality. Noise-aware training explicitly models mismatched words in the reference, reference-independent evaluation bypasses reference dependence, there is also course learning with samples sorted by difficulty, and distillation methods for training captions by quality weighting. These methods each have their strengths, but they all have one thing in common: all reference captions are retained during training, and no one has specifically introduced a debiasing mechanism in the "selecting a supervision target for each video" step.

[0005] Video description quality is typically adopted wait Tuple overlap metric evaluation. TF-IDF weighted The tuple matching reward is consistent with the consensus of the reference set. This TF-IDF weighting also makes the metric sensitive to length, and this length sensitivity is not uniform across content complexity, but is more pronounced on complex videos.

[0006] Besides the aforementioned approaches, several other existing technologies in this field address the issue from different levels. One type is metric correction, which introduces additional constraints to n-tuple metrics such as CIDEr to correct their matching rules, with the intervention point located at the evaluation metric itself. Another type is complexity adaptation, which calculates video complexity based on video spatiotemporal features and constructs a length reward accordingly to optimize model parameters using reinforcement learning reward signals, with the intervention point located at the optimization objective of training. A third type is annotation quality weighting, which scores the quality of reference subtitles and uses it as a dynamic weight for cross-entropy loss, with the intervention point located at the loss function. Although these approaches have different focuses, they share a common feature: the intervention is applied to the evaluation metric, loss function, or reinforcement learning reward level, while the supervision target itself still uses all references or is selected based on the highest single metric, and none of them introduce a debiasing mechanism in the step of selecting a supervision target for each video. However, those skilled in the art have found that short description bias is transmitted to the model through the selected supervision target. If the bias is not debiased at this stage, it is difficult to block the source transmission of bias, regardless of how it is corrected at the loss or reward level.

[0007] In summary, the problems with existing technologies lie in several areas. The most direct is that the granularity of description cannot keep up with the complexity: the finely tuned video model does not adaptively grow with content complexity, and the description is too short for complex videos, resulting in a "granularity-complexity mismatch." This mismatch is not harmless; it causes quality to collapse directly with complex content, while simple videos are already approaching the upper limit of human judgment. The entire quality gap is precisely concentrated in the complex segments that most require adaptation. The problem goes a step further: compressing the supervision target into a single reference for each video, and then... When selecting the highest score, The short-length bias is inherited with the selection. In complex videos, the short reference is systematically picked out. The bias is passed to the model through the supervised target. Instead of being corrected, it is amplified. Summary of the Invention

[0008] This invention addresses several pain points of existing video description methods: description granularity does not adapt to content complexity, quality collapse occurs with complex content, and short description bias is hidden in the selection of supervision targets. Specific pain points are described below.

[0009] 1) Granularity-complexity mismatch: The fine-tuned video model outputs a nearly constant length of description across the entire complexity range, failing to grow with complexity like human-referenced subtitles. Instead, it writes too short descriptions for the most complex videos with the richest content and the most detailed descriptions.

[0010] 2) Quality Collapse in Complex Content: The aforementioned length mismatch is not a successful compromise of trading quality for simplicity. The reality is that the quality of complex content collapses drastically, simple videos saturate early on, while complex videos only reach a small fraction of their score. The entire quality gap is concentrated in the areas where adaptive content is most lacking.

[0011] 3) Short description bias for monitoring objectives: Compress the monitoring objectives into a single reference, and then... When the highest scorer is selected, The TF-IDF length bias will cause the selected reference to be systematically too short in complex videos. The bias is passed to the model through the supervised target, and the landing point is exactly where the human has the longest target and where the model needs the longest target.

[0012] The innovative aspects of this invention are as follows: 1) Based on Complexity binning and granular adaptive measurement of semantic diversity. Instead of training a separate visual model to infer complexity, this approach directly reads human-referenced subtitles and defines... Semantic diversity is the ratio of the union size of the content words in each reference subtitle to the total length of the content words, then further divided by... Equal-frequency binning. Granularity adaptability is further defined as the slope of the model output length as a function of complexity binning, used to diagnose "granularity-complexity mismatch" and quantify how fine-tuning erases the adaptability of the pre-trained base and collapses on complex content. To rule out the possibility that it's "just a problem with a specific model," multiple heterogeneous models were also included for validation, confirming that the mismatch is systemic rather than model-specific.

[0013] 2) Root cause localization of mismatch. The root cause of the granularity-complexity mismatch was located within the cross-entropy training paradigm. The highest reference is used as the supervision target, leading to a tendency for short descriptions. This tendency is particularly pronounced in complex videos, resulting in the model learning overly short references precisely where longer descriptions are needed.

[0014] 3) Bias-free reference selection framework. A bias-free scoring method is introduced into the supervision target selection process. ,in It is a quality metric It is a bias metric. It is the depolarization strength. Let This means it degenerates into pure quality selection; therefore, this framework is a strict generalization rather than a replacement for existing methods. Bias metric It is intentionally left as a pluggable slot and three instantiations are given: content accessibility (D1), peer divergence (D2), and length deviation (D3), which correspond to the three learnability perspectives of "accessibility", "consensus" and "length", respectively.

[0015] 4) Length bias instantiation and quality-length synergy. Through training-free testing and full training validation, the length bias form D3 was determined to be the most stable and directly addresses the root cause, improving quality on both simple and complex videos and generalizing across datasets. To further eliminate the suspicion that it "simply lengthens subtitles," a pure length control experiment was conducted, demonstrating that the gain comes from the synergy between quality and length, rather than simply length matching.

[0016] This invention is applicable to scenarios such as video content understanding, video retrieval, intelligent video editing, video content review, automatic annotation of short videos and film and television content, and visually impaired auxiliary description. By selecting biased targets for supervision of large video models, the length of the generated video description is adaptively matched with the content complexity, thereby improving the overall quality of the video description.

[0017] The technical solution of this invention is as follows: A complexity-adaptive video description method based on biased reference selection includes: Step 1: After obtaining the input video and its human-referenced subtitles, extract the content words and calculate... Semantic diversity is assessed by dividing the data into equal-frequency buckets to obtain complexity buckets, and the granularity adaptability is calculated. Step 2: For each candidate reference subtitle, calculate the bias metric using the three instantiation schemes D1, D2, and D3 respectively. Then, min-max normalization is performed within the video to make the bias signals between different candidates comparable; where D1, D2, and D3 are bias metrics. The three instantiation schemes respectively regard the bias as the length deviation of content reachability, peer divergence, and relative bucket mean; Step 3: Use bias-corrected scoring The selection rules, given Select the monitoring targets ; Step 4: For the selected single monitoring target Cross-entropy loss is used to efficiently fine-tune the parameters of the base model, and the trained base model is output. Step 5: After preprocessing the input video to be described, input it into the trained base model and output the video description.

[0018] According to a preferred embodiment of the present invention, the specific implementation process of step 1 includes: Step 1.1: V2c semantic diversity; Set video of The reference subtitle is ,in, For video The Reference subtitles, , For video Total number of manually referenced subtitles; Define video of Semantic diversity The ratio of the union size of the content words in each reference subtitle to the total length of the content words is calculated using the following formula: ; In the formula, The union operator for sets, subscript For reference, the subtitle numbering is used. Indicates the video of Find the union of the sets of content words in each reference subtitle; It is the set of content words obtained after removing stop words and words with a length of less than three. The value is ; Step 1.2: Complexity bucketing; For each dataset, the videos are sorted by V2c semantic diversity from smallest to largest and divided into equal-frequency complexity buckets, i.e., the result of complexity bucketing. Step 1.3 Granularity Adaptive Measurement: Let the video description generation model to be evaluated be... For any model capable of generating video descriptions, in the case of equal-frequency complexity buckets The average output subtitle length is Length is measured in terms of the number of real words; Model Granularity adaptability for right The slope is calculated using the following formula: ; In the formula, This refers to the numbering of the buckets with equal frequency complexity after binning the videos according to their complexity from lowest to highest. It is the average bucket number. yes The overall average; This refers to the bucket number.

[0019] Further preferred, the video description generation model adopts Qwen2-VL-7B-Instruct.

[0020] According to a preferred embodiment of the present invention, step 2 includes the following specific implementation process: Step 2.1: Systematic bias analysis; When supervision is compressed to a single reference per video, the highest quality video is selected. When measuring quality, the selected target for: ; In the formula, This is a reference subtitle. Compared to video Other references Score; For video of A collection of manually referenced subtitles; Step 2.2: Three instantiation methods for bias; Bias measurement It is reserved as a pluggable slot, and three instantiation methods are provided, each corresponding to a different learnability perspective; D1 treats bias as content accessibility; let... To fine-tune the front base model for video The generated subtitles are categorized as follows: those containing content that the base model can already generate are considered easy to fit, while those containing content that the base model has not yet generated are considered difficult to fit. Definition: ; In the formula, Reference subtitles for D1 instantiation In the video Bias metric on; It is a set of content words; is the Jacobian coefficient, which is the ratio of the number of elements in the intersection of two sets to the number of elements in the union of two sets, and its value ranges from [0,1]. Measurement Reference Subtitles The degree of overlap between the content words and the content already generated by the base model; D2 considers bias as peer divergence; points similar to other references in the same video are considered easy to learn from, while outliers deviating from peers are considered difficult to learn. Definition: In the formula, Reference subtitles for D2 instantiation In the video Bias metric on; For video Total number of reference subtitles; To traverse the video Internal The iteration variables for all other reference subtitles; Indicates from the reference subtitle set Remove from middle The remaining reference set; For reference subtitles and The calculated CIDEr score; equals 1 minus With the rest Average consensus similarity of the references; D3 considers the bias as the length deviation relative to the barrel mean; the length deviation from the typical barrel length represents the strongest load-bearing metric, defined as: In the formula, Reference subtitles for D3 instantiation In the video Bias metric on; For reference subtitles word count, For video Belonging bucket The average reference length.

[0021] According to a preferred embodiment of the present invention, step 3 specifically includes the following steps: Step 3.1: Remove biased selection frames; For video With candidate reference Define selection rating : ; In the formula, For quality measurement, Compared to other reference train-CIDEr; measure The degree to which the bias deviates from the learnable objective in the dimension of interest; , for the debiasing intensity; the selected monitoring target for: ; Step 3.2: Normalization and debiasing intensity; For each training video, pre-calculate the bias metric of each candidate reference under D1, D2, and D3; Normalize each bias term to minimum-maximum within the video. to reduce polarization intensity Comparable across different definitions.

[0022] According to a preferred embodiment of the present invention, step 4 includes the following specific implementation process: Step 4.1: Efficient fine-tuning of the model and parameters; Using Qwen2-VL-7B-Instruct as the base model, the visual encoder is connected to the 7B parametric language model; The LoRA parameter efficient fine-tuning method is adopted: freeze all the original weights of the base model, and then adjust the weight matrix of the selected projection layer. Bypass insertion of a pair of low-rank matrices and During forward computation, the output of this projection layer is... Become , This is the scaling factor; only the low-rank matrix is ​​updated during training. , The base weights remain unchanged, thus enabling domain adaptation of the base model with a small number of trainable parameters; Step 4.2: Supervision of objectives and training; The supervision goal is output. Using standard cross-entropy loss regression as a single reference; the loss function is: ; In the formula, These are trainable parameters; For monitoring objectives The total number of lexical units; For its first Each word element, For the first All lexical units preceding the given lexical unit; For video The input conditions are the visual word sequence of the sampled frame and the input cues. For the base model in the first The lexical conditional probability distribution of the step.

[0023] A complexity-adaptive video description system based on bias-reference selection includes: The complexity binning module is configured to: after obtaining the input video and its manually referenced subtitles, extract content words and calculate... Semantic diversity is assessed by dividing the data into equal-frequency buckets to obtain complexity buckets, and the granularity adaptability is calculated. The bias metric module is configured to calculate the bias metric for each candidate reference caption using three instantiation schemes: D1, D2, and D3. Then, min-max normalization is performed within the video to make the bias signals between different candidates comparable; where D1, D2, and D3 are bias metrics. The three instantiation schemes respectively regard the bias as the length deviation of content reachability, peer divergence, and relative bucket mean; The debiasing reference selection module is configured to use debiasing scoring. The selection rules, given Select the monitoring targets ; The model fine-tuning module is configured to: for a selected single supervised objective Cross-entropy loss is used to efficiently fine-tune the parameters of the base model, and the trained base model is output. The video description output module is configured to: preprocess the input video to be described, input it into the trained base model, and output the video description.

[0024] The beneficial effects of this invention are as follows: 1) Restore positive granularity adaptation: The fine-tuned video model thus regains the ability to positively adapt as the description length increases with complexity, partially reverting to the manually labeled granularity curve, and reversing the tendency of fine-tuning to erase adaptation.

[0025] 2) Complex Bucket Mass Recovery: Gain is concentrated in complex buckets. Recovery is performed relative to pure mass on the most complex buckets. Approximately 2.52, which is precisely where pure mass selection is almost ineffective, thus providing much-needed relief.

[0026] 3) Quality improvement for both simple and complex videos: The quality of both simple and complex videos... , , The improvement doesn't mean sacrificing simple videos for complex ones. Furthermore, the average length has only increased slightly and remains far below the human reference length.

[0027] 4) Cross-dataset generalization: The ranking remains consistent across the two independent datasets, MSR-VTT and MSVD. In other words, the gain from length bias does not depend on a specific reference distribution.

[0028] 5) Engineering-friendly: Only the selection rules for the supervised target are modified; the model structure and loss function are not affected, allowing direct embedding into existing supervised fine-tuning pipelines. Implementation costs are low, and controllability is strong. Attached Figure Description

[0029] Figure 1 This is a block diagram of the overall structure of the adaptive video description system based on biased reference selection according to the present invention. Figure 2 This is a flowchart illustrating the complexity-adaptive video description method based on biased reference selection according to the present invention. Figure 3 This is a schematic diagram illustrating the granularity-complexity mismatch concept of the present invention; where (a) is the reference answer and the actual output of the model for a simple video, and (b) is a complex video; Figure 4 This is a graph showing the variation of the granularity adaptability of the present invention with the complexity of bucketing. Figure 5 This is a graph showing the variation of the bucketing CIDEr of this invention with bucketing complexity. Figure 6 The CIDEr increment curves for each method relative to the full reference baseline are shown in the bucketed graph. Figure 7 For training loss curves; Figure 8 Comparison charts for qualitative descriptions of complex videos. Detailed Implementation

[0030] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0031] Example 1 A complexity-adaptive video description method based on biased reference selection, such as... Figure 2 As shown, it includes: Step 1: After obtaining the input video and its human-referenced subtitles, extract the content words and calculate... Semantic diversity is used to obtain complexity buckets by equal frequency binning, and granularity adaptability is calculated; the output is complexity coordinates, which lay the foundation for mismatch diagnosis and subsequent bias measurement.

[0032] Step 2: For each candidate reference subtitle, calculate the bias metric using the three instantiation schemes D1, D2, and D3 respectively. Then, min-max normalization is performed within the video to make the bias signals between different candidates comparable; where D1, D2, and D3 are bias metrics. The three instantiation schemes respectively regard the bias as the length deviation of content accessibility, peer divergence, and relative bucket mean; corresponding to three different learnability perspectives, the specific definitions and calculation processes of the three are given in detail in step 2.2; Step 3: Use bias-corrected scoring Replace the original pure The maximization selection rule, given Select the monitoring targets ; Step 4: For the selected single monitoring target The cross-entropy loss method is used to efficiently fine-tune the parameters of the base model and output the trained base model. The key is to only change the supervision target, while keeping the loss function and model structure unchanged.

[0033] Step 5: After preprocessing the input video to be described, input it into the trained base model and output the video description.

[0034] Example 2 The difference between the complexity-adaptive video description method based on bias-reference selection described in Example 1 and the method described in Example 1 is as follows: The specific implementation process of step 1 includes: This invention provides complexity coordinates for mismatch diagnosis and bias measurement. To avoid introducing inductive bias into the visual model, the invention directly infers complexity from human reference captions: videos described by multiple annotators using mutually exclusive terms are considered content-rich, while videos where annotators converge on the same few terms are considered content-simple.

[0035] Step 1.1: V2c semantic diversity; Set video of The reference subtitle is ,in, For video The Reference subtitles, , For video Total number of manually referenced subtitles; Define video of Semantic diversity (V2c: Lexical diversity) is the ratio of the union size of the content words in each reference subtitle to the total length of the content words. The calculation formula is as follows: ; In the formula, The union operator for sets, subscript For reference, the subtitle numbering is used. Indicates the video of Find the union of the sets of content words in each reference subtitle; It is the set of content words obtained after removing stop words and words with a length of less than three; the numerator is the number of elements in the union of the content words of each reference subtitle, and the denominator is the sum of the lengths of the content words of each reference subtitle. The value is When the reference subtitles share almost no content words, the value tends to be 1, indicating that the video contains complex, multifaceted content; conversely, when the reference subtitles repeat the same content words, the value tends to be 0, indicating that the video content is simple. The reason for choosing this value is... Instead of referring to length dispersion or the number of different action verbs, there are reasons for both. The former confuses length with breadth of content, while the latter relies on a hand-written verb dictionary and ignores the description of objects, scenes and attributes.

[0036] Figure 3 The concept of granularity-complexity mismatch is illustrated as follows: the length of manually referenced subtitles increases with complexity (blue), while the output of the fine-tuned video model is almost constant in length (orange); the two match for simple videos, but complex videos show a granularity gap that is not described by the model.

[0037] Step 1.2: Complexity bucketing; For each dataset, videos are sorted by V2c semantic diversity from lowest to highest (from simple content to multifaceted content) and divided into equal-frequency complexity buckets, i.e., complexity binning. Each bucket contains an equal number of videos, and the buckets are arranged from simple to complex. Cross-bucket averaging is then applied. The monotonically increasing average reference caption length confirms that human annotators themselves write longer captions on more complex videos. This binning is the basis for all subsequent analyses: granularity adaptability is the output length's slope across bins, and binning... Gaps are reported in the quality of the barrel report.

[0038] Step 1.3 Granularity Adaptive Measurement: Let the video description generation model to be evaluated be... For any model capable of generating video descriptions, in the case of equal-frequency complexity buckets The average output subtitle length is Length is measured in terms of the number of real words; Model Granularity adaptability for right The slope is calculated using the following formula: ; In the formula, The numbering of each equal-frequency complexity bucket after binning the videos according to their complexity from smallest to largest is used; the larger the bucket number, the higher the video complexity. It is the average bucket number. yes The overall average; A positive value means the model can write longer subtitles on more complex videos; This means that length is insensitive to complexity; A negative value indicates that the video actually becomes shorter on complex videos. This is based on human annotators as a reference, and represents the human adaptiveness. It should be clearly positive, which is also the target that the ideal model should match. This refers to the bucket number.

[0039] Figure 4 and Figure 5 Co-location mismatch: Figure 4 The manual reference length increases sharply with complexity, the fine-tuned model is almost flat, the pre-trained base retains a weak positive slope, and the heterogeneous model mPLUG-2 is also flat. Figure 5 The fine-tuned model shows near saturation in simple buckets and sharp collapse in complex buckets. mPLUG-2 exhibits a homogeneous shape with only an overall upward shift, while the pre-trained base shows uniformly weaker performance across all buckets. This indicates that the issue is resolved with simple videos; the overall quality gap is concentrated in the complex content that requires the most adaptation and involves the longest human reference.

[0040] The video description generation model uses Qwen2-VL-7B-Instruct.

[0041] The specific implementation process of step 2 includes: A bias metric is calculated for each candidate reference subtitle. In a supervised fine-tuning pipeline, both the loss function and the supervision target can induce a tendency towards short subtitles. However, the cross-entropy loss itself is length-independent; it actually inherits the length distribution carried by the supervision target. Therefore, this invention focuses on the selection of the supervision target.

[0042] Step 2.1: Systematic bias analysis; When supervision is compressed to a single reference per video, the standard practice is to select the highest quality one. When measuring quality, the selected target for: ; In the formula, This is a reference subtitle. Compared to video Other references Score; CIDEr is a universally accepted quality assessment standard in the video description field. It scores based on the TF-IDF consensus mechanism of n-grams—segmenting candidate sentences and reference sentences into n-grams, weighting each n-gram according to the inverse document frequency (TF-IDF) of the corpus statistics, calculating the cosine similarity between the weighted vectors of candidate sentences and each reference sentence, applying a length penalty, and finally scoring the video. The rest The average of the references is obtained. The higher the CIDEr score, the greater the consensus between the candidate sentence and the reference set. For video of A collection of manually referenced subtitles;

[0043] This selection method is denoted as the best reference selection (best_ref). The problem lies in... Not length-neutral: Its TF-IDF weighted reward reuse high consensus Tuples, penalty, and introduction of rarity The tuples, and this weighting is asymmetrical with respect to length. In simple videos, short subtitles are sufficient to achieve consensus. Tuples score highly; however, in complex videos, short subtitles can still cover a few consensus words, while long subtitles disperse the probability quality to rare words. Therefore, the bias of the metric on length is more strongly biased towards short subtitles in complex videos. best_ref takes the highest-scoring reference, and this bias is inherited by the supervised target as is. The selected reference is systematically shorter than the bucket mean in complex videos, which happens to fall in the place where the human has the longest target and the model needs the longest target.

[0044] Step 2.2: Three instantiation methods for bias; Bias measurement It is reserved as a pluggable slot, and three instantiation methods are provided, each corresponding to a different learnability perspective; D1 treats bias as content accessibility; let... To fine-tune the front base model for video The generated subtitles are categorized as follows: those containing content that the base model can already generate are considered easy to fit, while those containing content that the base model has not yet generated are considered difficult to fit. Definition: ; In the formula, Reference subtitles for D1 instantiation In the video The bias metric is the deviation quantity defined in this invention; It is a set of content words; is the Jacobian coefficient, which is the ratio of the number of elements in the intersection of two sets to the number of elements in the union (also known as the intersection-union ratio), and its value ranges from [0,1]. Measurement Reference Subtitles The degree of overlap between the content words and the content already generated by the base model: the less overlap, the worse the accessibility of that content to the base model and the more difficult it is to fit. The larger.

[0045] D1 is closest to the literal meaning of learnability; it asks whether the model has reached the target content.

[0046] D2 considers bias as peer divergence; points similar to other references in the same video are considered easy to learn from, while outliers deviating from peers are considered difficult to learn. Definition: In the formula, Reference subtitles for D2 instantiation In the video The bias metric is the deviation quantity defined in this invention; For video Total number of reference subtitles; To traverse the video Internal The iteration variables for all other reference subtitles; Indicates from the reference subtitle set Remove from middle The remaining reference set; For reference subtitles and The calculated CIDEr score; equals 1 minus With the rest Average consensus similarity of the references: The greater the disagreement with one's companions, The older you get, the harder it is to learn.

[0047] D2 does not require base reasoning; it can be calculated from the reference set alone.

[0048] D3 considers the bias as the length deviation relative to the barrel mean; the length deviation from the typical barrel length represents the strongest load-bearing metric, defined as: In the formula, Reference subtitles for D3 instantiation In the video The bias measure, or deviation, is shown in the reference subtitle. The word count deviation relative to the average reference length of its respective complexity bucket; For reference subtitles word count, For video Belonging bucket The average reference length. D3 corresponds most simply and directly to the aforementioned length offset.

[0049] The specific implementation process of step 3 includes: Step 3.1: Remove biased selection frames; This invention retains the single-reference selection pipeline, but injects a correction quality metric into the bias of the selected target, formalizing it as bias-corrected quality selection. For video... With candidate reference Define selection rating :

[0050] ; In the formula, For quality measurement, Compared to other reference train-CIDEr; For reference subtitles Quality metrics—There are various indicators for evaluating the quality of video descriptions; this invention adopts CIDEr, i.e. ; For relative video The remaining references are the train-CIDEr, which uses the training set as the statistical caliber (the method for obtaining CIDEr is in step 2.1, where the document frequency statistics of n-grams are obtained from the training set reference caption corpus, hence the name train-CIDEr, which is used for reference quality evaluation during the training phase and is distinct from the CIDEr calculated on the test set during the evaluation phase). measure The degree to which the bias deviates from the learnable objective in the dimension of interest; , for the debiasing intensity; the selected monitoring target for: ; The time-degenerate relationship is best_ref, therefore this framework is a strict generalization of pure quality selection, not a replacement. There are two design considerations here. First, the correction acts on the supervised objective, not the training loss. The model is still trained on a single reference with standard cross-entropy; only the rule of "which reference to choose" changes. This localizes the intervention, and the comparison with best_ref remains controllable. Second, It's intentionally left as an operable slot, rather than having a fixed definition. After all, which instantiation truly hits the root cause is ultimately a matter of experience.

[0051] Step 3.2: Normalization and debiasing intensity; that is: perform minimum-maximum normalization within the video to make the bias signals between different candidates comparable; For each training video, pre-compute the bias metric for each candidate reference under D1, D2, and D3; each bias term is then min-max normalized within the video to... to reduce polarization intensity Comparable across different definitions. The supervision objective is given. The following is obtained from the above formula , It degenerates into a special case of best_ref.

[0052] For video In the instantiation ( Below, each candidate reference subtitle will be displayed. The bias metric is normalized as follows: ; In the formula, ·and ·Videos The minimum and maximum values ​​of the bias metric for all candidate references within the video are calculated; that is, the minimum value of all biases within the video is subtracted from the calculated bias, and then divided by the difference between the maximum and minimum values ​​of all biases within the video, so that the normalized bias metric value falls within the range of... The normalized bias measure is still denoted as... .

[0053] The specific implementation process of step 4 includes: Step 4.1: Efficient fine-tuning and configuration of the model and parameters; Using Qwen2-VL-7B-Instruct as the base model, a visual encoder is connected to a 7B-parameter language model. This 7B-parameter language model is a publicly available open-source model, consisting of a visual encoder and a language model. The visual encoder encodes each sampled frame of the input video, converting each frame into a sequence of visual lexical units, supporting video / image input at any resolution. The language model is a Transformer-structured decoder with approximately 7 billion (7B) parameters, taking a sequence of interwoven visual and text lexical units as input, and regressively generating text output word by word. "Instruct" indicates that the model has been fine-tuned with instructions and can complete the video description generation task according to the instructions in the input prompts. The connection between the visual encoder and the language model constitutes the video-to-text description generation model in this embodiment. The LoRA (Low-Rank Adaptation) parameter fine-tuning method is employed: all original weights of the base model are frozen, and the weight matrix of the selected projection layer is adjusted accordingly. Bypass insertion of a pair of low-rank matrices and (its rank) (much smaller than the dimension of the weight matrix), during forward computation, the output of this projection layer is... Become , This is the scaling factor; only the low-rank matrix is ​​updated during training. , The base weights remain unchanged, thus enabling domain adaptation of the base model with a small number of trainable parameters; rank is inserted into the query, key, value, output, gating, ascending, and descending projections of the 7B-parameter language model. Scaling A LoRA adapter with dropout=0.05 was used to train only the LoRA parameters while freezing the pedestal weights; optimization was performed using AdamW with a learning rate of [missing information]. After a 100-step warm-up, cosine decay occurs, weights decay by 0.01, gradient accumulation is 8, and a single training round consists of 814 update steps; 8 frames are sampled per video, using bf16 precision. The above hyperparameters are fixed across variants, ensuring that differences arise only from the reference selection rule.

[0054] Step 4.2: Supervision of objectives and training; The supervision goal is output. Using standard cross-entropy loss regression as a single reference; the loss function is: ; In the formula, These are trainable parameters; in this embodiment, they are LoRA adapter parameters. For monitoring objectives The total number of lexical units; For its first Each word element, For the first All lexical elements preceding the given lexical element; For video The input conditions are the visual word sequence of the sampled frame and the input cues. For the base model in the first The step-by-step lexical conditional probability distribution. Training is the process of maximizing the likelihood of the supervised target word-by-word.

[0055] No length instructions are issued during training; therefore, all variants are evaluated on the full test set using a greedy decoding algorithm with a bundle width of 1 and no length constraints. The base model for all variants is the same: the untuned Qwen2-VL-7B-Instruct. The difference lies in the supervision objective: full-reference fine-tuning (Vanilla) samples one reference per round per video across all 20 references; the best reference (best_ref) uses only a single reference. The highest reference; this invention focuses on the debiased target. Training.

[0056] The specific training process in this invention is as follows: (1) Preprocessing: For each training video, calculate the V2c semantic diversity and determine its corresponding equal-frequency complexity bucket according to step 1. Pre-calculate the bias metric of each candidate reference under D1, D2, and D3 according to step 2 and normalize it within the video. (2) Target selection: Follow the bias-reduced scoring in step 3. In the given Select the monitoring targets ; (3) Batch construction: Sample 8 frames for each video, construct video description instruction prompts, and do not issue length instructions; (4) Forward pass and loss: The model encodes visual words in each sampled frame, concatenates them with cue words, and then performs an autoregressive forward pass. The teacher forces the supervision target to be used. Standard cross-entropy loss is calculated per-word meta-analysis; (5) Parameter update: Backpropagation using the AdamW optimizer is used to update only the LoRA adapter parameters and freeze the base weights (hyperparameter configuration is the same as in step 4.1). (6) Training terminates: The training ends after a total of 814 update steps in a single round, resulting in a finely tuned video model.

[0057] After step 5, the greedy decoder outputs a video description, evaluating granularity adaptability and quality; including: Reasoning and Decoding: For each video in the test set, sample 8 frames consistent with the training and construct the same input prompts, but do not issue any length instructions; the fine-tuned base model generates video descriptions word by word in an autoregressive manner, directly selecting the word with the highest conditional probability at each step (i.e., greedy decoding with a bundle width of 1), without imposing length constraints, until the end word is generated or the preset maximum length is reached, and the output description of the video is obtained. Granularity adaptability evaluation; for each video in the entire test set, calculate the average output real word length of the base model on each equal-frequency complexity bucket. Adaptability with granularity Adaptive to the granularity of manually referenced subtitles. For reference, we examined whether the output length of the base model increases synchronously with the video complexity; Quality assessment; for each output description, calculate the relative quality of all human references to the video. Score as a quality indicator (in) The primary metric is supplemented by Bleu_4, METEOR, ROUGE_L, and average subtitle length, and reports are submitted separately for each bucket based on its frequency complexity. The focus is on evaluating the description quality on high-complexity buckets (complex videos); Comparative evaluation: The method of this invention is compared with the full reference fine-tuning baseline (Vanilla) and the best reference baseline (best_ref) under the same untuned pedestal model and the same decoding configuration. The differences between the variants are only due to the selection rules of the supervision target, thus verifying the improvement of the debiased reference selection on granularity adaptability and the quality of complex video description.

[0058] Figure 3 This is a schematic diagram illustrating the concept of granularity-complexity mismatch in this invention; Figure 3 In the text, (a) shows the reference answer and the actual output of the model for a simple video, and (b) shows a complex video. The reference answer is generated manually, while the model generates 6 words. In other words, for a simple video, the results generated manually and by the model are similar. In (b), the reference answer and the generated result show that the model generates 6 words to describe the video, while the manual version requires 13 words.

[0059] Evaluations were performed on two standard video description benchmarks: MSR-VTT provided 6513 training and 2990 test videos, with 20 references per test video; MSVD provided 1200 training and 670 test videos. Each method was trained and evaluated separately on both datasets. To examine behavior across complexity, [the following was done]... The test videos were sorted and divided into 5 buckets of equal frequency, with each bucket containing 598 videos from the MSR-VTT database. The highest... The fifth barrel, bkt4, is the most complex barrel with the most severe mismatch. (Report) The main metric, with appended , , Both the average subtitle length and the average subtitle length conform to the MS COCO subtitle evaluation protocol.

[0060] Before training any variant, we first verify that each bias instantiation is as predicted by its assumptions. This verification is training-free and operates on a training set that actually occurs, already holding each reference for each video. With three types of biases, the definition of arbitrary bias is as follows: Down The value is determined by the simple argmax. Two questions are examined across five buckets: the frequency with which the bias removal rule changes the reference relative to best_ref, and how the bias of the selected reference shifts with complexity. The results are shown in Table 1.

[0061] Table 1. Verification table of three bias instantiation methods and their training-free mechanism;

[0062] The three instantiation methods exhibit different behaviors. D1 aligns with the assumption: the base can access fewer reference words on complex videos, at its highest... Finding a reference point becomes an even more difficult goal. D2, on the other hand, appears lazy; its semantics are too similar to those of the reference points in the video, making it almost impossible to disturb outliers. The sorting is performed, so it is discarded during the training scan. D3 is a paradox: its reselection rate is moderate, the best_ref bias does not increase with complexity but rather decreases slightly, yet training results show that D3 brings the greatest complexity bucket recovery. The reason is that it directly corrects the length bias, rather than through the learnable channel named by its hypothesis.

[0063] Table 2 presents a comparison on MSR-VTT, with each row representing a single link, changing only one factor each time. Vanilla used all 20 reference supervisions. len_ref, random_ref, and best_ref all used single reference supervisions, differing only in their selection methods: by the mean length of the matching reference, random, and the highest train-CIDEr, respectively. This invention uses a bias-free selection method.

[0064] Table 2 Comparison table on MSR-VTT (unit: %; average length unit: words);

[0065] Table 2 contains several layers of information. First, regarding best_ref: its contribution lies in the selection criterion, not the number of references. Two single-reference controls without quality signals both lag behind Vanilla, but once the highest train-CIDEr is selected as the single reference, it surpasses Vanilla to 72.07. Further adjusting for bias, this invention reaches 73.22, 1.15 higher than best_ref. and The length increased synchronously; however, the average length only increased from 6.47 to 7.25, still far from the human reference length, so the gain was not a side effect of "longer subtitles". The length column also reveals diagnostic information: the base average of 12.90 words reflects the pre-training tendency towards lengthiness; after fine-tuning, it collapsed to about 6.5 words, precisely the aforementioned tendency towards short subtitles due to cross-entropy; the invention section recovered to 7.25, making it the only fine-tuned variant to restore the length. As for... Slightly lower, that's because the subtitles are longer. Tuple precision was exchanged , , Gain. Table 3 repeats the same control on MSVD: best_ref vs. Vanilla. The value increased from 126.75 to 155.20, and then by 2.29 to 157.49. All metrics surpassed best_ref, and the ranking on MSR-VTT was reproduced on the second dataset.

[0066] Table 3. Comparison table on MSVD (unit: %; average length unit: words);

[0067] D1 and D3 scans that passed the mechanism test , For the best_ref anchor, report both the entire set and the complex bucket. This was used to differentiate between global improvement and recovery focused on diagnostic predictions. The results are shown in Table 4.

[0068] Table 4. Debiased intensity scan table for D1 and D3 (MSR-VTT; CIDEr and B-4, unit: %; length unit: word);

[0069] The conclusions in Table 4 can be viewed separately. The effective window for D1 is too narrow: Both measures rise at the same time, but once If you continue to increase the gain, it will start to reverse, until... hour It's actually lower than best_ref, which shows that a slight reachability modification is beneficial, but when it outweighs the dominant one... The item was already corrected at that time. D3 is correct. Quite Robust: Complete Series It doesn't collapse within the narrow band of 72.02 to 73.22, and the complex bucket index turns positive across all three intensities (33.67 to 34.09, compared to 31.57 for best_ref). The peak value of the complex bucket index appears at... Recovery 2.52, peak value of the entire series is Such a wide platform means that the length deviation offset targets a stable structural bias, rather than some fine-grained trade-off. They follow different paths: D1... Still carrying Rise, D3 decreases at all intensities Both decreased, but , , The gain of D3 is not the same as that of D3. Tuple precision driven. There's also a behavioral characteristic: D3 follows... The monotony lengthening subtitles were increased, from 6.47 words in best_ref to 7.25, 7.39, and 7.97 words respectively, and the complex bucket subtitles were increased from 6.59 to 8.18 words. This is exactly what the length correction mechanism should look like.

[0070] Figure 6 Tracing best_ref and the present invention D3( (relative to Vanilla's five barrels) Incremental, Vanilla with zero baseline. The gain of best_ref shrinks with complexity, decreasing from +4.57 in bkt0 to +0.81 in bkt4. Pure quality selection is almost useless for the most complex content, which directly proves: The shortest yet highest-scoring reference was chosen for the complex video, and the bias was propagated. The gain of this invention remains consistent, from +6.34 in bkt0 to +3.33 in bkt4. The difference of 2.52 at bkt4 is the independent contribution of the debiasing term, compensating for the bucket where pure quality selection failed. The two are actually complementary: best_ref carries most of the gain in the simple bucket (where the metric length bias is weak), while debiasing carries most of the gain in the complex bucket (where the bias is strong). This method is equivalent to taking the best of each bucket.

[0071] To verify that the gain of D3 is not simply a length effect, a pure length-only reference len_only was trained: for each video, a reference whose length is closest to the average of its complexity buckets is selected, and the reference is not included in the score. Item. If length matching were the sole reason, this comparison should have matched or even surpassed D3. The results are shown in Table 5.

[0072] Table 5. Length Attribution Comparison Table for D3 Gain (MSR-VTT; CIDEr and B-4 Units: %; Length Units: words);

[0073] The results in Table 5 are quite telling. (len_only complete set) It's 7.89 lower than D3 and 1.56 lower than complex bucket, yet the subtitle length of 8.62 words actually exceeds the human upper bound. This shows that length without quality constraints will lead the model to lengthy but low-consensus subtitles. Therefore, length adjustment is necessary, but not sufficient; the quality signal carries the main gain, while the length term merely shapes where it falls. In other words, the length term acts as a regularization term for selection, not a target: it pulls the selected reference back slightly from the extreme of measuring short lengths, but the selection is still mainly based on... Consensus is reached, therefore D3 allows , , Rising together, only Slightly lower. The complex bucket is precisely where the two parts work together: the length of `best_ref` is 6.59, far less than the 10.57 of the manual input. Only 31.57; D3 stretches the length to 7.58. It mentions 34.09; while len_only increases the length to 8.82. However, it is only 32.53. What drives the recovery of complex buckets is the synergy between mass and length, not length itself.

[0074] Figure 7 The training losses of Vanilla, best_ref, D3 (based on this invention), and len_only were tracked. Vanilla was trained with all 20 references for 20 epochs, totaling 16280 steps, while the other three were trained with a single reference for 814 steps per epoch. The horizontal axis was normalized according to their respective lengths. The loss curves followed the quality of the supervised target, not the amount of training: the high-quality single-reference best_ref and D3 achieved lower final-state losses with only one-twentieth of the steps, and even outperformed Vanilla on the test set; len_only could not achieve this, meaning that one high-quality reference is better than twenty repetitions. The final-state losses of the three single-references were ranked as follows: D3 (11.82). best_ref(13.25) len_only(14.83) is exactly the value used in the test. Sort ( The reverse order of the reference. Although the biased reference is shorter than the extreme of the bucket mean, it is still based on consensus. Therefore, the model regresses a target that is both achievable and aligned with the evaluation metric.

[0075] Figure 8Take a complex bucket MSR-VTT video and compare the outputs of Vanilla, best_ref, and this invention. The video shows a woman posing in a yellow bikini, and human references repeatedly mention the bikini and its color. Vanilla collapses into the generic template "a woman is posing on the beach," losing the distinguishing attribute; best_ref restores it to "a woman is posing in a yellow bikini"; this invention provides "a woman in a yellow bikini is posing for a picture," combining the recurring attribute, the posing action, and the framing mentioned by multiple annotators. This example confirms the core finding at the sample level: for complex content, pure quality objectives cause the model to drift towards short and generic descriptions, while length-aware bias guides it towards attributes that are actually agreed upon in the references.

[0076] The above describes the specific implementation of this invention. Overall, this invention relies on the collaborative operation of four main modules to streamline the entire process: the complexity bucketing module uses... Semantic diversity is binned, and granularity adaptability is calculated to complete mismatch diagnosis; the bias measurement module calculates and normalizes biases according to three instantiation methods: D1 / D2 / D3; the bias removal reference selection module uses... Select monitoring targets The video description model fine-tuning module then uses a single line. Cross-entropy fine-tuning base model. The four modules are closely connected and progressively build upon each other. Throughout the process, only the supervision target is changed, while the loss function and model structure remain unchanged. This restores positive granularity adaptation, improves the quality of both simple and complex videos, and generalizes across datasets. It can adapt to batch processing needs in various scenarios such as video content understanding, retrieval, and automatic annotation.

[0077] Example 3 A complexity-adaptive video description system based on biased reference selection, such as Figure 1 As shown, it includes: The complexity binning module is configured to: after obtaining the input video and its manually referenced subtitles, extract content words and calculate... Semantic diversity is used to obtain complexity buckets by equal frequency binning, and granularity adaptability is calculated; the output is complexity coordinates, which lay the foundation for mismatch diagnosis and subsequent bias measurement.

[0078] The bias metric module is configured to calculate the bias metric for each candidate reference caption using three instantiation schemes: D1, D2, and D3. Then, min-max normalization is performed within the video to make the bias signals between different candidates comparable; where D1, D2, and D3 are bias metrics. The three instantiation schemes respectively regard the bias as the length deviation of content accessibility, peer divergence, and relative bucket mean; corresponding to three different learnability perspectives, the specific definitions and calculation processes of the three are given in detail in step 2.2; The debiasing reference selection module is configured to use debiasing scoring. Replace the original pure The maximization selection rule, given Select the monitoring targets ; The model fine-tuning module is configured to: for a selected single supervised objective The cross-entropy loss method is used to efficiently fine-tune the parameters of the base model and output the trained base model. The key is to only change the supervision target, while keeping the loss function and model structure unchanged.

[0079] The video description output module is configured to: preprocess the input video to be described, input it into the trained base model, and output the video description.

Claims

1. A complexity-adaptive video description method based on bias-removed reference selection, characterized in that, include: Step 1: After obtaining the input video and its human-referenced subtitles, extract the content words and calculate... Semantic diversity is assessed by dividing the data into equal-frequency buckets to obtain complexity buckets, and the granularity adaptability is calculated. Step 2: For each candidate reference subtitle, calculate the bias metric using the three instantiation schemes D1, D2, and D3 respectively. Then, min-max normalization is performed within the video to make the bias signals between different candidates comparable; where D1, D2, and D3 are bias metrics. The three instantiation schemes respectively regard the bias as the length deviation of content reachability, peer divergence, and relative bucket mean; Step 3: Use bias-corrected scoring The selection rules, given Select the monitoring targets ; Step 4: For the selected single monitoring target Cross-entropy loss is used to efficiently fine-tune the parameters of the base model, and the trained base model is output. Step 5: After preprocessing the input video to be described, input it into the trained base model and output the video description.

2. The complexity-adaptive video description method based on bias-removal reference selection according to claim 1, characterized in that, The specific implementation process of step 1 includes: Step 1.1: V2c semantic diversity; Set video of The reference subtitle is ,in, For video The Reference subtitles, , For video Total number of manually referenced subtitles; Define video of Semantic diversity The ratio of the union size of the content words in each reference subtitle to the total length of the content words is calculated using the following formula: ； In the formula, The union operator for sets, subscript For reference, the subtitle numbering is used. Indicates the video of Find the union of the sets of content words in each reference subtitle; It is the set of content words obtained after removing stop words and words with a length of less than three. The value is ; Step 1.2: Complexity bucketing; For each dataset, the videos are sorted by V2c semantic diversity from smallest to largest and divided into equal-frequency complexity buckets, i.e., the result of complexity bucketing. Step 1.3 Granularity-adaptive measurement; Let the video description generation model to be evaluated be... For any model capable of generating video descriptions, in the case of equal-frequency complexity buckets The average output subtitle length is Length is measured in terms of the number of real words; Model Granularity adaptability for right The slope is calculated using the following formula: ； In the formula, This refers to the numbering of the buckets with equal frequency complexity after binning the videos according to their complexity from lowest to highest. It is the average bucket number. yes The overall average; This refers to the bucket number.

3. The complexity-adaptive video description method based on bias-removal reference selection according to claim 2, characterized in that, The video description generation model uses Qwen2-VL-7B-Instruct.

4. The complexity-adaptive video description method based on bias-removal reference selection according to claim 1, characterized in that, The specific implementation process of step 2 includes: Step 2.1: Systematic bias analysis; When supervision is compressed to a single reference per video, the highest quality video is selected. When measuring quality, the selected target for: ； In the formula, This is a reference subtitle. Compared to video Other references Score; For video of A collection of manually referenced subtitles; Step 2.2: Three instantiation methods for bias; Bias measurement It is reserved as a pluggable slot, and three instantiation methods are provided, each corresponding to a different learnability perspective; D1 treats bias as content accessibility; let... To fine-tune the front base model for video The generated subtitles are categorized as follows: those containing content that the base model can already generate are considered easy to fit, while those containing content that the base model has not yet generated are considered difficult to fit. Definition: ； In the formula, Reference subtitles for D1 instantiation In the video Bias metric on; It is a set of content words; is the Jacobian coefficient, which is the ratio of the number of elements in the intersection of two sets to the number of elements in the union of two sets, and its value ranges from [0,1]. Measurement Reference Subtitles The degree of overlap between the content words and the content already generated by the base model; D2 considers bias as peer divergence; points similar to other references in the same video are considered easy to learn from, while outliers deviating from peers are considered difficult to learn. Definition: In the formula, Reference subtitles for D2 instantiation In the video Bias metric on; For video Total number of reference subtitles; To traverse the video Internal The iteration variables for all other reference subtitles; Indicates from the reference subtitle set Remove from middle The remaining reference set; For reference subtitles and The calculated CIDEr score; equals 1 minus With the rest Average consensus similarity of the references; D3 considers the bias as the length deviation relative to the barrel mean; the length deviation from the typical barrel length represents the strongest load-bearing metric, defined as: In the formula, Reference subtitles for D3 instantiation In the video Bias metric on; For reference subtitles word count, For video Belonging bucket The average reference length.

5. The complexity-adaptive video description method based on bias-removal reference selection according to claim 1, characterized in that, The specific implementation process of step 3 includes: Step 3.1: Remove biased selection frames; For video With candidate reference Define selection rating : ； In the formula, For quality measurement, Compared to other reference train-CIDEr; measure The degree to which the bias deviates from the learnable objective in the dimension of interest; , for the debiasing intensity; the selected monitoring target for: ； Step 3.2: Normalization and debiasing intensity; For each training video, pre-calculate the bias metric of each candidate reference under D1, D2, and D3; Normalize each bias term to minimum-maximum within the video. to reduce polarization intensity Comparable across different definitions.

6. A complexity-adaptive video description method based on bias-reference selection according to any one of claims 1-5, characterized in that, The specific implementation process of step 4 includes: Step 4.1: Efficient fine-tuning of the model and parameters; Using Qwen2-VL-7B-Instruct as the base model, the visual encoder is connected to the 7B parametric language model; The LoRA parameter efficient fine-tuning method is adopted: freeze all the original weights of the base model, and then adjust the weight matrix of the selected projection layer. Bypass insertion of a pair of low-rank matrices and During forward computation, the output of this projection layer is... Become , This is the scaling factor; only the low-rank matrix is ​​updated during training. , The base weights remain unchanged, thus enabling domain adaptation of the base model with a small number of trainable parameters; Step 4.2: Supervision of objectives and training; The supervision goal is output. Using standard cross-entropy loss regression as a single reference; the loss function is: ； In the formula, These are trainable parameters; For monitoring objectives The total number of lexical units; For its first Each word element, For the first All lexical units preceding the given lexical unit; For video The input conditions are the visual word sequence of the sampled frame and the input cues. For the base model in the first The lexical conditional probability distribution of the step.

7. A complexity-adaptive video description system based on bias-removal reference selection, characterized in that, include: The complexity binning module is configured to: after obtaining the input video and its manually referenced subtitles, extract content words and calculate... Semantic diversity is assessed by dividing the data into equal-frequency buckets to obtain complexity buckets, and the granularity adaptability is calculated. The bias metric module is configured to calculate the bias metric for each candidate reference caption using three instantiation schemes: D1, D2, and D3. Then, min-max normalization is performed within the video to make the bias signals between different candidates comparable; where D1, D2, and D3 are bias metrics. The three instantiation schemes respectively regard the bias as the length deviation of content reachability, peer divergence, and relative bucket mean; The debiasing reference selection module is configured to use debiasing scoring. The selection rules, given Select the monitoring targets ; The model fine-tuning module is configured to: for a selected single supervised objective Cross-entropy loss is used to efficiently fine-tune the parameters of the base model, and the trained base model is output. The video description output module is configured to: preprocess the input video to be described, input it into the trained base model, and output the video description.

Images