Training method of video time positioning model, video time positioning method, equipment and medium

Abstract

The invention relates to the technical field of computer vision, particularly provides a training method of a video time positioning model, a video time positioning method, equipment and a medium, and aims to solve the problem of large video time positioning error. In order to achieve the purpose, the model training method comprises the steps that multiple frames of images are sampled from a training video to serve as training data, the training data and a first preset query text are coded to obtain visual features and text query features, and a video time positioning model is trained based on the visual features and the text query features, obtaining a plurality of candidate answers, respectively calculating the relative advantage value of each candidate answer based on the real answer of the first preset query text and the plurality of candidate answers, adjusting the parameters of the video time positioning model based on each relative advantage value, and continuing to execute the step of sampling multiple frames of images from the training video. Therefore, the performance and accuracy of video time positioning can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a training method, video temporal localization method, device, and medium for a video temporal localization model. Background Technology

[0002] Video temporal localization aims to identify the start and end timestamps of target events in videos of varying lengths. In other words, given a text event query, the goal is to predict the start and end timestamps of the corresponding event within the video. Therefore, unlike semantic question answering, video temporal localization requires sophisticated modeling of both visual content and temporal information. Accurate temporal localization is crucial for understanding long videos and holds immense potential in practical applications such as autonomous driving and artificial intelligence.

[0003] Current technologies primarily apply Multimodal Large Language Models (MLLM) to video temporal localization, using short videos as training data to train the MLLM model. An end-to-end training paradigm is employed, where multiple visual input frames and a text query are given, and the model directly outputs the start and end timestamps of the target event. However, MLLM models trained in this way struggle to handle more complex and longer videos. Using absolute timestamps for prediction on long videos is more prone to producing illusions, making it difficult to achieve satisfactory localization results. Summary of the Invention

[0004] This application addresses the shortcomings of the prior art by proposing a training method, a video time-localization method, an apparatus, and a medium for a video time-localization model.

[0005] In a first aspect, embodiments of this application provide a training method for a video time-localization model, comprising: sampling multiple frames of images from a training video at equal intervals as training data, wherein the actual duration of the training video is less than or equal to a preset duration; encoding the training data and a first preset query text to obtain visual features and text query features; training the video time-localization model based on the visual features and the text query features to obtain multiple candidate answers; using a group relative strategy optimization algorithm, calculating the relative advantage value corresponding to each candidate answer based on the real answer corresponding to the first preset query text and the multiple candidate answers, wherein the relative advantage value is used to characterize the advantage of its corresponding candidate answer relative to other candidate answers in terms of matching degree with the real answer; adjusting the parameters of the video time-localization model based on each relative advantage value, and continuing to execute the step of sampling multiple frames of images from the training video at equal intervals as training data until the training status of the video time-localization model meets preset conditions, thereby obtaining the video time-localization model.

[0006] Secondly, embodiments of this application provide a video time-based localization method, comprising: acquiring a target video and a second preset query text; using a video time-based localization model, based on the second preset query text, performing video time-based localization on the target video to obtain a localization result, wherein the video time-based localization model is trained using the training method of the video time-based localization model described in the first aspect above.

[0007] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method described in the first aspect above.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0009] The technical solutions provided in this application embodiment have at least the following technical effects or advantages: The training method for the video temporal localization model in this application embodiment samples multiple frames of images from a training video at equal time intervals as training data. The actual duration of the training video is less than or equal to a preset duration. The training data and a first preset query text are encoded to obtain visual features and text query features. By sampling at equal time intervals across the entire training video, multiple frames can uniformly cover the beginning, middle, and end of the video. Compared to random frame sampling or only extracting short segments, this method is more conducive to the video temporal localization model learning the global temporal structure and semantic distribution of the video. It reduces high redundancy between adjacent frames while ensuring coverage, which is beneficial for collecting training samples with higher information density under limited computing resources. At the same time, the video temporal localization model does not need to process a large number of almost identical consecutive frames, resulting in lower computational cost, more efficient training, and enhanced sensitivity and generalization ability of the model to changes in temporal position.

[0010] Furthermore, the video temporal localization model is trained based on visual features and text query features to obtain multiple candidate answers. Then, using a group relative strategy optimization algorithm, based on the real answer corresponding to the first preset query text and multiple candidate answers, the relative advantage value corresponding to each candidate answer is calculated. This relative advantage value is used to characterize the advantage of its corresponding candidate answer in terms of matching degree with the real answer compared to other candidate answers. Based on each relative advantage value, the parameters of the video temporal localization model are adjusted, and the step of sampling multiple frames of images from the training video at equal intervals as training data continues until the training status of the video temporal localization model meets the preset conditions, thus obtaining the video temporal localization model. By comparing the matching degree between each candidate answer and the real answer, the relative advantage value of each candidate answer is learned. Essentially, this is comparative learning, which improves the quality of candidate answers and the final positioning accuracy. The relative advantage value directly reflects whether a candidate answer matches the real answer better or worse than other candidate answers. In the training process of the video time positioning model, not only is the best candidate answer encouraged to obtain a high advantage value, but the gap between the best, second-best, and incorrect candidate answers is also widened among multiple candidate answers. This gives the video time positioning model a stronger discriminative ability and can significantly improve the performance of the video time positioning model in video time positioning, making the time positioning of videos more accurate.

[0011] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0012] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a training method for a video temporal localization model provided in an embodiment of this application is shown; Figure 2 This illustration shows a structural schematic diagram of a video temporal positioning model provided in an embodiment of this application; Figure 3 A flowchart of a video time positioning method provided in an embodiment of this application is shown; Figure 4 The distribution diagram of the training dataset and test dataset provided in the embodiments of this application is shown; Figure 5This paper presents a comparison chart showing the results of video time positioning using the video time positioning model provided in this application embodiment and the video time positioning model provided by the prior art. Figure 6 A schematic diagram of the structure of a training device for a video temporal localization model provided in an embodiment of this application is shown; Figure 7 This paper shows a schematic diagram of the structure of a video time positioning device provided in an embodiment of this application; Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0013] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0014] Video temporal localization aims to identify the start and end timestamps of target events in videos of varying lengths. In other words, given a text event query, the goal is to predict the start and end timestamps of the corresponding event within the video. Therefore, unlike semantic question answering, video temporal localization requires sophisticated modeling of both visual content and temporal information. Accurate temporal localization is crucial for understanding long videos and holds immense potential in practical applications such as autonomous driving and artificial intelligence.

[0015] Current technologies primarily apply Multimodal Large Language Models (MLLM) to video temporal localization, using short videos as training data to train the MLLM model. An end-to-end training paradigm is employed, where multiple visual input frames and a text query are given, and the model directly outputs the start and end timestamps of the target event. However, MLLM models trained in this way struggle to handle more complex and longer videos. Using absolute timestamps for prediction on long videos is more prone to producing illusions, making it difficult to achieve satisfactory localization results.

[0016] The above challenges can be attributed to the fact that long videos contain richer visual cues and more dispersed event distributions, which requires models to make temporal inferences based on visual context. Existing temporal localization models largely lack this inference ability, mainly due to two aspects: (1) training paradigm, i.e., existing methods widely adopt supervised fine-tuning to optimize the model, which easily leads to the model memorizing the distribution of training data and performing poorly when dealing with out-of-distribution data, making the model lack inference ability and difficult to handle more complex and longer videos; (2) temporal representation, i.e., absolute timestamps make it difficult for the model to provide a unified temporal representation for videos of different durations, and using absolute timestamps for prediction is more likely to produce illusions on long videos.

[0017] Therefore, to overcome the aforementioned challenges in existing technologies, we propose a long-video temporal localization technique based on reinforcement learning fine-tuning (RFT), a framework designed to enhance the model's inference capabilities in long-video temporal localization. Unlike supervised learning fine-tuning (SFT) that directly supervises the model's output, this application allows the model to engage in extensive exploration and trial and error through an explicit inference process, generating a set of candidate answers. These multiple candidate answers enable the video temporal localization model to explore different temporal localization results based on the visual context and textual queries of the long video, rather than simply memorizing the distribution of the training data.

[0018] To guide this process, we designed two reward functions: the intersection-union ratio reward function, which encourages accurate alignment between candidate answer intervals and true answer intervals; and the intersection compactness reward function, which avoids greedy prediction and promotes precise positioning.

[0019] To further improve inference performance, we replace absolute timestamps with relative frame order. By transforming the temporal localization problem into an inference problem based on frame position, rather than precisely predicting timestamps, we ensure a unified representation for videos of different lengths. As can be seen, the training method of our video temporal localization model simplifies the temporal localization process and makes it more adaptable to variations in video length. By setting a reward function, our video temporal localization model can utilize both visual context and relative frame order, rather than relying on the memory of patterns between visual features and absolute timestamps. This enhances the temporal inference capability of the video temporal localization model, resulting in superior generalization ability when processing long videos.

[0020] Based on this, embodiments of this application provide a training method for a video temporal localization model. The following detailed description of the embodiments of this application is provided in conjunction with the accompanying drawings.

[0021] See Figure 1 The flowchart shown illustrates a training method for a video temporal localization model, which specifically includes the following steps: Step 101: Sample multiple frames of images from the training video at equal intervals as training data. The actual duration of the training video is less than or equal to the preset duration.

[0022] In this embodiment, equal-interval sampling refers to selecting video frames sequentially at fixed time intervals along the timeline of the training video as training data. For example, assuming the training video is L seconds long and N frames need to be sampled, the entire time segment is divided into N equal parts with an interval of approximately Δt = L / N. The sampling time points are then 0, Δt, 2Δt, ..., (N...). 1) Take one frame at each Δt second.

[0023] The preset duration can be a duration preset by a person skilled in the art according to actual needs, or a duration obtained by adjusting the preset duration according to actual needs. This application does not impose a specific limitation. For example, the preset duration in the embodiments of this application can be 60 seconds, 50 seconds, 30 seconds, 10 seconds, 5 seconds, 3 seconds, etc.

[0024] Training videos can be obtained by those skilled in the art from publicly released legal video datasets, video data legally collected by business systems with user authorization, and demonstration or simulation videos recorded by the developer. For each training video, at least one pre-defined query text and its corresponding real answer are pre-labeled to form sample data for training the video temporal localization model. During data collection and use, relevant laws, regulations, and privacy protection standards are followed to ensure the legality and compliance of the training videos' sources.

[0025] Step 102: Encode the training data and the first preset query text respectively to obtain visual features and text query features.

[0026] In this embodiment, the first preset query text refers to the query requirement content used for video localization from the training data. For example, the first preset query text could be "a man is drinking water". Furthermore, the first preset query text may also include prompt information, such as "output the reasoning process and answer with relevant frame numbers". This application does not impose specific limitations.

[0027] In one embodiment of this application, training data can be input into a visual encoder, and the visual encoder can be used to encode the training data to obtain visual features. A first preset query text can be input into a text encoder, and the text encoder can be used to encode the first preset query text to obtain text query features.

[0028] Step 103: Train the video time localization model based on visual features and text query features to obtain multiple candidate answers.

[0029] See Figure 2 A schematic diagram of a video temporal localization model is shown. Figure 2 As shown, the video time-localization model includes a generation module that inputs visual features and text query features into the video time-localization model. The generation module in the video time-localization model generates multiple candidate answers and outputs them based on the text query features and visual features.

[0030] The generation module generates multiple candidate answers in a targeted manner based on text query features and visual features. This enables the generation of candidate answers that are closer to the query semantics, reduces a large number of candidate answers that are irrelevant to the query, improves the accuracy of the candidate answer set, and is more beneficial for subsequent reward calculation and relative advantage calculation.

[0031] Step 104: Using the group relative strategy optimization algorithm, based on the real answer and multiple candidate answers corresponding to the first preset query text, calculate the relative advantage value of each candidate answer. The relative advantage value is used to characterize the advantage of the corresponding candidate answer in terms of matching degree with the real answer compared with other candidate answers.

[0032] In this embodiment of the application, the group relative strategy optimization algorithm refers to a strategy optimization method that optimizes the parameters of a video temporal localization model by comparing the relative merits of each candidate answer with the true answer within a group of candidate answers.

[0033] In one implementation, a group relative strategy optimization algorithm can be used to calculate the relative advantage value of each candidate answer based on the real answer corresponding to the first preset query text and multiple candidate answers.

[0034] The relative advantage value directly reflects whether a candidate answer matches the true answer better or worse than other candidate answers. During the training of the video temporal localization model, it not only encourages the best candidate answer to obtain a high advantage value, but also widens the gap between the best, second-best, and incorrect candidate answers within each candidate answer, enabling the video temporal localization model to obtain stronger discrimination ability, thereby improving the localization accuracy of the video temporal localization model.

[0035] Based on the above implementation, in some modified implementations, a group relative strategy optimization algorithm is used to calculate the relative advantage value of each candidate answer based on the real answer and multiple candidate answers corresponding to the first preset query text. Specifically, the reward value between the real answer and each candidate answer can be calculated, and then the relative advantage value of each candidate answer can be calculated based on each reward value.

[0036] It should be noted that the reward value is used to characterize the degree of matching between the candidate answer and the real answer in the time dimension.

[0037] For each candidate answer, the reward value is calculated separately from the real answer. This allows for a more detailed determination of the quality of the candidate answer, which helps to accurately optimize the video temporal localization model. The model not only focuses on whether there is coverage, but also on whether the boundaries are tight and whether there is little redundancy, thereby improving the temporal localization quality of the video temporal localization model.

[0038] Furthermore, based on the reward value of each candidate, the relative advantage value of each candidate answer is calculated to represent its superiority or inferiority relative to other candidates within the group. This allows better candidate answers to rank higher in the entire group, directly optimizing the ranking results and better meeting the actual needs of video time positioning. Through intra-group comparison, the gap between candidate answers with high and low rewards is actively widened, enabling the video time positioning model to learn to identify candidates that "seem similar but have different qualities" during training, significantly improving the fine-grained positioning capability.

[0039] Based on the above implementation, in some modified implementations, candidate answers may include candidate start frame index and candidate end frame index, and true answers may include true start time index and true end time index. The reward value between the true answer and each candidate answer is calculated respectively. Specifically, the candidate start frame index and candidate end frame index corresponding to each candidate answer are converted into candidate start time index and candidate end time index corresponding to each candidate answer. Then, the reward value between the true answer and each candidate answer is calculated based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer.

[0040] Based on the above implementation method, in some modified implementation methods, the candidate start frame index and candidate end frame index corresponding to each candidate answer are converted into the candidate start time index and candidate end time index corresponding to each candidate answer. Specifically, the conversion can be performed using the following formula:

[0041] Among them, when When t represents the candidate start frame index corresponding to the g-th candidate answer, t represents the candidate start time index corresponding to the g-th candidate answer; when When t represents the candidate end frame index corresponding to the g-th candidate answer, t represents the candidate end time index corresponding to the g-th candidate answer, L represents the total duration of the training video in seconds, and N represents the total number of images in the training data.

[0042] Based on the above implementation, in some modified implementations, the reward value between the true answer and each candidate answer is calculated based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer. Specifically, this can be done by: calculating a first reward value between the true answer and each candidate answer using the intersection-union reward function, based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer, respectively, where the intersection-union reward function is used to characterize the accuracy between each candidate answer and the true answer; and calculating a second reward value between the true answer and each candidate answer using the intersection compactness reward function, where the intersection compactness reward function is used to characterize the compactness between each candidate answer and the true answer; finally, using the reward function, the reward value between the true answer and each candidate answer is calculated based on each first reward value and each second reward value.

[0043] Based on the above implementation method, in some modified implementation methods, the first reward value between the true answer and each candidate answer is calculated using the intersection-union reward function, based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer, respectively. Specifically, the first reward value between the true answer and each candidate answer is calculated using the following intersection-union reward function, based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer:

[0044] in, This represents the g-th first reward value. This represents the interval between the candidate start time index and the candidate end time index corresponding to the g-th candidate answer. This represents the interval between the true start time index and the true end time index corresponding to the true answer. For example, suppose the candidate start time index and the candidate end time index corresponding to the g-th candidate answer calculated using the above implementation method are respectively... and The actual start time index and actual end time index corresponding to the actual answer are respectively and ,So, [ , ], [ , ].

[0045] It should be noted that the intersection-union ratio (IUGR) reward function is used to characterize the accuracy between each candidate answer and the true answer. Specifically, the IUGR reward function uses the ratio of the intersection size to the union size as the reward value. It not only reflects whether the candidate answer covers the true answer, but also measures whether there is a large area of ​​redundancy in the candidate answer that exceeds the true range. If the candidate answer is too large or too small, as long as it causes the union size to increase significantly, even if there is some overlap, the IUGR value will decrease significantly. Therefore, the IUGR reward function in this application emphasizes the degree of bidirectional matching between the candidate answer and the true answer in terms of position and length, thereby making the candidate answer closer to the true answer. It is suitable for scenarios that require the prediction result to "hit the correct position" and "have clean boundaries without unnecessary trailing".

[0046] Based on the above implementation method, in some modified implementation methods, a second reward value between the true answer and each candidate answer is calculated using the intersection compactness reward function, based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer. Specifically, the second reward value between the true answer and each candidate answer is calculated using the following intersection compactness reward function, based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer:

[0047] in, This represents the g-th second reward value. This represents the interval between the candidate start time index and the candidate end time index corresponding to the g-th candidate answer. This represents the interval between the actual start time index and the actual end time index corresponding to the true answer. This indicates the preset intermediate parameters.

[0048] It should be noted that, The value can be a value preset by someone skilled in the art according to actual needs, or it can be a value obtained by someone skilled in the art after adjusting a preset value according to actual needs. This application embodiment does not specifically limit the value. In this application embodiment, It can be any value within the range of 0-1. For example, It can be 0.5.

[0049] The intersection compactness reward function is used to characterize the compactness between each candidate answer and the true answer. Specifically, this intersection compactness reward function, under the premise of already "basically covering" the true answer range, further encourages the candidate answer range to be as compact as possible and not too long, avoiding the video time model from predicting long and wide time periods in order to ensure coverage, thus preventing greedy prediction.

[0050] Based on the above implementation method, in some modified implementation methods, a reward function is used to calculate the reward value between the true answer and each candidate answer based on each first reward value and each second reward value. Specifically, based on each first reward value and each second reward value, the following reward function is used to calculate the reward value between the true answer and each candidate answer:

[0051] in, This represents the g-th reward value. This represents the g-th first reward value. This represents the g-th second reward value. This indicates the preset weighting coefficients.

[0052] It should be noted that, The weighting coefficients can be pre-set values ​​by those skilled in the art according to actual needs, or they can be weighting coefficients obtained by those skilled in the art after adjusting pre-set values ​​according to actual needs. This application embodiment does not specifically limit the weighting coefficients. In this application embodiment, It can be any value within the range of 0-1. For example, It can be 0.5.

[0053] Based on the above implementation method, in some modified implementation methods, the relative advantage value corresponding to each candidate answer is calculated based on each reward value. Specifically, the relative advantage value corresponding to each candidate answer can be calculated based on each reward value using the following relative advantage function:

[0054] in, This represents the relative advantage value corresponding to the g-th candidate answer. Let G represent the reward value between the g-th candidate answer and the true answer, and let G represent the total number of candidate answers.

[0055] Within the framework of the group relative strategy algorithm, different reward functions such as intersection-union ratio reward and intersection compactness reward can be flexibly used to evaluate the quality of candidate answers from different dimensions such as boundary compactness and coverage integrity. This allows the video temporal localization model to simultaneously consider accuracy and recall, boundary alignment and coverage during training, resulting in significant improvements in localization accuracy, robustness and convergence efficiency of the trained video temporal localization model.

[0056] Step 105: Adjust the parameters of the video temporal localization model based on the relative advantage values.

[0057] In one implementation, the parameters of the video temporal localization model can be adjusted by optimizing the objective function using a group relative strategy based on each relative advantage value.

[0058] in, This represents the objective function for group-relative policy optimization. These represent the parameters of the video temporal localization model. This indicates that for the training samples (text query features q and multiple candidate answers) The expected value is obtained by taking these training samples as the old strategy before updating the video temporal localization model. The generated tokens are G, which represents the total number of candidate answers, and token, which represents the smallest processing unit of the video temporal localization model. This indicates the strategy used in training the video time localization model. This represents the ratio of the probability of the new and old strategies achieving the same action at the 1st token step. Let G represent the relative advantage value at the th token step corresponding to the th candidate answer, where each G represents the relative advantage value at the th token step. It includes the relative advantage values ​​corresponding to multiple tokens. This represents the candidate answer corresponding to the current token of the g-th candidate answer, where each It contains multiple candidate answers corresponding to multiple tokens. This represents the preset hyperparameters. Indicates will Limited to the interval [ ] within, if Less than Then take ,like Greater than Then take Otherwise, retain the original value. This indicates the preset reference strategy. This represents the weighting coefficient of the preset penalty item. This represents the KL divergence between the training strategy of the video temporal localization model and the preset reference strategy.

[0059] It should be noted that the objective function is optimized using a group-relative strategy to adjust the parameters of the video temporal localization model. That is, in each training iteration, the old strategy is first used... Generate multiple candidate answers for a given text query feature q. Then, within multiple candidate answers, for all token-level decisions, based on their relative advantage values... Probability ratio between old and new strategies The truncated payoff is calculated and averaged within the group and within the sequence to guide the video temporal localization model to increase the generation probability of high-dominance candidate tokens and decrease the generation probability of low-dominance tokens; simultaneously, the current strategy is incorporated. Reference Strategy The KL divergence penalty term limits the range of policy changes, thereby achieving stable group-to-group policy optimization training.

[0060] When training a video temporal localization model using an objective function based on group-relative policy optimization, it constructs candidate-level and token-level relative advantage values ​​by comparing multiple candidate time periods within the same query. These relative advantage values ​​are then updated using policy probability ratios and pruning operations. This approach effectively leverages the relative strengths and weaknesses of candidates, significantly improving the model's ability to distinguish between similar time intervals and enhancing overall localization accuracy. Furthermore, by truncating the policy ratio and introducing KL divergence constraints with the reference policy, it effectively limits the magnitude of single updates, preventing excessive policy shifts and ensuring stable training while avoiding performance fluctuations. Simultaneously, this objective function supports efficient optimization on offline candidate data generated by older policies, reducing the overhead of video sampling and feature extraction, and exhibiting good engineering usability and scalability.

[0061] Step 106: Determine whether the training status of the video time localization model meets the preset conditions. If it does, proceed to step 107; otherwise, proceed to step 101.

[0062] Step 107: Obtain the video temporal localization model.

[0063] In this embodiment, the training status of the video temporal localization model includes the relative advantage value of the model meeting a first preset requirement, the reward value meeting a second preset requirement, or the training iterations reaching a preset number. This preset number can be the maximum number of training iterations for the video temporal localization model, such as 1000, 3000, or 5000. Therefore, when the video temporal localization model is trained in the current round, the calculated reward value or relative advantage value is compared with the reward value or relative advantage value calculated in the previous round. It is determined whether the change in the reward value and / or relative advantage value of the current round relative to the reward value and / or relative advantage value of the previous round is within a preset range. If it is within the preset range, training ends; if it exceeds the range, it is determined whether the training iterations of the video temporal localization model have reached the predicted number. If the preset number has not been reached, the parameters of the video temporal localization model can be adjusted based on each relative advantage value using a group relative strategy to optimize the objective function; if the preset number has been reached, training ends. By using the relative advantage value and / or reward value, as well as the number of training iterations, to determine whether the training of the video time localization model has ended, it is possible to avoid the training of the video time localization model entering an infinite loop due to the relative advantage value and / or reward value failing to meet the preset requirements.

[0064] The training method for the video temporal localization model in this application embodiment samples multiple frames of images from a training video at equal time intervals as training data. The actual duration of the training video is less than or equal to a preset duration. The training data and a first preset query text are encoded to obtain visual features and text query features. By sampling at equal time intervals across the entire training video, multiple frames can uniformly cover the beginning, middle, and end of the video. Compared to random frame sampling or only extracting short segments, this method is more conducive to the video temporal localization model learning the global temporal structure and semantic distribution of the video. It reduces high redundancy between adjacent frames while ensuring coverage, which is beneficial for collecting training samples with higher information density under limited computing resources. At the same time, the video temporal localization model does not need to process a large number of almost identical consecutive frames, resulting in lower computational cost, more efficient training, and enhanced sensitivity and generalization ability of the model to changes in temporal position.

[0065] Furthermore, the video temporal localization model is trained based on visual features and text query features to obtain multiple candidate answers. Then, using a group relative strategy optimization algorithm, based on the real answer corresponding to the first preset query text and multiple candidate answers, the relative advantage value corresponding to each candidate answer is calculated. This relative advantage value is used to characterize the advantage of its corresponding candidate answer in terms of matching degree with the real answer compared to other candidate answers. Based on each relative advantage value, the parameters of the video temporal localization model are adjusted, and the step of sampling multiple frames of images from the training video at equal intervals as training data continues until the training status of the video temporal localization model meets the preset conditions, thus obtaining the video temporal localization model. By comparing the matching degree between each candidate answer and the real answer within the group, the relative advantage value of each candidate answer is learned. Essentially, this is comparative learning, which improves the quality of candidate answers and the final positioning accuracy. The relative advantage value directly reflects whether a candidate answer matches the real answer better or worse than other candidate answers. During the training process of the video time positioning model, not only is the best candidate answer encouraged to obtain a high advantage value, but the gap between the best, second-best, and incorrect candidate answers is also widened among multiple candidate answers. This gives the video time positioning model a stronger discriminative ability and can significantly improve the performance of the video time positioning model in video time positioning, making the time positioning of videos more accurate.

[0066] See Figure 3 This application also provides a video time positioning method, which includes the following steps: Step 201: Obtain the target video and the second preset query text.

[0067] In one embodiment of this application, the target video can be obtained from a dataset specifically used by those skilled in the art for storing long video temporal positioning benchmarks. The duration of the target video can be less than or equal to a preset duration, or it can be greater than a preset duration; this application does not impose a specific limitation. For the specific distribution of target video selection durations and the distribution of training video selection durations in the above embodiments, please refer to... Figure 4 As shown.

[0068] It should be noted that benchmark datasets for long video temporal localization may include QVHighlights (QueriedVideo Highlights, a query-driven dataset of video highlights), with an average video length of 150 seconds; ActivityNet-Long (ActivityNet Captions, an active video caption dataset), with an average video length of 209 seconds; HiREST (High / Hierarchical REtrieval and Segmentation, a high-quality long video retrieval and temporal localization dataset), with an average video length of 261 seconds; and TACoS (TACoS – Temporal Activity COmpactSequence / Temporal Activity COrporal Segmentation, a temporal activity semantic segment dataset for text-guided video temporal localization), with an average video length of 368 seconds.

[0069] Step 202: Using the video time localization model, based on the second preset query text, perform video time localization on the target video to obtain the localization result.

[0070] In this embodiment, the video time-localization model is trained using the training method for the video time-localization model provided in the above embodiments.

[0071] The video time-localization method provided in this application embodiment is based on the same inventive concept as the video time-localization model training method provided in the above embodiments, and has the same beneficial effects as the methods used, operated or implemented therein.

[0072] Furthermore, to verify the performance of the video temporal localization model in the embodiments of this application, this application uses the video temporal localization model VTG-Reasoner trained in the above embodiments of this application, the first type of video understanding multimodal large language model (General Video Understanding MLLMS) in the prior art, including Video-ChatGPT, VideoLLaMA2, and the second type of video temporal localization multimodal large language model (Video TemporalGrounding MLLMs) in the prior art, including TimeChat, VTimeLLM, HawkEye, Momentor, VTG-LLM, VideoChat-Flash, and TimeSuite to perform video temporal localization processing on the selected test video. Specifically, for any of the aforementioned long video temporal localization benchmark datasets, select one video as the target video and set M test samples for this target video. These test samples include a second preset query text and its corresponding real answer. Then, using the given model, perform video temporal localization on the target video based on each of the second preset query texts, obtaining M localization results. Calculate the ratio of the overlap length between each localization result and the real answer's time segment to the length of the union of the time segments of each localization result and the real answer. If R@0.5 is to be used as the evaluation metric, obtain the total number of ratios greater than or equal to 0.5, and then count the number of ratios greater than... The ratio of the total number of values ​​equal to 0.5 to the total number of positioning results M is used as the evaluation index R@0.5. To use R@0.3 as the evaluation index, the total number of values ​​with a ratio greater than or equal to 0.3 is obtained, and then the ratio of this total number to the total number of positioning results M is used as the evaluation index R@0.3. To use R@0.7 as the evaluation index, the total number of values ​​with a ratio greater than or equal to 0.7 is obtained, and then the ratio of this total number to the total number of positioning results M is used as the evaluation index R@0.7. The final comparison results of using each model for video temporal localization of the target video are shown in Table 1 below.

[0073] Table 1 Where Method represents the name of the specific model used for video time localization processing of the test video, Training Size represents the total number of target videos, and K represents the unit kilobytes.

[0074] As shown in Table 1 above, the video temporal localization model VTG-Reasoner of this application significantly improves all evaluation metrics compared to other existing models. Therefore, the video temporal localization model of this application demonstrates strong generalization ability from short to long videos. Furthermore, although trained on a dataset with an average duration of 38 seconds, it still outperforms other video temporal localization models on videos 10 times longer than the training set. Moreover, other models trained based on the SFT framework, such as VTimeLLM and HawkEye, used 170,000 and 715,000 test samples respectively, while the video temporal localization model of this application achieved superior performance using only 16,000 test samples extracted from their training datasets.

[0075] Furthermore, by statistically analyzing the accuracy of some existing models and the video temporal localization model of this application on a general semantic question answering benchmark for videos, the performance of the video temporal localization model in the embodiments of this application is further verified. The specific statistical results are shown in Table 2 below:

[0076] Table 2 Wherein, Model represents the model name, MVBench and VideoMME represent the evaluation frameworks of the general video question answering benchmark, and Avg / Avg, w / o subs represent the accuracy under each evaluation framework. Based on the comparison data in Table 2, it can be seen that the video temporal localization model of this application does not weaken the general understanding ability of videos. The video temporal localization model of this application performs excellently in both evaluation frameworks, surpassing other models in the prior art.

[0077] Furthermore, to further verify the effectiveness of the video temporal localization model in this application, ablation experiments were conducted. Specifically, the video temporal localization model of this application was trained using various methods as shown in the table below, and then the trained video temporal localization model was used for testing. Evaluation metrics were calculated, and the ablation experiment results are shown in Table 3 below:

[0078] Table 3 It is evident that the evaluation metrics of the video time localization model trained using the training method of the video time localization model in the above embodiments of this application are optimal.

[0079] Finally, a bar chart is generated by comparing the video temporal localization results of the target video with those of some existing technology models and the video temporal localization model of this application. This allows for a clearer comparison, as shown in the specific results below. Figure 5As shown in Figure (a), the horizontal axis represents the total duration of the target video, and the vertical axis represents the proportion of the actual answer falling within the corresponding time interval. In Figures (b)-(d), the horizontal axis represents the total duration of the target video, and the vertical axis represents the proportion of the time interval of the localization result falling within the time interval of the actual answer. It is evident that existing models TimeSuite and TimeChat tend to concentrate predictions within shorter time intervals, resulting in smaller errors between the predicted results and the actual answer's time interval. However, when the total duration of the target video is longer, the error between the predicted results and the actual answer's time interval becomes significantly different. That is, when the total duration of the target video is longer, the video time localization performance of existing models TimeSuite and TimeChat drops significantly. In contrast, the video time localization model VTG-Reasoner of this application closely approximates the actual answer's time interval from short to long total video durations. Therefore, the video time localization model of this application demonstrates stronger generalization ability from short to long videos.

[0080] See Figure 6 This application also provides a training apparatus for a video temporal localization model, which is used to execute the training method for the video temporal localization model described in the above embodiments. The apparatus includes: The first acquisition module 301 is used to sample multiple frames of images from the training video at equal intervals as training data, wherein the actual duration of the training video is less than or equal to a preset duration. Encoding module 302 is used to encode the training data and the first preset query text respectively to obtain visual features and text query features; Training module 303 is used to train the video time localization model based on the visual features and the text query features to obtain multiple candidate answers; The calculation module 304 is used to use a group relative strategy optimization algorithm to calculate the relative advantage value of each candidate answer based on the real answer corresponding to the first preset query text and the multiple candidate answers. The relative advantage value is used to characterize the advantage of the corresponding candidate answer in terms of matching degree with the real answer relative to other candidate answers. The adjustment module 305 is used to adjust the parameters of the video temporal localization model based on each relative advantage value, and continue to execute the step of sampling multiple frames of images from the training video at equal intervals as training data until the training status of the video temporal localization model meets the preset conditions, thereby obtaining the video temporal localization model.

[0081] The training device for the video time-localization model provided in this application embodiment and the training method for the video time-localization model provided in the above embodiments are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0082] See Figure 7 This application also provides a video time positioning device for performing the video time positioning method described in the above embodiments. The device includes: The second acquisition module 401 is used to acquire the target video and the second preset query text; The time positioning module 402 is used to perform video time positioning on the target video based on the second preset query text using a video time positioning model to obtain a positioning result. The video time positioning model is trained by the training method of the video time positioning model described in the above embodiment.

[0083] The video time positioning device provided in this application embodiment and the video time positioning method provided in the above embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0084] This application also provides an electronic device corresponding to the training method and video temporal localization method for the video temporal localization model provided in the foregoing embodiments. Please refer to... Figure 8 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 8 As shown, the electronic device 50 may include: a processor 500, a memory 501, a bus 502, and a communication interface 503. The processor 500, the communication interface 503, and the memory 501 are connected via the bus 502. The memory 501 stores a computer program that can run on the processor 500. When the processor 500 runs the computer program, it executes the training method and the video time positioning method of the video time positioning model provided in any of the foregoing embodiments of this application.

[0085] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0086] Bus 502 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 501 is used to store programs. After receiving an execution instruction, processor 500 executes the program. The training method and video time-localization method of the video time-localization model disclosed in any of the foregoing embodiments of this application can be applied to processor 500, or implemented by processor 500.

[0087] The processor 500 may be an integrated circuit with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 500 or by instructions in software form. The processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. The processor 500 reads the information in memory 501 and, in conjunction with its hardware, completes the steps of the above method.

[0088] The electronic device provided in this application embodiment and the training method and video time positioning method of the video time positioning model provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, run or implement.

[0089] This application also provides a computer-readable storage medium corresponding to the training method and video time-localization method of the video time-localization model provided in the foregoing embodiments, wherein a computer program (i.e., a program product) is stored thereon. When the computer program is run by a processor, it executes the training method and video time-localization method of the video time-localization model provided in any of the foregoing embodiments.

[0090] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0091] This application also provides a computer program product corresponding to the training method and video time-localization method of the video time-localization model provided in the foregoing embodiments, including a computer program that is executed by a processor to implement the training method and video time-localization method of the video time-localization model provided in the above embodiments.

[0092] The computer-readable storage medium and computer program product provided in the above embodiments of this application are based on the same inventive concept as the training method and video time positioning method of the video time positioning model provided in the embodiments of this application, and have the same beneficial effects as the methods adopted, run or implemented by the application stored therein.

[0093] It should be noted that: The algorithms and displays provided herein are not inherently related to any particular computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the teachings herein. The required structure for constructing such devices is apparent from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0094] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0095] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various aspects of the invention, in the description of exemplary embodiments of this application above, various features of this application are sometimes grouped together into a single embodiment, figure, or description thereof.

[0096] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed herein and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed herein may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0097] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this application and form different embodiments.

[0098] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the virtual machine creation apparatus according to embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0099] It should be noted that the above embodiments are illustrative of this application and not limiting of it, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. Among the units enumerating several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0100] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a video temporal localization model, characterized in that, include: Multiple frames of images are sampled from the training video at equal intervals as training data, and the actual duration of the training video is less than or equal to a preset duration. The training data and the first preset query text are encoded to obtain visual features and text query features; The video time localization model is trained based on the visual features and the text query features to obtain multiple candidate answers; Using a group relative strategy optimization algorithm, based on the real answer corresponding to the first preset query text and the multiple candidate answers, the relative advantage value corresponding to each candidate answer is calculated respectively. The relative advantage value is used to characterize the advantage of its corresponding candidate answer in terms of matching degree with the real answer relative to other candidate answers. Based on the relative advantage values, the parameters of the video temporal localization model are adjusted, and the step of sampling multiple frames of images from the training video at equal intervals as training data continues until the training status of the video temporal localization model meets the preset conditions, thus obtaining the video temporal localization model.

2. The training method for the video temporal localization model according to claim 1, characterized in that, The group relative strategy optimization algorithm calculates the relative advantage value of each candidate answer based on the true answer corresponding to the first preset query text and the multiple candidate answers, including: Calculate the reward value between the true answer and each of the candidate answers, whereby the reward value is used to characterize the degree of matching between the candidate answer and the true answer in the time dimension; Based on each of the aforementioned reward values, calculate the relative advantage value corresponding to each of the aforementioned candidate answers.

3. The training method for the video temporal localization model according to claim 2, characterized in that, The candidate answers include candidate start frame indices and candidate end frame indices, and the true answers include true start time indices and true end time indices. The calculation of reward values ​​between the true answers and each of the candidate answers includes: The candidate start frame index and candidate end frame index corresponding to each candidate answer are respectively converted into the candidate start time index and candidate end time index corresponding to each candidate answer; The reward value between the true answer and each candidate answer is calculated based on the true start time index and the true end time index corresponding to the true answer, and the candidate start time index and the candidate end time index corresponding to each candidate answer.

4. The training method for the video temporal localization model according to claim 3, characterized in that, The calculation of the reward value between the true answer and each candidate answer, based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer, includes: Based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer, a first reward value between the true answer and each candidate answer is calculated using the intersection-union (IUU) reward function, where the IUU reward function characterizes the accuracy between each candidate answer and the true answer; and, The intersection compactness reward function is used to calculate the second reward value between the true answer and each candidate answer, whereby the intersection compactness reward function is used to characterize the compactness between each candidate answer and the true answer. Using a reward function, the reward value between the true answer and each candidate answer is calculated based on each of the first reward value and each of the second reward values.

5. The training method for the video temporal localization model according to claim 4, characterized in that, The step of calculating the first reward value between the true answer and each candidate answer using the intersection-union reward function, based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer, includes: Based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer, the first reward value between the true answer and each candidate answer is calculated using the following intersection-union reward function: in, This represents the g-th first reward value. This represents the interval between the candidate start time index and the candidate end time index corresponding to the g-th candidate answer. This represents the interval between the true start time index and the true end time index corresponding to the true answer.

6. The training method for the video temporal localization model according to claim 4, characterized in that, The step of calculating the second reward value between the true answer and each candidate answer using the intersection compactness reward function, based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer, includes: Based on the true start time index and true end time index corresponding to the true answer, and the candidate start time index and candidate end time index corresponding to each candidate answer, the second reward value between the true answer and each candidate answer is calculated using the following intersection compactness reward function: in, This represents the g-th second reward value. This represents the interval between the candidate start time index and the candidate end time index corresponding to the g-th candidate answer. This represents the interval between the true start time index and the true end time index corresponding to the true answer. This indicates the preset intermediate parameters.

7. The training method for the video temporal localization model according to claim 2, characterized in that, The step of calculating the relative advantage value corresponding to each candidate answer based on each of the reward values ​​includes: Based on each of the aforementioned reward values, the relative advantage value corresponding to each of the aforementioned candidate answers is calculated using the following relative advantage function: in, This represents the relative advantage value corresponding to the g-th candidate answer. Let G represent the reward value between the g-th candidate answer and the true answer, where G represents the total number of candidate answers.

8. A video time positioning method, characterized in that, include: Obtain the target video and the second preset query text; Using a video time-based localization model, the target video is located based on the second preset query text to obtain a localization result. The video time-based localization model is trained using the training method of the video time-based localization model according to any one of claims 1-7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-8.

Images