A model training method, apparatus, device, medium, and computer program product

Abstract

This invention provides a model training method, apparatus, device, medium, and computer program product, relating to the field of artificial intelligence technology. The method includes: extracting multiple sample feature slices at different temporal resolutions from at least one original training video to obtain a training dataset; inputting the training dataset into a multimodal model to be trained to obtain multiple responses corresponding to each sample feature slice; evaluating the responses using a reward function to obtain evaluation parameters corresponding to each response; obtaining update parameters for the multimodal model to be trained based on the evaluation parameters; and adjusting the multimodal model to be trained using the update parameters to obtain a video inference model. By extracting multiple sample feature slices at different temporal resolutions, the multimodal model to be trained faces multiple perspectives simultaneously, forcing the model to learn to integrate information from different time scales, thereby constructing a more comprehensive and robust internal representation, fundamentally overcoming the limitations of a single perspective.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, device, medium, and computer program product. Background Technology

[0002] Multimodal large-scale models have made significant progress in video understanding, but their video reasoning capabilities still face bottlenecks. Compared to images, videos contain a vast amount of information, shifting from static two-dimensional to dynamic spatiotemporal dimensions. This massive amount of information leads to significant redundancy, meaning that the information extracted during video reasoning is often only partial and fails to capture the key information needed to solve complex problems, thus hindering the improvement of video reasoning capabilities. Summary of the Invention

[0003] The purpose of this invention is to provide a model training method, apparatus, device, medium, and computer program product to solve the problem in the prior art where multimodal large models only focus on partial frame information of the video when performing video inference, resulting in one-sided information extraction and thus inference errors.

[0004] To achieve the above objectives, embodiments of the present invention provide a model training method, comprising:

[0005] Extract multiple sample feature slices with different temporal resolutions from at least one original training video to obtain the training dataset;

[0006] The training dataset is input into the multimodal model to be trained to obtain multiple responses corresponding to each of the sample feature slices; wherein, the multimodal model to be trained is used to analyze the sample feature slices to obtain the inference results of the sample feature slices; the response includes the inference results;

[0007] The response is evaluated using a reward function to obtain the evaluation parameters corresponding to each response;

[0008] Based on the evaluation parameters, the updated parameters of the multimodal model to be trained are obtained; wherein, the updated parameters are model parameters that enable the objective function corresponding to the multimodal model to be trained to meet the constraint conditions;

[0009] The updated parameters are used to adjust the multimodal model to be trained to obtain a video inference model.

[0010] Optionally, the method, wherein evaluating the response using a reward function to obtain evaluation parameters corresponding to each response, includes:

[0011] The scalar reward value corresponding to each of the responses is determined using the reward function.

[0012] The consensus advantage of a response is obtained based on the scalar reward value corresponding to each response; wherein the evaluation parameters include the consensus advantage.

[0013] Optionally, the method, wherein obtaining the consensus advantage of a response based on the scalar reward value corresponding to each response, includes:

[0014] Based on the scalar reward value, mean reward, and standard deviation of reward corresponding to each response, the macroscopic temporal advantage corresponding to each response is obtained; wherein, the mean reward is the average of the scalar reward values ​​corresponding to all the sample feature slices in the training dataset; and the standard deviation of reward is the standard deviation of the scalar reward values ​​corresponding to all the sample feature slices in the training dataset.

[0015] The micro-temporal advantage corresponding to each response is obtained based on the scalar reward value, the mean target reward, and the standard deviation of the target reward corresponding to each response; wherein, the mean target reward is the average of the scalar reward values ​​corresponding to all sample feature slices in the training dataset that have the same temporal resolution as the sample feature slice corresponding to the response; and the standard deviation of the target reward is the standard deviation of the scalar reward values ​​corresponding to all sample feature slices in the training dataset that have the same temporal resolution as the sample feature slice corresponding to the response.

[0016] The consensus advantage of a response is obtained based on the macro-time advantage and the micro-time advantage corresponding to each response.

[0017] Optionally, the method, wherein obtaining the updated parameters of the multimodal model to be trained based on the evaluation parameters includes:

[0018] Define the optimization strategy model for the multimodal model to be trained;

[0019] Obtain a first probability of generating the corresponding response by inputting the sample feature slice into the optimization strategy model, and a second probability of generating the corresponding response by inputting the sample feature slice into the multimodal model to be trained;

[0020] Based on the first probability, the second probability, and the evaluation parameters corresponding to each response, obtain the loss term corresponding to each response;

[0021] The objective function is obtained by summing the loss terms corresponding to the responses of all the sample feature slices in the training dataset.

[0022] The updated parameters are obtained when the objective function meets the constraints.

[0023] Optionally, the method further includes:

[0024] Determine a pre-trained multimodal base model;

[0025] Constructing multimodal thinking chain reasoning data;

[0026] The multimodal basic model is fine-tuned using the multimodal thinking chain inference data to obtain the multimodal model to be trained.

[0027] Optionally, the method, wherein extracting multiple sample feature slices with different temporal resolutions from at least one original training video to obtain a training dataset includes:

[0028] Multiple first sampling start points are determined in the original training video based on the sampling step size corresponding to multiple time resolutions; wherein, the first sampling start point is the start point corresponding to the sample feature slice in the original training video;

[0029] Based on a preset random distribution function, generate multiple random offsets;

[0030] Using the second sampling starting point as a reference, the original training video is extracted to obtain the sample feature slices according to the sampling step size; wherein, the second sampling starting point is the sampling starting point obtained by offsetting the first sampling starting point according to the random offset.

[0031] To achieve the above objectives, embodiments of the present invention provide a model training apparatus, comprising:

[0032] The first acquisition module is used to extract multiple sample feature slices with different temporal resolutions from at least one original training video to obtain the training dataset.

[0033] The second acquisition module is used to input the training dataset into the multimodal model to be trained, and acquire multiple responses corresponding to each of the sample feature slices; wherein, the multimodal model to be trained is used to analyze the sample feature slices and obtain the inference results of the sample feature slices; the response includes the inference results;

[0034] The third acquisition module is used to evaluate the response using a reward function and acquire the evaluation parameters corresponding to each response;

[0035] The fourth acquisition module is used to acquire the update parameters of the multimodal model to be trained based on the evaluation parameters; wherein the update parameters are model parameters that enable the objective function corresponding to the multimodal model to be trained to meet the constraint conditions.

[0036] The fifth acquisition module is used to adjust the multimodal model to be trained using the updated parameters to acquire the video inference model.

[0037] To achieve the above objectives, embodiments of the present invention provide a network device, including: a processor, a memory, and a program or instructions stored in the memory and executable on the processor; wherein, when the processor executes the program or instructions, it implements the model training method described above.

[0038] To achieve the above objectives, embodiments of the present invention provide a readable storage medium having a program or instructions stored thereon, wherein the program or instructions, when executed by a processor, implement the steps in the model training method described above.

[0039] To achieve the above objectives, embodiments of the present invention provide a computer program product, which includes computer instructions that, when executed by a processor, implement the steps of the model training method described above.

[0040] The beneficial effects of the above-described technical solution of the present invention are as follows:

[0041] In this embodiment of the invention, by extracting multiple sample feature slices with different temporal resolutions, the multimodal model to be trained faces multiple perspectives simultaneously. This forces the model to learn to integrate information from different time scales, thereby constructing a more comprehensive and robust internal representation. This fundamentally overcomes the limitations of a single perspective and solves the problem in the prior art where large multimodal models only focus on some frame information of the video when performing video inference, resulting in one-sided information extraction and subsequent inference errors. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the model training method described in an embodiment of the present invention;

[0043] Figure 2 This is a flowchart of the model training method described in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the standard deviation of the reward curve for the model training method described in this embodiment of the invention;

[0045] Figure 4 This is a schematic diagram of the mean reward curve of the model training method described in the embodiment of the present invention.

[0046] Figure 5This is a schematic diagram of the model training device described in an embodiment of the present invention. Detailed Implementation

[0047] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0048] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0049] In various embodiments of the present invention, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0050] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0051] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0052] For ease of understanding, the following describes some aspects of the embodiments of the present invention:

[0053] like Figure 1 As shown, an embodiment of the present invention provides a model training method, which includes:

[0054] Step S10: Extract multiple sample feature slices with different temporal resolutions from at least one original training video to obtain the training dataset;

[0055] It should be noted that, as Figure 2 As shown, in step 201, K sample feature slices with different temporal resolutions are extracted from the original training video to obtain a multi-resolution temporal pyramid. (i.e., the training dataset), each Corresponding to a different target frame rate .

[0056] Step S20: Input the training dataset into the multimodal model to be trained, and obtain multiple responses corresponding to each of the sample feature slices; wherein, the multimodal model to be trained is used to analyze the sample feature slices and obtain the inference results of the sample feature slices; the response includes the inference results;

[0057] It should be noted that, in Figure 2 In step 202, the multi-resolution temporal pyramid is input into the policy model (i.e., the multimodal model to be trained), and a set (n) of candidate inference results or answers are generated for each input, denoted as the response. .

[0058] Step S30: Evaluate the response using a reward function to obtain the evaluation parameters corresponding to each response;

[0059] It should be noted that, in Figure 2 In step 203, the reward function is used to evaluate the response. For each response... Based on the reward function, a series of scalar reward values ​​are calculated. In step 204, based on the scalar reward value... The advantage of the hierarchical consensus is calculated by obtaining the evaluation parameters corresponding to each of the aforementioned responses.

[0060] Step S40: Obtain the updated parameters of the multimodal model to be trained based on the evaluation parameters; wherein, the updated parameters are model parameters that enable the objective function corresponding to the multimodal model to be trained to meet the constraint conditions.

[0061] It should be noted that, in Figure 2 In step 205, based on the hierarchical consensus advantage (i.e., the evaluation parameters), the update parameters that enable the objective function to reach the preset constraints are obtained. The constraints can be to maximize the objective function. An optimizer is used to maximize the objective function to achieve the optimization and parameter update of the multimodal model to be trained. The optimizers that can be used include the Adaptive Moment Estimation (Adam), the Improved Adaptive Moment Estimation (Adam with Weight Decay, AdamW), the Muon (Momentum Orthogonalized by Newton-Schulz, Muon) optimizer, etc.

[0062] Step S50: Adjust the multimodal model to be trained using the updated parameters to obtain the video inference model;

[0063] It should be noted that, in Figure 2In step 206, the policy model is updated using an objective function, that is, the training multimodal model is adjusted using the update parameters to obtain a video inference model.

[0064] In this embodiment, by extracting multiple sample feature slices with different temporal resolutions, the multimodal model to be trained faces multiple perspectives simultaneously. This forces the model to learn to integrate information from different time scales, thereby constructing a more comprehensive and robust internal representation. This fundamentally overcomes the limitations of a single perspective and solves the problem in the prior art where large multimodal models only focus on some frame information of the video when performing video inference, resulting in one-sided information extraction and subsequent inference errors.

[0065] Optionally, the method, wherein step S30 includes:

[0066] The scalar reward value corresponding to each of the responses is determined using the reward function.

[0067] The consensus advantage of a response is obtained based on the scalar reward value corresponding to each response; wherein the evaluation parameters include the consensus advantage.

[0068] In this embodiment, for each of the aforementioned responses Based on the reward function, a series of scalar reward values ​​are calculated. The reward function can be a formatted reward function, a precision reward function, a combination of formatted and precision rewards, or a pre-trained reward evaluation model. It utilizes macroscopic temporal advantage. Evaluate the global performance of a response across all temporal scales (i.e., multiple stated temporal resolutions) to facilitate the model's learning of general knowledge across temporal resolutions; and leverage micro-temporal advantages. Evaluating the local performance of a response within its specific temporal scale (i.e., the temporal resolution) encourages the model to perform refined evidence mining at each scale. Finally, macroscopic temporal advantage is employed. and micro-temporal advantages The consensus advantage obtained after combination is used as the evaluation parameter, balancing the requirements for cross-resolution consistency and the requirements for specific resolution accuracy.

[0069] Optionally, the method, wherein obtaining the consensus advantage of a response based on the scalar reward value corresponding to each response, includes:

[0070] Based on the scalar reward value, mean reward, and standard deviation of reward corresponding to each response, the macroscopic temporal advantage corresponding to each response is obtained; wherein, the mean reward is the average of the scalar reward values ​​corresponding to all the sample feature slices in the training dataset; and the standard deviation of reward is the standard deviation of the scalar reward values ​​corresponding to all the sample feature slices in the training dataset.

[0071] The micro-temporal advantage corresponding to each response is obtained based on the scalar reward value, the mean target reward, and the standard deviation of the target reward corresponding to each response; wherein, the mean target reward is the average of the scalar reward values ​​corresponding to all sample feature slices in the training dataset that have the same temporal resolution as the sample feature slice corresponding to the response; and the standard deviation of the target reward is the standard deviation of the scalar reward values ​​corresponding to all sample feature slices in the training dataset that have the same temporal resolution as the sample feature slice corresponding to the response.

[0072] The consensus advantage of a response is obtained based on the macro-time advantage and the micro-time advantage corresponding to each response.

[0073] In this embodiment, the reward for the current response is used... The reward is calculated by normalizing the mean and standard deviation of all responses across all levels (i.e., multiple time resolutions), thus rewarding "consensus" reasoning that performs well across various time scales. The specific calculation process for this macroscopic time-series advantage involves: first, calculating the scalar reward value corresponding to each response. The average value (K is the maximum index of the time resolution) (i.e., the average reward), followed by all Calculate the standard deviation (i.e., the standard deviation of the reward), using... Subtract all The average of the values ​​(i.e., the mean of the rewards) is then compared with all the average values. The standard deviation of the reward (i.e., the standard deviation of the reward) can be used to obtain the macroscopic time series advantage under parameters i,k. The specific formula for the macroscopic time series advantage can be expressed as:

[0074] ;

[0075] In the formula, mean() represents the mean and std() represents the standard deviation.

[0076] The micro-temporal advantage is calculated by normalizing the reward distribution of the current level (i.e., all sample feature slices with the same temporal resolution corresponding to the response). This rewards "specific" inferences that perform exceptionally well at a particular temporal resolution. This is achieved by applying all reward values ​​at the same temporal resolution K. - Calculate the mean (i.e., the mean of the target reward) and the standard deviation (i.e., the standard deviation of the target reward), and then use... Subtracting the mean (i.e., the mean of the target reward) and dividing by the standard deviation (i.e., the standard deviation of the target reward) yields the micro-temporal advantage for the corresponding i and k parameters. The formula for the aforementioned micro-temporal advantage can be expressed as:

[0077] .

[0078] Calculate the final consensus advantage ( The final advantage function is formed by averaging the macro-time series advantage and the micro-time series advantage. (i.e., the aforementioned consensus advantage), the formula is:

[0079] .

[0080] Optionally, the method, wherein step S40 includes:

[0081] Define the optimization strategy model for the multimodal model to be trained;

[0082] Obtain a first probability of generating the corresponding response by inputting the sample feature slice into the optimization strategy model, and a second probability of generating the corresponding response by inputting the sample feature slice into the multimodal model to be trained;

[0083] Based on the first probability, the second probability, and the evaluation parameters corresponding to each response, obtain the loss term corresponding to each response;

[0084] The objective function is obtained by summing the loss terms corresponding to the responses of all the sample feature slices in the training dataset.

[0085] The updated parameters are obtained when the objective function meets the constraints.

[0086] In this embodiment, the consensus advantage is adopted. The evaluation parameters, i.e., the policy model (i.e., the multimodal model to be trained), are updated using a policy optimization algorithm. Definition It is the old policy model, that is, the policy model before the parameters are updated by the policy optimization algorithm (i.e., the multimodal model to be trained). For the new strategy model we want to optimize (i.e., the optimization strategy model), the purpose of step S40 is to... Optimize in the direction we need. For each time resolution video... (i.e., the sample feature slices), and each candidate inference branch i at the corresponding time resolution. From The i-th output sampled (i.e., the response). Representing the new policy model (i.e., the optimization strategy model) input The situation arises The probability of the response (i.e., the first probability). The input represents the old policy model (i.e., the multimodal model to be trained). The situation arises The probability of the response (i.e., the second probability). As the cutoff factor, clip is a cutoff function that guarantees... / The value falls on arrive Between. We make (That is, the loss term) is:

[0087] ;

[0088] in, It is under the parameters k and i Through the advantage function This guides the optimization direction of the policy model (i.e., the multimodal model to be trained), and sets all values ​​under i and k. (i.e., the loss terms) are accumulated and divided by n and K to obtain the final objective function:

[0089] .

[0090] The optimization and parameter update of the multimodal model to be trained are achieved by maximizing the objective function through an optimizer, and the updated parameters are obtained. Furthermore, the constraints can be set according to actual needs, such as to reach a certain number of iterations. Figure 3As shown, the standard deviation of the reward curve (accuracyreward / std) obtained by the Hard-ThresholdClipped Proximal Policy Optimization (HTC-PO) algorithm in this embodiment of the invention is lower than the standard deviation of the reward curve obtained by the Group Relative Policy Optimization (GPRO) algorithm. Figure 4 As shown, the mean reward curve (accuracy reward / mean) obtained by HTC-PO in this embodiment of the invention is higher than that obtained by GPRO. Because HTC-PO is exposed to multiple temporal views during training, the model learns to be less sensitive to frame rate changes, learning the essential content of events rather than their appearance at a specific sampling rate. This makes the trained video inference model more temporally robust, exhibiting better generalization performance when faced with unknown videos with varying frame rates, motion speeds, and event densities. In contrast, the GRPO model is trained at a single fixed frame rate, and its performance may decline when encountering videos with significantly different temporal characteristics from the training data.

[0091] Optionally, the method further includes:

[0092] Determine a pre-trained multimodal base model;

[0093] Constructing multimodal thinking chain reasoning data;

[0094] The multimodal basic model is fine-tuned using the multimodal thinking chain inference data to obtain the multimodal model to be trained.

[0095] In this embodiment, a pre-trained multimodal base model is selected. A supervised cold start training is performed on the model using a small amount of pre-constructed multimodal thought chain reasoning data, allowing the model to initially become accustomed to chain-like reasoning paths based on visual information. In one implementation, the pre-trained multimodal base model is Qwen2.5-7B, and the multimodal thought chain reasoning data is constructed by extracting 3000 entries from the Video-R1-CoT-165k dataset. This multimodal thought chain reasoning data includes image-text thought chain data and video-text thought chain data. Supervised fine-tuning training methods include full fine-tuning, low-rank matrix optimizer (Lora) fine-tuning, and layerwise importance sampling (Lisa) fine-tuning. The multimodal base model is fine-tuned using the multimodal thought chain reasoning data to obtain the multimodal model to be trained.

[0096] Optionally, the method, wherein step S10 includes:

[0097] Multiple first sampling start points are determined in the original training video based on the sampling step size corresponding to multiple time resolutions; wherein, the first sampling start point is the start point corresponding to the sample feature slice in the original training video;

[0098] Based on a preset random distribution function, generate multiple random offsets;

[0099] Using the second sampling starting point as a reference, the original training video is extracted to obtain the sample feature slices according to the sampling step size; wherein, the second sampling starting point is the sampling starting point obtained by offsetting the first sampling starting point according to the random offset.

[0100] In this embodiment, a raw video is received. (i.e., the original training videos), and convert them into a video set containing K different temporal resolutions (i.e., the training dataset), i.e., the temporal pyramid. Each Corresponding to a different target frame rate The specific implementation of the time sequence pyramid construction consists of the following two sub-steps:

[0101] (1) Uniform temporal downsampling: using a fixed sampling step size Frames are extracted from the original training video to create a baseline time-series view, i.e., to determine the first sampling start point.

[0102] (2) Random timing jitter: During the training phase, in order to enhance the robustness of the model, a random offset is introduced in each sampling window, which forces the model to understand the content of the event rather than relying on its fixed position in the window. That is, a random offset is added to the first sampling start point to obtain the second sampling start point, so that the position difference between each second sampling start point and the corresponding first sampling start point is random.

[0103] The following is a specific first embodiment of the present invention. The first embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and processes, but the protection scope of the present invention is not limited to the first embodiment.

[0104] In the specific process of the first embodiment, firstly, based on Qwen2.5-VL-7B, 2000 data points are extracted from the Video-R1-260K dataset for a full-scale model fine-tuning, and the fine-tuned model is used as the policy model. Then, all videos are uniformly downsampled to 64 frames, 32 frames, and 16 frames respectively. On this basis, 0-1 seconds of random jitter is added to each frame to construct video temporal pyramids at different temporal resolutions (i.e., the training dataset). During the reinforcement learning training process, 1000 data points are sampled from the Compositional Language and Elementary Visual Reasoning (CLEVER), Next Phase of Question-Answering to Explaining Temporal Actions (NeXT-QA), and Satellite image scene graph dataset (STAR) training sets, for a total of 3000 data points for model training. Based on the policy model (i.e., the multimodal model to be trained), macroscopic output responses of videos at different temporal resolutions are obtained. Then, for responses at the same temporal resolution, eight microscopic output responses are sampled with a temperature coefficient of 1.0. For all macroscopic and microscopic output responses, a reward function combining format reward and precision reward is used to obtain their reward values. Based on these reward values, macroscopic temporal advantage and microscopic temporal advantage are calculated, and the consensus advantage is obtained by combining the two advantages. The policy is updated based on the consensus advantage, and the entire 3000 data points are iteratively trained through reinforcement learning in one round to finally obtain a video inference model with reasoning capabilities.

[0105] This invention constructs a temporal pyramid by extracting sample feature slices at different temporal resolutions from videos, generating responses at multiple time scales and forming diverse reward values ​​at different time scales. It constructs a macro-temporal advantage by integrating reward values ​​at different time scales and rewards from different inference paths to reward global logical consistency and achieve cross-scale consensus. For the same inference path, reward values ​​at different time scales construct a micro-temporal advantage to reward the precise use of local evidence and achieve intra-scale fine-tuning. By combining macro-temporal and micro-temporal advantages to form a unified and balanced hierarchical consensus advantage, and combining this with reinforcement learning training strategies, the model is driven to integrate multi-scale temporal information to achieve better parameter optimization.

[0106] The advantages of the embodiments of the present invention are as follows:

[0107] 1. Address the root cause of "perceptual bias" in video reasoning:

[0108] This is HTC-PO's core advantage. Existing models' understanding of video heavily relies on a fixed temporal resolution, which leads to "perceptual bias."

[0109] Sparse views (low frame rate): cause the model to form only superficial assumptions and lack detailed evidence.

[0110] Dense view (high frame rate): causes the model to be overwhelmed by details, making it impossible to abstract the main narrative.

[0111] Methods like GRPO optimize subsequent thought processes based on a single, potentially biased view. If this initial view is flawed (e.g., it misses keyframes), then even the best subsequent inference optimizations may be built on an incorrect or incomplete foundation.

[0112] In this embodiment of the invention, HTC-PO constructs a temporal pyramid, forcing the model to simultaneously confront multiple perspectives and seek consensus among them. This forces the model to learn to integrate information from different time scales, thereby constructing a more comprehensive and robust internal representation, fundamentally overcoming the limitations of a single perspective.

[0113] 2. Provide richer and more meaningful hierarchical monitoring signals:

[0114] GRPO's supervision signal is relatively simple: "how much better is this answer than the other answers in the same group?" HTC-PO, on the other hand, designs a more complex Hierarchical Consensus Advantage mechanism, providing two levels of supervision signals:

[0115] Macro-level Advantage: This evaluates the global consistency of an answer across all temporal scales. This signal guides the model to learn general logical knowledge across resolutions, ensuring that conclusions do not contradict themselves when the frame rate changes.

[0116] Micro-level advantage: This assesses the strength of evidence supporting an answer at its current scale. This signal encourages models to refine their exploration of visual evidence at each scale, ensuring that reasoning is "substantiated."

[0117] This kind of supervisory signal, which combines "macro logic" and "micro evidence," is much richer than a single "relatively good or bad" signal. It can more effectively guide the model to learn complex reasoning that has both a global perspective and detailed insights.

[0118] 3. Significantly improves the model's temporal robustness and generalization ability:

[0119] Because the HTC-PO was exposed to multiple temporal views during training, the model learned to be less sensitive to changes in frame rate. It learned the essential content of events, rather than their appearance at a specific sampling rate. This made the trained model more temporally robust, exhibiting better generalization performance when faced with unknown videos with varying frame rates, motion speeds, and event densities.

[0120] In contrast, the GRPO model, trained at a single fixed frame rate, may experience performance degradation when encountering videos with temporal characteristics significantly different from the training data. Methods like GRPO focus on choosing a single path (single view) and then working on improving the driving experience (optimizing the thought process). In contrast, the HTC-PO used in this invention observes multiple paths simultaneously (multiple temporal views) and, through comparative analysis, identifies the most reliable path that truly leads to the destination (correct understanding). Therefore, the technical advantage of HTC-PO lies in its ability to go beyond simply optimizing the "reasoning process" and delve into the core of "video understanding." By addressing the fusion and consistency of multi-scale temporal information, it establishes a more solid and reliable foundation for subsequent complex reasoning.

[0121] like Figure 5 As shown, to achieve the above objectives, embodiments of the present invention provide a model training apparatus, comprising:

[0122] The first acquisition module 501 is used to extract multiple sample feature slices with different temporal resolutions from at least one original training video to obtain a training dataset.

[0123] The second acquisition module 502 is used to input the training dataset into the multimodal model to be trained and acquire multiple responses corresponding to each of the sample feature slices; wherein, the multimodal model to be trained is used to analyze the sample feature slices and obtain the inference results of the sample feature slices; the response includes the inference results;

[0124] The third acquisition module 503 is used to evaluate the response using a reward function and acquire the evaluation parameters corresponding to each response;

[0125] The fourth acquisition module 504 is used to acquire the update parameters of the multimodal model to be trained based on the evaluation parameters; wherein the update parameters are model parameters that enable the objective function corresponding to the multimodal model to be trained to meet the constraint conditions.

[0126] The fifth acquisition module 505 is used to adjust the multimodal model to be trained using the updated parameters to acquire the video inference model.

[0127] Optionally, in the aforementioned apparatus, the third acquisition module 503 includes:

[0128] The first determining unit is used to determine the scalar reward value corresponding to each of the responses using the reward function;

[0129] The first acquisition unit is configured to acquire the consensus advantage of a response based on the scalar reward value corresponding to each response; wherein the evaluation parameters include the consensus advantage.

[0130] Optionally, in the apparatus, the first acquiring unit includes:

[0131] The first acquisition component is used to acquire the macroscopic temporal advantage corresponding to each response based on the scalar reward value, the mean reward, and the standard deviation of the reward corresponding to each response; wherein the mean reward is the average of the scalar reward values ​​corresponding to all the sample feature slices in the training dataset; and the standard deviation of the reward is the standard deviation of the scalar reward values ​​corresponding to all the sample feature slices in the training dataset.

[0132] The second acquisition component is used to acquire the micro-temporal advantage corresponding to each response based on the scalar reward value, the target reward mean, and the target reward standard deviation value corresponding to each response; wherein, the target reward mean is the average of the scalar reward values ​​corresponding to all sample feature slices in the training dataset that have the same temporal resolution as the sample feature slice corresponding to the response; and the target reward standard deviation is the standard deviation of the scalar reward values ​​corresponding to all sample feature slices in the training dataset that have the same temporal resolution as the sample feature slice corresponding to the response.

[0133] The third acquisition component is used to acquire the consensus advantage of the response based on the macro-time advantage and the micro-time advantage corresponding to each response.

[0134] Optionally, in the aforementioned apparatus, the fourth acquisition module 504 includes:

[0135] The first processing unit is used to define the optimization strategy model of the multimodal model to be trained;

[0136] The second acquisition unit is used to acquire a first probability of generating the corresponding response by inputting the sample feature slice into the optimization strategy model, and a second probability of generating the corresponding response by inputting the sample feature slice into the multimodal model to be trained.

[0137] The third acquisition unit is used to acquire the advantage function corresponding to each response based on the first probability, the second probability and the evaluation parameters corresponding to each response;

[0138] The fourth acquisition unit is used to accumulate the advantage functions corresponding to the responses of all the sample feature slices in the training dataset to obtain the target function;

[0139] The fifth acquisition unit is used to acquire the updated parameters when the objective function meets the constraint conditions.

[0140] Optionally, the device further includes:

[0141] The first determination module is used to determine a pre-trained multimodal base large model;

[0142] The first processing module is used to construct multimodal thought chain reasoning data;

[0143] The sixth acquisition module is used to fine-tune the multimodal basic large model through the multimodal thinking chain inference data to acquire the multimodal model to be trained.

[0144] Optionally, in the aforementioned apparatus, the first acquisition module 501 includes:

[0145] The second determining unit is used to determine multiple first sampling starting points in the original training video according to the sampling step size corresponding to multiple time resolutions; wherein, the first sampling starting point is the starting point corresponding to the sample feature slice in the original training video.

[0146] The first generation unit is used to generate multiple random offsets according to a preset random distribution function;

[0147] The sixth acquisition unit is used to extract the sample feature slices from the original training video according to the sampling step size, with the second sampling starting point as a reference; wherein, the second sampling starting point is the sampling starting point obtained by offsetting the first sampling starting point according to the random offset.

[0148] It should be noted that the apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0149] To achieve the above objectives, embodiments of the present invention provide a network device, including: a processor, a memory, and a program or instructions stored in the memory and executable on the processor; wherein, when the processor executes the program or instructions, it implements the model training method described above.

[0150] To achieve the above objectives, embodiments of the present invention provide a readable storage medium having a program or instructions stored thereon, wherein the program or instructions, when executed by a processor, implement the steps in the model training method described above.

[0151] To achieve the above objectives, embodiments of the present invention provide a computer program product, which includes computer instructions that, when executed by a processor, implement the steps of the model training method described above.

[0152] It should be further noted that the terminals described in this specification include, but are not limited to, smartphones, tablets, etc., and many of the functional components described are referred to as modules in order to emphasize the independence of their implementation.

[0153] In this embodiment of the invention, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0154] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0155] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0156] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of the invention. Therefore, the invention should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention complete and convey the scope of the invention to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0157] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A model training method, characterized in that, include: Extract multiple sample feature slices with different temporal resolutions from at least one original training video to obtain the training dataset; The training dataset is input into the multimodal model to be trained to obtain multiple responses corresponding to each of the sample feature slices; wherein, the multimodal model to be trained is used to analyze the sample feature slices to obtain the inference results of the sample feature slices; the response includes the inference results; The response is evaluated using a reward function to obtain the evaluation parameters corresponding to each response; Based on the evaluation parameters, the updated parameters of the multimodal model to be trained are obtained; wherein, the updated parameters are model parameters that enable the objective function corresponding to the multimodal model to be trained to meet the constraint conditions; The updated parameters are used to adjust the multimodal model to be trained to obtain a video inference model.

2. The method according to claim 1, characterized in that, The response is evaluated using a reward function to obtain evaluation parameters for each response, including: The scalar reward value corresponding to each of the responses is determined using the reward function. The consensus advantage of a response is obtained based on the scalar reward value corresponding to each response; wherein the evaluation parameters include the consensus advantage.

3. The method according to claim 2, characterized in that, Based on the scalar reward value corresponding to each of the responses, the consensus advantage of the responses is obtained, including: Based on the scalar reward value, mean reward, and standard deviation of reward corresponding to each response, the macroscopic temporal advantage corresponding to each response is obtained; wherein, the mean reward is the average of the scalar reward values ​​corresponding to all the sample feature slices in the training dataset; and the standard deviation of reward is the standard deviation of the scalar reward values ​​corresponding to all the sample feature slices in the training dataset. The micro-temporal advantage corresponding to each response is obtained based on the scalar reward value, the mean target reward, and the standard deviation of the target reward corresponding to each response; wherein, the mean target reward is the average of the scalar reward values ​​corresponding to all sample feature slices in the training dataset that have the same temporal resolution as the sample feature slice corresponding to the response; and the standard deviation of the target reward is the standard deviation of the scalar reward values ​​corresponding to all sample feature slices in the training dataset that have the same temporal resolution as the sample feature slice corresponding to the response. The consensus advantage of a response is obtained based on the macro-time advantage and the micro-time advantage corresponding to each response.

4. The method according to claim 1, characterized in that, Based on the evaluation parameters, the updated parameters of the multimodal model to be trained are obtained, including: Define the optimization strategy model for the multimodal model to be trained; Obtain a first probability of generating the corresponding response by inputting the sample feature slice into the optimization strategy model, and a second probability of generating the corresponding response by inputting the sample feature slice into the multimodal model to be trained; Based on the first probability, the second probability, and the evaluation parameters corresponding to each response, obtain the loss term corresponding to each response; The objective function is obtained by summing the loss terms corresponding to the responses of all the sample feature slices in the training dataset. The updated parameters are obtained when the objective function meets the constraints.

5. The method according to claim 1, characterized in that, The method further includes: Determine a pre-trained multimodal base model; Constructing multimodal thinking chain reasoning data; The multimodal basic model is fine-tuned using the multimodal thinking chain inference data to obtain the multimodal model to be trained.

6. The method according to claim 1, characterized in that, Extract multiple sample feature slices with different temporal resolutions from at least one original training video to obtain the training dataset, including: Multiple first sampling start points are determined in the original training video based on the sampling step size corresponding to multiple time resolutions; wherein, the first sampling start point is the start point corresponding to the sample feature slice in the original training video; Based on a preset random distribution function, generate multiple random offsets; Using the second sampling starting point as a reference, the original training video is extracted to obtain the sample feature slices according to the sampling step size; wherein, the second sampling starting point is the sampling starting point obtained by offsetting the first sampling starting point according to the random offset.

7. A model training device, characterized in that, include: The first acquisition module is used to extract multiple sample feature slices with different temporal resolutions from at least one original training video to obtain the training dataset. The second acquisition module is used to input the training dataset into the multimodal model to be trained, and acquire multiple responses corresponding to each of the sample feature slices; wherein, the multimodal model to be trained is used to analyze the sample feature slices and obtain the inference results of the sample feature slices; the response includes the inference results; The third acquisition module is used to evaluate the response using a reward function and acquire the evaluation parameters corresponding to each response; The fourth acquisition module is used to acquire the update parameters of the multimodal model to be trained based on the evaluation parameters; wherein the update parameters are model parameters that enable the objective function corresponding to the multimodal model to be trained to meet the constraint conditions. The fifth acquisition module is used to adjust the multimodal model to be trained using the updated parameters to acquire the video inference model.

8. A network device, comprising: A processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the model training method as described in any one of claims 1-6.

9. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps in the model training method as described in any one of claims 1-6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the model training method as described in any one of claims 1-6.

Images