Sample generation, model training, video question answering methods, devices, intelligent agents, equipment, media and products
By selecting target and interference sample segments during the training of the long video question answering model, the problems of unanswerable and difficulty mismatch in model training are solved, thereby improving the model's self-evolution ability and the accuracy of answer generation.
Patent Information
- Application Number
- CN202611467742.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-20
- Publication Date
- 2026-11-10
AI Technical Summary
Long video question answering models suffer from problems during training, such as being unanswerable, ambiguous, relying on prior language knowledge, and having a mismatch between difficulty and ability, which limits their ability to continuously evolve.
Training samples are generated by identifying target and interference samples from multiple candidate sample segments of a sample video based on guidance information. The target sample segments contain real evidence, while the interference sample segments semantically match them but do not contain the answer. These samples are used to train the retrieval and answering modules of the model.
It improves the continuous self-evolution capability of the long video question answering model during training, and enhances the accuracy and interpretability of the model in evidence retrieval and answer generation.
Smart Images

Figure CN123068964A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision, deep learning, and large models, and more specifically, to a sample generation, model training, video question answering method, apparatus, intelligent agent, device, medium, and product. Background Technology
[0002] Long videos (such as videos of tens of minutes to several hours) contain a large amount of temporally dispersed, multimodal data. Therefore, long video understanding and question answering tasks require models to retrieve evidence and reason about answers from information spanning long periods of time and multiple modalities. Summary of the Invention
[0003] In view of this, this disclosure provides a sample generation, model training, video question answering method, apparatus, agent, device, medium, and product.
[0004] According to one aspect of this disclosure, a sample generation method is provided, comprising: determining a target sample segment for training in the current round from a plurality of candidate sample segments of a sample video based on guidance information for training in the current round, wherein the guidance information indicates the capability dimension that the model obtained in the previous round of training needs to be adjusted in the current round of training; determining interference sample segments from a plurality of candidate sample segments based on a sample question and a sample answer obtained based on the target sample segment, wherein the interference sample segments semantically match the target sample segment but not semantically match the sample answer; and using the sample question, the sample answer, the target sample segment, and the interference sample segments as training samples for training in the current round.
[0005] According to another aspect of this disclosure, a model training method is provided, comprising: obtaining training samples for the current training round, wherein the training samples are obtained using a sample generation method, and the training samples include sample questions, sample answers, target sample fragments, and interference sample fragments; training a retrieval module in the model obtained in the previous training round using the training samples to obtain a trained retrieval module; and, while keeping the model parameters of the trained retrieval module unchanged, training an answer module in the model obtained in the previous training round using the sample questions, the evidence fragments, and the sample answers to obtain a model obtained in the current training round.
[0006] According to another aspect of this disclosure, a video question answering method is provided, comprising: in response to receiving a question to be processed for a target video, inputting the target video and the question to be processed into a retrieval module in a trained model to obtain a target segment, wherein the target video includes multiple candidate segments, and the trained model is obtained using a video question answering method; and inputting the target question and the target segment into an answering module in the trained model to obtain the target answer.
[0007] According to another aspect of this disclosure, a sample generation apparatus is provided, comprising: a first determining module, configured to determine a target sample segment for training in the current round from a plurality of candidate sample segments of a sample video based on guidance information for training in the current round, wherein the guidance information indicates the capability dimension that the model obtained in the previous round of training needs to be adjusted in the current round of training; a second determining module, configured to determine interference sample segments from the plurality of candidate sample segments based on a sample question and a sample answer obtained based on the target sample segment, wherein the interference sample segments semantically match the target sample segment but not semantically match the sample answer; and a sample generation module, configured to use the sample question, the sample answer, the target sample segment, and the interference sample segments as training samples for training in the current round.
[0008] According to another aspect of this disclosure, a model training apparatus is provided, comprising: an acquisition module for acquiring training samples for the current training round, wherein the training samples are obtained using a sample generation device, and the training samples include sample questions, sample answers, target sample fragments, and interference sample fragments; a first training module for training a retrieval module in a model obtained in the previous training round using the training samples, thereby obtaining a trained retrieval module; and a second training module for training an answer module in the model obtained in the previous training round using the sample questions, the evidence fragments, and the sample answers, while keeping the model parameters of the trained retrieval module unchanged, thereby obtaining a model obtained in the current training round.
[0009] According to another aspect of this disclosure, a video question-answering apparatus is provided, comprising: a first question-answering module, configured to, in response to receiving a question to be processed for a target video, input the target video and the question to be processed into a retrieval module in a trained model to obtain a target segment, wherein the target video includes multiple candidate segments, and the trained model is obtained using a model training device; and a second question-answering module, configured to input the target question and the target segment into an answering module in the trained model to obtain the target answer.
[0010] According to another aspect of this disclosure, a large-model-based intelligent agent is provided, comprising: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a target large model based on the target task, and obtaining output information by calling the target large model to execute the above method; and an output module for outputting the output information obtained by the processing module.
[0011] According to another aspect of this disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.
[0012] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program or instructions thereon, which, when executed by a processor, implement the steps of the above-described method.
[0013] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the steps of the described method.
[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0015] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0016] Figure 1 This illustration schematically shows a system architecture for applying sample generation, model training, and video question answering methods according to embodiments of the present disclosure;
[0017] Figure 2 A flowchart illustrating a sample generation method according to an embodiment of the present disclosure is shown schematically;
[0018] Figure 3 This illustration schematically shows an example diagram of a process for segmenting a sample video according to an embodiment of the present disclosure;
[0019] Figure 4 This illustration schematically shows an example diagram of a process for obtaining guidance information for the current training round according to an embodiment of the present disclosure;
[0020] Figure 5This illustration schematically depicts an example of a process for determining interfering sample fragments from multiple candidate sample fragments based on sample questions and sample answers obtained from target sample fragments, according to an embodiment of the present disclosure.
[0021] Figure 6 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown schematically.
[0022] Figure 7 This illustration schematically shows an example diagram of a model training process according to an embodiment of the present disclosure;
[0023] Figure 8 A flowchart illustrating a video question-and-answer method according to an embodiment of the present disclosure is shown schematically;
[0024] Figure 9 This illustration schematically shows an example diagram of a video question-and-answer process according to an embodiment of the present disclosure;
[0025] Figure 10 A block diagram of a sample generation apparatus according to an embodiment of the present disclosure is shown schematically;
[0026] Figure 11 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown schematically;
[0027] Figure 12 A block diagram of a video question-and-answer device according to an embodiment of the present disclosure is shown schematically;
[0028] Figure 13 A schematic diagram illustrating the structure of a large-model-based intelligent agent according to embodiments of the present disclosure; and
[0029] Figure 14 A block diagram of an electronic device suitable for implementing sample generation, model training, and video question answering methods according to embodiments of the present disclosure is illustrated. Detailed Implementation
[0030] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.
[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0032] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0033] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).
[0034] Long-form video question answering methods can include learnable video retrieval methods, course-based self-evolving video understanding methods, and time-oriented reasoning self-evolving methods.
[0035] Learnable video retrieval methods involve retrieving relevant video segments from a long video based on an input question, and then inputting the corresponding visual tokens into a language model to provide an answer. Curriculum-based self-evolving video understanding methods utilize the model's current capability estimation to adjust the difficulty of self-generated question-and-answer tasks, gradually increasing the complexity of the training task across dimensions such as perception, recognition, and understanding. Temporal reasoning-oriented self-evolving methods generate training tasks through a Questioner-Solver and leverage rewards such as temporal perturbation sensitivity and temporal localization to enhance the video model's temporal reasoning capabilities.
[0036] However, the above methods all follow the order of finding evidence based on the question and then generating the answer. This results in problems such as unanswerable, ambiguous, dependent on language priors, and mismatch between difficulty and ability in the generated training samples, which restricts the continuous self-evolution ability of long video question answering models.
[0037] To address this, this disclosure proposes a sample generation scheme. For example, based on guidance information for the current training round, a target sample segment for the current training round is determined from multiple candidate sample segments of the sample video, wherein the guidance information indicates the capability dimension that the model obtained in the previous training round needs to adjust in the current training round; based on the sample question and sample answer obtained based on the target sample segment, interference sample segments are determined from multiple candidate sample segments, wherein the interference sample segments semantically match the target sample segment but not semantically match the sample answer; the sample question, sample answer, target sample segment, and interference sample segments are used as training samples for the current training round.
[0038] According to embodiments of this disclosure, since the selection of target sample segments is based on guiding information used for the current training round, evidence can be selected focusing on capability dimensions that the model obtained from the previous training round has not yet fully mastered and are at the capability boundary. Because both the sample question and sample answer are derived from real target sample segments, and the interference sample segments match the target sample segments in content but do not contain content used to obtain the sample answer, the answerability of the samples and the certainty of the answers are improved. This allows the model to learn which segments truly contain the evidence needed to answer the question during subsequent training. Based on this, by combining sample questions, sample answers, target sample segments, and interference sample segments into training samples for the current training round, the continuous self-evolution capability of subsequent long-video question-answering model training is enhanced.
[0039] In the embodiments disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information and any other types of information comply with relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solutions of this disclosure, user authorization or consent is obtained before acquiring or collecting user personal information and any other types of information.
[0040] Figure 1 The illustration schematically depicts a system architecture according to embodiments of the present disclosure where sample generation, model training, and video question answering methods can be applied. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.
[0041] like Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0042] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).
[0043] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0044] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.
[0045] It should be noted that the sample generation, model training, and video question answering methods provided in this disclosure embodiment can generally be executed by the server 105. Accordingly, the sample generation, model training, and video question answering devices provided in this disclosure embodiment can generally be located in the server 105.
[0046] Alternatively, the sample generation, model training, and video question answering methods provided in this disclosure can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the sample generation, model training, and video question answering apparatus provided in this disclosure can also be located in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.
[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0048] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.
[0049] Figure 2 A flowchart illustrating a sample generation method according to an embodiment of the present disclosure is shown schematically.
[0050] like Figure 2 As shown, the sample generation method 200 may include operations S210 to S230.
[0051] In operation S210, based on the guidance information used for the current training round, the target sample segment for the current training round is determined from multiple candidate sample segments of the sample video. The guidance information indicates the capability dimension that the model obtained from the previous training round needs to be adjusted in the current training round.
[0052] In operation S220, based on the sample question and sample answer obtained from the target sample fragment, interference sample fragments are determined from multiple candidate sample fragments, wherein the interference sample fragments are semantically matched with the target sample fragments but not semantically matched with the sample answers.
[0053] In operation S230, the sample question, sample answer, target sample fragment, and interference sample fragment are used as training samples for the current round of training.
[0054] The sample video is the original long video V used to generate training samples. It is a long video that has not been manually labeled line by line and can be surveillance footage, home life video, meeting / live broadcast video, movie clips, etc. For example, a sample video could be a 30-minute home life video recording a series of events such as people entering the kitchen, retrieving items, pouring water, and entering and leaving rooms. Candidate sample segments are several video clips obtained by segmenting the sample video, which can be selected and filtered later. For example, a 30-minute video can be segmented into hundreds of candidate sample segments of about 5 seconds each.
[0055] The guiding information, provided before each round of self-evolutionary training, indicates what types of samples should be prioritized for generation. It originates from the capability evaluation results of the model obtained from the previous training round. It's important to note that the guiding information doesn't directly specify a particular segment; rather, it indicates which capability dimensions need adjustment. Based on this, segments matching these weaker dimensions are selected from the candidate sample segments as target sample segments, ensuring that the generated training task falls on the learning frontier of the model obtained from the previous training round.
[0056] The target sample fragment is a fragment that truly contains the evidence needed to answer the question, determined from multiple candidate sample fragments based on guiding information. The target sample fragment can be a single candidate sample fragment or a set of multiple discontinuous candidate sample fragments. For example, suppose the evaluation found that the previous model achieved only 30% accuracy on counting questions but 95% accuracy on single-fragment attribute recognition; then the guiding information indicates that counting is the capability dimension that needs adjustment in the current round. Therefore, fragments involving this type of information can be prioritized accordingly.
[0057] Once the target sample fragment is obtained, a sample question and a sample answer can be generated from the target sample fragment. Then, using the sample question and sample answer as a reference, interfering sample fragments can be selected from the candidate sample fragments. Interfering sample fragments must meet the requirements of semantic matching with the target sample fragment and semantic non-match with the sample answer.
[0058] The sample question is a natural language question constructed by reverse engineering the semantic information of the target sample fragment. That is, the evidence precedes the question, rather than creating a question out of thin air and then finding evidence to support it. The sample answer is a standard answer extracted from the target sample fragment that uniquely corresponds to the question. For example, based on the target sample fragment "The man took milk from the refrigerator," the sample question "After entering the kitchen, what did the man take from the refrigerator?" would be "milk."
[0059] Interference sample segments are those that are semantically similar to the target sample segment but do not semantically match the sample answer and do not contain evidence supporting the answer. In other words, they are similar to the target sample segment in terms of characters, scenes, actions, or text, but cannot truly support the standard answer. For example, for the question "What did the man take out of the refrigerator after entering the kitchen?" and the answer "milk", interference sample segments could be 05:21–05:30 "The same man stood next to the refrigerator but took out mineral water" or 18:21–18:30 "Another person took out juice from the refrigerator".
[0060] For example, from adjacent time periods, similar people, similar actions, or semantically similar segments, segments that have been verified not to contain evidence of the answer can be selected as interference sample segments. This makes the interference sample segments similar to the target sample segments in terms of people, scenes, actions, or textual semantics, but they cannot support the correct answer.
[0061] Alternatively, a visual / text dual-tower model can be used to calculate the semantic similarity between each candidate segment and the target sample segment and sort them in descending order; then, a teacher model can be used to determine whether each candidate segment can support the sample answer; segments with high similarity but determined not to support the sample answer can be taken as interference sample segments.
[0062] Alternatively, the target sample fragment itself can be subject to controlled perturbations, such as replacing key entities, obscuring answer-related content, scrambling the action time sequence, or replacing local ASR text, to generate variant fragments that are highly similar to the target fragment but whose semantic information is destroyed and cannot support the sample answer, thus serving as interference sample fragments.
[0063] Training samples are complete samples composed of four elements: sample question, sample answer, target sample fragment, and interference sample fragment. They are used for training in the current round. For example, training sample = {sample question "What did the man take out of the refrigerator after entering the kitchen?", sample answer "milk", target sample fragment [10:30, 10:40], interference sample fragment [05:21, 05:30]}.
[0064] In the embodiments of this disclosure, since the selection of target sample segments is based on guiding information used for the current training round, evidence can be selected focusing on capability dimensions that the model obtained from the previous training round has not yet fully mastered and are at the capability boundary. Because both the sample question and sample answer are derived from real target sample segments, and the interference sample segments match the target sample segments in content but do not contain content used to obtain the sample answer, the answerability of the samples and the certainty of the answers are improved. This allows the model to learn which segments truly contain the evidence needed to answer the question during subsequent training. Based on this, by combining the sample question, sample answer, target sample segment, and interference sample segment into training samples for the current training round, the continuous self-evolution capability of the subsequent long video question answering model training is enhanced.
[0065] Figure 3 The illustration shows an example diagram of a process for segmenting a sample video according to an embodiment of the present disclosure.
[0066] like Figure 3 As shown, in embodiment 300 of the sample video segmentation, sample video 310 can be divided into multiple candidate sample segments according to a preset method. The segmentation method can be configured according to actual business needs and is not limited here. For example, segmentation by shot, fixed-length segmentation, or variable-length segmentation, etc. The following explanation uses fixed-length segmentation as an example.
[0067] The partitioning can include two stages: a first stage of partitioning into Blocks and a second stage of partitioning into Clips. For example, taking the intervals 0~t1, t1~t2, ..., t5~t6 as all 't', in the first stage, the sample video 310 can be partitioned into video block 1 (0~t3) and video block 2 (t3~t6) according to 3t. In the second stage, for video block 1, it can be further partitioned into segment 1, segment 2, and segment 3 according to 't', and for video block 2, it can be further partitioned into segment 4, segment 5, and segment 6 according to 't', thus forming a hierarchical structure of video -> video block (Block) -> segment (Clip).
[0068] According to embodiments of this disclosure, the guidance information for the current training round can be obtained by: dividing the validation sample set according to multiple capability dimensions to obtain sub-validation sample sets for each capability dimension; processing each sub-validation sample set by calling the model and reference model obtained in the previous training round to obtain the evaluation results for each capability dimension, wherein the parameter size of the reference model is the same as the parameter size of the model obtained in the previous training round; and determining the guidance information based on the evaluation results for each capability dimension.
[0069] The validation set is a set of samples used to evaluate the model's capabilities. It includes questions and their corresponding videos, standard answers, and standard evidence. The validation set does not participate in updating the model's training parameters; it is only used to measure the model's current capability level.
[0070] Capability dimensions are used to characterize the capabilities of long-video understanding models, and may include, but are not limited to: question types, evidence time span, number of evidence segments, modal combinations, interference similarity, and number of reasoning steps. Question types may include single-segment attribute recognition, Optical Character Recognition (OCR) / Automatic Speech Recognition (ASR) reading, action recognition, sequence identification, state change, counting, multi-event aggregation, cross-segment causal reasoning, cross-modal consistency, and long-term evidence combination.
[0071] For the validation sample set, it can be divided according to capability dimensions, resulting in multiple sub-validation sample sets for each capability dimension. A sub-validation sample set is a subset of samples belonging to a specific capability dimension after the validation sample set has been divided according to those dimensions. Each capability dimension corresponds to a sub-validation sample set, used for independent evaluation of that dimension. For example, all counting problems can be grouped into the counting sub-validation sample set; problems requiring long-distance, multi-evidence combinations can be grouped into the long-term evidence combination sub-validation sample set.
[0072] For example, samples can be statically divided according to preset dimensions to form sub-validation sample sets for each dimension. Alternatively, multi-dimensional features can be extracted for each validation sample, such as question type labels, evidence fragment duration and quantity, whether it is cross-modal, and similarity scores between interference fragments and target fragments. Clustering algorithms can then be used to automatically cluster samples with similar semantics / features into several subsets, and each subset can be labeled with the capability dimension it represents. This is not limited here.
[0073] After obtaining sub-validation sample sets for each of the multiple ability dimensions, both the model trained in the previous round and the reference model can be processed simultaneously for each sub-validation sample set to obtain their performance and the difference between them in each dimension. The reference model is a model with a larger parameter scale than the model trained in the previous round; for example, the reference model could be a teacher model, and the model trained in the previous round could be a student model. Because the reference model is more capable and can provide stable answers across all ability dimensions, it can be used as a benchmark to measure the gap in student abilities.
[0074] The evaluation results are quantitative indicators obtained on each capability dimension after the model processes each sub-validation sample set. For example, the evaluation results may include information such as answer accuracy, evidence IoU, average number of frames viewed, and failure type, as well as the difference in teacher and student capabilities.
[0075] In one embodiment, the model obtained from the previous training round and the reference model can each answer the sub-validation samples, and the accuracy rate of the answers, the IoU of evidence, the average number of frames viewed, the failure type, etc., can be statistically analyzed. The difference between the correct probabilities of the two models is used to constitute the difference in teacher-student ability, thus obtaining the assessment result of the ability dimension. In another embodiment, in addition to comparing the final answer accuracy rate, the two models can be further invoked and their confidence distributions for the answers can be output. The difference between the confidence scores of the two models can be calculated to obtain the assessment result, making the assessment more refined in reflecting the degree of weakness in ability.
[0076] After obtaining the evaluation results, the evaluation results of each ability dimension can be combined to identify dimensions where the student model and teacher model have large gaps and are at the forefront of learning. These dimensions can be used as guiding information output to guide the direction of sample generation in the current round.
[0077] For example, we can prioritize identifying the capability dimensions that need strengthening from samples where the reference model can provide stable answers, where the success rate of the model obtained from the previous training round is within the target learning range or has just failed, and where the evidence quality is high. These dimensions and their difficulty levels can then be used as guiding information. Alternatively, we can sort the differences in teacher-student capabilities across all dimensions from largest to smallest, and select several dimensions whose capability differences exceed a preset threshold or are ranked high, but have not yet reached the target success rate, as guiding information.
[0078] In the embodiments of this disclosure, the validation sample set is divided according to multiple capability dimensions to obtain sub-validation sample sets for each capability dimension. This refines capability assessment from a single overall indicator to independent assessments of each dimension, thereby more accurately reflecting the true capability differences across each capability dimension. By calling the model trained in the previous round and a reference model with a larger parameter scale to process each sub-validation sample set, each capability dimension can simultaneously obtain comparison results between the student model and the reference model. This allows the capability gap in each dimension to be measured against the more capable reference model, enhancing the reliability and discriminative power of the assessment results. Furthermore, by determining guidance information based on the assessment results of each capability dimension, the guidance information accurately points to the capability dimensions where the current model is truly weak and has room for improvement. This guides subsequent training to generate samples at the forefront of learning, improving the targeting and efficiency of the training.
[0079] Figure 4 The illustration shows an example schematic diagram of the process for obtaining guidance information for the current training round according to an embodiment of the present disclosure.
[0080] like Figure 4 As shown, in embodiment 400, which determines the guidance information used for the current training round, the validation sample set 410 can be divided according to multiple capability dimensions to obtain sub-validation sample sets for each capability dimension. For example, sub-validation sample set 411 is obtained by dividing according to capability dimension 1, sub-validation sample set 412 is obtained by dividing according to capability dimension 2, ..., and sub-validation sample set 41N is obtained by dividing according to capability dimension N, where N is a positive integer.
[0081] By calling the model 421 and reference model 422 obtained from the previous training round, each sub-validation sample set is processed to obtain the evaluation results for multiple capability dimensions. Taking the sub-validation sample set 412 of capability dimension 2 as an example, the sub-validation sample set 412 can be distributed and input into the model 421 and reference model 422 obtained from the previous training round to obtain the evaluation result 432 of capability dimension 2. Similarly, the evaluation results 431 of capability dimension 1, ..., and 43N of capability dimension N can be obtained. Based on this, the guidance information 440 can be determined according to the evaluation results of multiple capability dimensions.
[0082] According to an embodiment of this disclosure, operation S220 may include the following operations: generating a sample question and a sample answer based on the first semantic information of the target sample fragment; filtering multiple candidate sample fragments to obtain interfering sample fragments with the goal of reducing at least one of the following: reducing the first difference between the first semantic information and the second semantic information of the candidate sample fragments, increasing the second difference between the sample question and the second semantic information, and increasing the third difference between the sample answer and the second semantic information.
[0083] The first semantic information is the set of semantic elements extracted from the target sample fragment, which may include entities, actions, state changes, temporal order, quantitative relationships, causal relationships, ASR / OCR text clues, etc. For example, for the target sample fragment "a man takes milk out of the refrigerator and pours it into a cup," its first semantic information includes: subject "man," actions "take out, pour in," object "milk," scene "kitchen, refrigerator," and sequential relationship "enter the kitchen → take milk → pour milk," etc. The second semantic information is the set of semantic elements of a candidate sample fragment.
[0084] A natural language question is constructed based on the first semantic information as a sample question, and a standard answer that uniquely corresponds to the sample question is determined as the standard answer. For example, the sample question "What did the man take out of the refrigerator after entering the kitchen?" is constructed based on the first semantic information, and the sample answer corresponding to the above sample question is "milk".
[0085] For example, preset question templates can be used, such as "What did the subject do at the location?" or "What did the subject do in sequence?", to generate sample questions, and the answer can be directly retrieved from the corresponding slot. Alternatively, the first semantic information of the target sample fragment can be used as input, and the multimodal large model can be constrained by prompt words to "generate only one answerable question and its unique answer based on the given fragment," and a self-checking constraint can be set that the answer must be directly corroborated by the visible content in the fragment, thereby generating sample questions and standard answers.
[0086] In one embodiment, the reward function for the learning value of the generated question can be shown in the following formulas (1) and (2).
[0087] rC=λ1G(q)+λ2N(q)+λ3C(q)+λ4QE(q)-λ5A(q); (1)
[0088] C(q)=exp[-(pS(q)-ptarget)2 / (2σ2)]; (2)
[0089] Wherein, rC represents the reward function value; G(q) represents the difference in ability between teachers and students, which is composed of the difference between the correct probability of the reference model and the correct probability of the model obtained from the previous training round; N(q) is semantic novelty, which represents the degree of semantic novelty of question q relative to historical training questions, and is used to encourage the generation of diverse questions; C(q) is the course matching term, which represents the degree of matching between the difficulty of the question and the current course objective (learning frontier); QE(q) is the quality assessment result; A(q) is the penalty term, which represents the degree of ambiguity, unanswerableness or non-uniqueness of the answer to the question, and is used to suppress low-quality questions; λ1~λ5 are weight coefficients; pS(q) represents the estimated success rate of the model obtained from the previous training round on this type of question; ptarget represents the target success rate; σ represents the Gaussian bandwidth parameter, which controls the sensitivity of the course matching function to the success rate deviation.
[0090] The first difference is the difference between the first semantic information of the target sample fragment and the second semantic information of the candidate sample fragment, which reflects the degree of semantic similarity between the two; the second difference is the difference between the sample question and the second semantic information, which reflects the degree of relevance between the content of the candidate fragment and the content asked in the question; the third difference is the difference between the sample answer and the second semantic information, which reflects whether the candidate fragment contains / supports the semantics of the answer.
[0091] For example, the candidate sample fragment "The same man takes mineral water" has a small first difference from the target sample fragment "The man takes milk from the refrigerator and pours it into a cup", a large second difference from the sample question "What did the man take out of the refrigerator after entering the kitchen?", and a large third difference from the sample answer "milk".
[0092] After generating sample questions and sample answers, interference sample fragments can be filtered based on the three differences mentioned above. Interference sample fragments are those that are obtained after filtering, are semantically similar to the target sample fragments, but do not semantically match the sample questions and sample answers, and therefore cannot truly support the sample answers.
[0093] For example, candidates can be selected from adjacent time periods, similar people, similar actions, or semantically similar segments, and then checked to see if they contain evidence of the answer. Alternatives can be selected by using the target segment, sample question, and sample answer as queries, respectively, and the semantic distance between the candidate segment and these three can be calculated using a visual text dual-tower model. A joint ranking function based on the smallest first difference, largest second difference, and largest third difference is used to score the candidate segment, and the highest-scoring candidate segment is selected as the interference sample segment.
[0094] In the embodiments of this disclosure, sample questions and answers are generated based on the first semantic information of the target sample fragment. Both questions and answers are constructed backwards from real-world evidence, making the generated questions answerable and the answers verifiable, thus improving the effectiveness and credibility of the generated training samples. By filtering multiple candidate sample fragments with the goal of reducing at least one of the following: reducing the first difference between the first semantic information and the second semantic information of the candidate sample fragment; increasing the second difference between the sample question and the second semantic information; and increasing the third difference between the sample answer and the second semantic information, the resulting interfering sample fragments are semantically highly similar to the target sample fragment but semantically far removed from the sample question and sample answer. This helps improve the model's accuracy in evidence localization and the robustness of answer determination, and enhances the interpretability of the final question and answer.
[0095] Figure 5 The illustration shows an example schematic diagram of a process for determining interfering sample fragments from multiple candidate sample fragments based on sample questions and sample answers obtained based on target sample fragments, according to an embodiment of the present disclosure.
[0096] like Figure 5 As shown, in embodiment 500 for determining interfering sample segments, a target sample segment 511 for the current training round can be determined from multiple candidate sample segments of the sample video 510 based on the guidance information 520 for the current training round.
[0097] Based on the first semantic information 530 of the target sample fragment 511, a sample question 540 and a sample answer 550 are generated. Multiple candidate sample fragments are screened to obtain interfering sample fragments 570, with the goal of reducing at least one of the following: reducing the first difference between the first semantic information 530 and the second semantic information 560 of the candidate sample fragments; increasing the second difference between the sample question 540 and the second semantic information 560; and increasing the third difference between the sample answer 550 and the second semantic information 560.
[0098] According to an embodiment of this disclosure, operation S240 may include the following operations: performing a quality assessment on the sample question, sample answer, target sample fragment, and interference sample fragment to obtain a quality assessment result; in response to the quality assessment result being passed, using the sample question, sample answer, target sample fragment, and interference sample fragment as training samples for the current round of training.
[0099] Quality assessment refers to the process of performing multidimensional validation on sample pairs to determine whether they possess quality attributes such as answerability, uniqueness of answer, sufficiency and necessity of evidence, and consistency of temporal location. The quality assessment result is a quantitative conclusion output from the quality assessment process. If the quality assessment is passed, the sample pair can be used as training samples and fed into the model training for the current round.
[0100] For example, multiple independent models or different samples / temperatures of the same model can be used to evaluate the quality of sample pairs. The quality can then be determined by combining the outputs of multiple models, thereby reducing the evaluation bias caused by the preference of a single model. Alternatively, a dedicated answerability classifier can be trained to evaluate the quality of sample pairs, and the classifier output can be weighted and fused with the rule verification results to obtain the quality evaluation result.
[0101] In the embodiments of this disclosure, by performing quality assessments on sample questions, sample answers, target sample segments, and interfering sample segments, the correspondence between the sample and the real content and evidence in the video can be verified from different aspects, thereby identifying and filtering out low-quality samples. In response to the passing quality assessment result, the sample questions, sample answers, target sample segments, and interfering sample segments are used as training samples for the current training round. This ensures that all samples entering the training pool undergo quality gating, enabling the model to learn based on high-quality samples. This improves the accuracy and interpretability of the model in evidence retrieval, answer generation, and evidence localization, ultimately enhancing the overall training effect of the long video understanding model.
[0102] According to embodiments of this disclosure, quality assessment of sample questions, sample answers, target sample segments, and interfering sample segments may include at least one of the following: inputting the sample question and target sample segment into a reference model to obtain a first probability that the reference model outputs a sample answer; inputting the sample question into the reference model to obtain a second probability that the reference model outputs a sample answer; inputting the sample question and interfering sample segment into the reference model to obtain a third probability that the reference model outputs a sample answer; and inputting the sample question into the reference model to determine the degree of temporal overlap between the output segment obtained by the reference model and the target sample segment.
[0103] The first probability, second probability, and third probability are the probabilities that the reference model will output a sample answer y under different evidence input conditions. Specifically, the first probability is the probability of outputting a sample answer when the input sample question and the target sample fragment are both input, and it can be used to measure the sufficiency of evidence (Ssuf); the second probability is the probability of outputting a sample answer when only the sample question is input, i.e., the target sample fragment is removed / masked, and it can be used to measure the necessity of evidence (Snec); the third probability is the probability of outputting a sample answer when the input sample question and the interference sample fragment are both input, and it can be used to measure the degree of misleadingness of the interference fragment.
[0104] The higher the first probability, the more likely the target sample fragment is to support the answer; the second probability should be lower than the first probability; the lower the third probability, the less likely the interference fragment will mislead the model to arrive at the answer.
[0105] The output segment is the time segment of evidence that the reference model believes supports the answer when answering a sample question. It can be used for time alignment comparison with the target sample segment. The degree of temporal overlap is the degree of coincidence between the output segment Ê predicted by the reference model and the target sample segment E on the time axis. It can be used to measure the temporal consistency of evidence localization. For example, if the output segment Ê = [10:35, 10:45] and the target sample segment E* = [10:36, 10:44], then their temporal intersection-union ratio (IoU) is close to 0.9, indicating a high degree of temporal overlap.
[0106] Besides intersection-over-union (IoU), various temporal similarity metrics can be used, such as temporal F1 score, L1 distance between start and end times, and the proportion of overlapping frames to the total number of frames. A weighted sum can then be used to determine the degree of temporal overlap, providing a more robust measure of temporal localization consistency. Alternatively, the output segment and the target sample segment can be mapped to fine-grained frame sequences, and the frame-by-frame overlap rate of the two sequences can be calculated as the degree of temporal overlap. This allows for a finer-grained measurement of the accuracy of evidence localization, reducing insensitivity to measurement caused by boundary errors.
[0107] In one embodiment, the quality assessment of the sample question, sample answer, target sample fragment and interference sample fragment can be achieved by the following formulas (3) to (5).
[0108] Ssuf=PA(y*|q,E*);(3)
[0109] Snec=PA(y*|q,E*)-PA(y*|q,E-); (4)
[0110] Sloc=IoU(Ê,E*); (5)
[0111] Where q represents the sample question, y* represents the sample answer, E* represents the target sample fragment, and E- represents the interference sample fragment; Ssuf is the evidence sufficiency score, representing the first probability that the reference model outputs the standard answer y* given the question q and the target sample fragment E*; Snec is the evidence necessity score, representing the second probability after replacing or removing the true evidence; PA(y*|q,E-) represents the third probability; Ê represents the output fragment, and Sloc represents the degree of temporal overlap.
[0112] In embodiments of this disclosure, a first probability of obtaining the output sample answer is obtained by inputting the sample question and the target sample fragment into the reference model. This probability measures the sufficiency of the target sample fragment in supporting the sample answer, thereby determining whether the evidence is sufficient and identifying samples where the evidence is insufficient to support the answer. A second probability of obtaining the output sample answer is obtained by inputting the sample question alone into the reference model. This probability measures the extent to which the model can still arrive at the sample answer after removing the target sample fragment, thereby determining the necessity of the target sample fragment for the answer and identifying false evidence questions that can be answered solely based on linguistic priors or question wording. A third probability of obtaining the output sample answer is obtained by inputting the sample question and interfering sample fragments into the reference model. This probability measures whether the interfering sample fragments will mislead the model into arriving at the sample answer, thereby determining the discriminativeness and validity of the interfering sample fragments and ensuring that the interfering fragments do not contain evidence for the answer.
[0113] Furthermore, by inputting sample questions into the reference model and determining the temporal overlap between the output segments obtained by the reference model and the target sample segments, the temporal accuracy of evidence localization can be measured, thereby filtering out samples with consistent evidence localization time. This allows for the filtering of low-quality samples, ensuring that the samples entering the training process have sufficient evidence support and reliable answer correspondences, thus improving the sample quality for training the long video understanding model and the final model performance.
[0114] The above are merely exemplary embodiments, but are not limited thereto. Other sample generation methods known in the art may also be included, as long as they can improve the continuous self-evolution capability of subsequent long video question answering model training.
[0115] Figure 6 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown schematically.
[0116] like Figure 6 As shown, the model training method 600 may include operations S610~S630.
[0117] In operation S610, training samples for the current training round are obtained, wherein the training samples are obtained using a sample generation method.
[0118] In operation S620, the retrieval module in the model obtained from the previous round of training is trained using training samples, resulting in the trained retrieval module.
[0119] In operation S630, while keeping the model parameters of the trained retrieval module unchanged, the response module in the model obtained in the previous round of training is trained using sample questions, evidence fragments, and sample answers to obtain the model obtained in the current round of training.
[0120] The training samples are sample pairs obtained using the aforementioned sample generation method and have passed quality verification. The training samples include sample questions, sample answers, target sample fragments, and interference sample fragments.
[0121] The model obtained from the previous training round, i.e., the student model after round t-1, is the training object for self-evolutionary training. It contains two functional modules: a retrieval module (Retriever) and an answering module (Answerer). The retrieval module performs multiple rounds of searching, inspecting, zooming, and selecting on long videos to locate and output the segments needed to answer the question. By training the retrieval module using training samples, it learns to determine which segments truly contain the evidence needed to answer the question.
[0122] The response module is used to answer questions based solely on evidence fragments selected / provided by the retrieval module. This round of training involves updating both modules separately based on the model. The parameters of the pre-trained retrieval module are frozen first, and then the response module is trained. For example, while freezing the retrieval module parameters, evidence Eτ obtained from the retrieval module's actual retrieval trajectory on the training samples can be used as input to the response module for training. This allows the response module to adapt to the distribution of the retrieval module's actual output, thereby narrowing the gap in evidence distribution between training and inference.
[0123] In the embodiments of this disclosure, by training the retrieval module using training samples, the retrieval module can learn to distinguish between genuine evidence and semantically similar but non-answering interference by comparing target sample fragments with interference sample fragments, thereby improving the accuracy of evidence localization and retrieval efficiency. By training the answer module using sample questions, evidence fragments, and sample answers while keeping the model parameters of the trained retrieval module unchanged, the training of the answer module will not interfere with or destroy the already trained retrieval capabilities; at the same time, the answer module answers based on evidence fragments, thereby avoiding it from bypassing the retrieval process and directly guessing the answer based on prior language knowledge, enhancing the evidentiary constraint and interpretability of the answer, and thus improving the overall accuracy and credibility of the model in long video question answering.
[0124] Figure 7 The illustration shows an example schematic diagram of a model training process according to an embodiment of the present disclosure.
[0125] like Figure 7 As shown, in Example 700 of the training model, the model Mstart obtained from the previous training round includes a retrieval module M1 and a response module M2 as an example.
[0126] Obtain training samples 710 for the current training round. Use the training samples 710 to train the retrieval module M1 in the model Mstart obtained from the previous training round, resulting in a trained retrieval module M4. In one embodiment, this training process may include two stages: first, based on the training samples 710, perform a first-stage training on the retrieval module M1 to obtain an intermediate retrieval module M3. This first-stage training is used to improve the retrieval module's ability to identify the correlation between the sample question and candidate sample segments; then, perform a second-stage training on the intermediate retrieval module M3 to obtain the trained retrieval module M4. This second-stage training is used to improve the intermediate retrieval module's ability to locate evidence segments in the sample video.
[0127] The first training phase is the initial stage of training the retrieval module. Its goal is to improve the module's ability to identify the relevance between the sample question and candidate sample segments; that is, to teach the module to determine which segment is more relevant. For example, the retrieval module might output a high relevance score for (q, E*) and a low score for (q, E-), thus learning to distinguish which segment is more relevant.
[0128] In one embodiment, E* can be used as positive samples, and E- and random negative samples as negative samples. Through contrastive loss training, the relevance score of positive samples can be higher than that of negative samples, thus learning the relevance ranking between the question and the fragment. Alternatively, the sample question and candidate sample fragments can be used as input to directly train a binary classifier to output whether the fragment is evidence supporting the answer. Using E* as the positive label and E- and random fragments as the negative labels, cross-entropy loss is used for training, enabling the retrieval module to directly learn to judge relevance.
[0129] The intermediate retrieval module is the retrieval module that has completed the first stage of training but has not yet undergone the second stage. At this point, it already has the ability to rank the relevance of question segments, but it does not yet have the strategy to perform multiple rounds of retrieval in the complete video, locate evidence, and stop at an appropriate time. For example, the intermediate retrieval module can score and rank given candidate segments, but it cannot yet autonomously execute the retrieval action sequence of search→zoom→select.
[0130] The second phase of training is an advanced stage of training the retrieval module. Its goal is to improve the intermediate retrieval module's ability to locate evidence fragments in sample videos. Specifically, it teaches the intermediate retrieval module how to locate evidence fragments step-by-step within the sample video and stop at the appropriate time. For example, the intermediate retrieval module executes the action sequence search_text→inspect_block→zoom_in→select_evidence→stop on a two-level index, and is rewarded more for finding correct evidence and viewing fewer frames.
[0131] In one embodiment, a high-quality, low-cost demonstration retrieval trajectory can be generated first by a reference model. Then, an intermediate retrieval module can be trained using a behavior-supervised imitation approach to gradually reproduce the expert action sequence. Fine-tuning can be done by incorporating rewards, thereby reducing the policy exploration space and accelerating convergence. Alternatively, the intermediate retrieval module can be treated as a policy network with an action space including search_text, inspect_block, zoom_in, select_evidence, stop, etc. Rewards can be based on whether the answer is correct, evidence IoU, and retrieval cost. Policy gradient algorithms such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) can be used to directly optimize multi-round action policies, enabling the retrieval module to learn to achieve a balance between correctness and cost.
[0132] In the embodiments of this disclosure, the retrieval module undergoes a first-stage training based on training samples. This stage aims to improve the retrieval module's ability to identify the correlation between sample questions and candidate sample segments, enabling the retrieval module to acquire basic question-segment matching and ranking capabilities, and to determine which candidate segment is more relevant to the question. A second-stage training is then conducted on the intermediate retrieval module to improve its ability to locate evidence segments in sample videos. This allows the intermediate retrieval module, building upon its existing relevance judgment capabilities, to further learn to perform retrieval actions step-by-step within the complete video, locate evidence, and stop at appropriate times. This achieves a balance between correctly finding evidence and minimizing video viewing, thereby improving the accuracy of evidence location and retrieval efficiency.
[0133] After obtaining the trained retrieval module M4, while keeping the model parameters of the trained retrieval module M4 unchanged, the response module M2 in the model obtained in the previous round of training can be trained using sample questions, evidence fragments and sample answers to obtain the trained response module M5. The trained retrieval module M4 and the trained response module M5 are then used as the model Mend obtained in the current round of training.
[0134] According to embodiments of this disclosure, a first-stage training of the retrieval module is performed based on training samples to obtain an intermediate retrieval module. This can include the following operations: determining a contrast loss using sample questions and target sample segments as positive samples and sample questions and interference sample segments as negative samples, wherein the contrast loss is used to constrain a first difference between the sample questions and target sample segments to be less than a second difference between the sample questions and interference sample segments; and adjusting the model parameters of the retrieval module based on the contrast loss, with the goal of reducing the first difference and increasing the second difference, until a predetermined termination condition is met, thereby obtaining the intermediate retrieval module.
[0135] Positive samples are sample pairs (q, E*) consisting of a sample question and a target sample fragment, representing the matching relationship between the question and true evidence. Negative samples are sample pairs (q, E-) consisting of a sample question and a distractor sample fragment, representing the non-matching relationship between the question and non-evidence / distractor fragments. For example, a positive sample pair (q, E*) means that "what was taken out" should match the fragment "taking milk"; a negative sample pair (q, E-) means that "what was taken out" should not match the fragment "taking mineral water".
[0136] Contrastive loss is a loss function constructed based on positive and negative sample pairs. It is used to constrain the difference between the question and the positive sample to be less than the difference between the question and the negative sample, that is, to ensure that the retrieval module scores higher for the relevance of the positive sample than for the negative sample. Specifically, the first difference is the semantic difference between the sample question and the target sample fragment (positive sample), and the smaller the difference, the more relevant they are; the second difference is the semantic difference between the sample question and the interference sample fragment, and the larger the difference, the less relevant they are.
[0137] In one embodiment, the contrast loss can be represented by the following formula (6).
[0138] Lretrieve=-log[exp(s(q,E+) / T) / (exp(s(q,E+) / T)+Σjexp(s(q,Ej) / T))]; (6)
[0139] Here, Lretrieve represents the contrast loss; s(·) represents the similarity function, which calculates the similarity score between the sample question and the candidate video segment; T represents the temperature parameter, which is used to adjust the sharpness of the similarity score distribution and control the difficulty of contrast learning.
[0140] After determining the contrast loss, parameter optimization can be performed based on the contrast loss. With the optimization objectives of reducing the difference between the problem and the target segment and increasing the difference between the problem and the negative sample segment, the parameters of the retrieval module are updated through backpropagation gradient. The iteration continues until the predetermined termination condition is met, resulting in an intermediate retrieval module with relevance discrimination capability.
[0141] The predetermined termination condition is the criterion for determining when training stops, used to control when the parameter adjustment process ends. For example, the predetermined termination condition may include at least one of the following: the contrast loss decreases by less than a threshold for several consecutive rounds, the proportion of positive samples with higher scores than negative samples on the validation set reaches a preset value, a preset number of training rounds / steps is reached, and the validation set relevance ranking metric (such as ranking accuracy) no longer improves, etc.
[0142] For example, a contrastive loss can be used as the objective function, with gradients calculated and retrieval module parameters updated via backpropagation, iterating repeatedly until the loss converges or a preset number of training steps is reached. Alternatively, adaptive optimizers such as Adam / AdamW can be used for parameter updates, coupled with learning rate scheduling, and a relevance ranking metric on the validation set (such as the proportion of positive samples scoring higher than negative samples) can be used as the stopping criterion. Training can be terminated early when the metric no longer improves to prevent overfitting.
[0143] In the embodiments of this disclosure, a contrast loss is determined by using sample questions and target sample segments as positive samples and sample questions and sample segments as negative samples. This contrast loss constrains a first difference between the sample questions and target sample segments to be less than a second difference between them. This allows the retrieval module to explicitly learn during training that the relevance between real evidence and questions should be stronger than that of interference or irrelevant segments, thereby establishing a discrimination objective for matching and ranking questions and evidence. Based on this, the model parameters of the retrieval module are adjusted according to the contrast loss, with the goal of reducing the first difference and increasing the second difference, until a predetermined termination condition is met. This allows the relevance measurement between questions and segments in the retrieval module to be continuously optimized during training, gradually increasing the discriminant difference between positive and negative samples. Ultimately, an intermediate retrieval module is obtained that can accurately identify the relevance between sample questions and candidate sample segments and effectively distinguish between real evidence and highly similar interference segments.
[0144] According to embodiments of this disclosure, a second-stage training of the intermediate retrieval module is performed to obtain a trained retrieval module, which may include the following operations: performing multiple rounds of retrieval on multiple candidate sample segments of the sample video according to the sample question to obtain an output segment set; adjusting the model parameters of the intermediate retrieval module based on the third difference between the target sample segment and the output segment set, and the resource consumption information corresponding to the retrieval trajectory of the multiple rounds of retrieval, until a predetermined termination condition is met to obtain the trained retrieval module.
[0145] The intermediate retrieval module can perform multiple rounds of retrieval actions on candidate sample segments of the sample video based on the sample question, gradually narrowing the scope, magnifying details, and finally selecting evidence to obtain the output segment set Eτ. Multi-round retrieval involves the retrieval module performing retrieval actions multiple times and sequentially on the candidate sample segments. Examples include text retrieval (search_text), visual retrieval (search_visual), block inspection (inspect_block), segment zooming (zoom_in), audio or subtitle reading (read_asr), evidence selection (select_evidence), and stopping the retrieval process (stop).
[0146] The multi-round retrieval process includes the intermediate retrieval module receiving a sample question, first performing a text search (search_text) to locate the relevant block, then performing a block inspection (inspect_block) and a fragment zoom-in (zoom_in) to the specific time period, and finally performing an evidence selection (select_evidence) to select the target evidence fragment, obtaining the output fragment set Eτ and recording the retrieval trajectory. The output fragment set Eτ is the final set of evidence fragments selected and output by the retrieval module after multiple rounds of retrieval, used by the subsequent answer module to provide an answer.
[0147] In one embodiment, the above-mentioned multi-round retrieval process can be that the intermediate retrieval module selects the next action step by step in the action space {search_text,search_visual,inspect_block,zoom_in,read_asr,select_evidence,stop} according to the current retrieval state S_t, and executes multiple rounds until stop, to obtain the output evidence set Eτ and the retrieval trajectory τ.
[0148] In another embodiment, the above-mentioned multi-round retrieval process can also be that the intermediate retrieval module selects only the single candidate segment action that maximizes the immediate benefit based on the current state at each step, such as selecting the unviewed segment that is most relevant to the question at each step, accumulating segment by segment until the accumulated evidence meets the threshold or reaches the budget, and then stopping to obtain the output segment set, thereby achieving a simple and efficient sequential retrieval.
[0149] The retrieval trajectory τ is a record that documents the sequence of actions and their parameters, viewed segments, retrieval results, and other information. Resource consumption information is a retrieval cost measure corresponding to the retrieval trajectory of multiple rounds of retrieval. For example, resource consumption information can be determined based on the ratio of viewed frames to total video frames (Fview / Ftotal), the number of retrieval actions (Naction), and the number of times irrelevant segments were viewed repeatedly (Rredundant), reflecting the number of frames, actions, and repeated views consumed in completing this retrieval.
[0150] The reward signal is constructed by combining the third difference between the target sample segment and the output segment set, and the resource consumption information of the trajectory. The parameters of the intermediate retrieval module are adjusted to achieve a balance between finding correct evidence and watching fewer videos, iterating until the termination condition is met. For example, if both trajectories find correct evidence, one might require watching 800 frames and 20 actions, while the other requires watching 50 frames and 5 actions; the latter will receive a higher reward. The parameters are adjusted accordingly to teach the intermediate retrieval module to find the correct evidence with fewer videos, until the reward converges.
[0151] In one embodiment, the reward for the second phase of training can be represented by the following formula (7).
[0152] rR=η1·I(ans=y*)+η2·IoU(Eτ,E*)-η3·Fview / Ftotal-η4·Naction-η5·Rredundant; (7)
[0153] Wherein, rR represents the reward of the retrieval module, comprehensively measuring the merits of the retrieval trajectory in terms of correctness, positioning accuracy, and cost; I(ans=y*) is an indicator function, indicating whether the final answer ans equals the standard answer y* (1 for correct, 0 for incorrect); ans represents the final answer given by the response module; Eτ represents the output fragment set; IoU(Eτ,E*) represents the time intersection-union ratio between the evidence set Eτ actually selected by the retrieval module and the target sample fragment (standard evidence) E*; Fview represents the number of frames viewed, i.e., the cumulative number of frames viewed during this retrieval process; Ftotal represents the total number of video frames, i.e., the total number of frames in the sample video; Fview / Ftotal represents the proportion of the number of viewed frames to the total number of video frames, used to measure the resource consumption of the retrieval; Naction represents the number of retrieval actions, i.e., the total number of retrieval actions executed in this retrieval trajectory; Rredundant represents the redundant viewing penalty, i.e., the penalty for repeatedly viewing confirmed irrelevant fragments, used to suppress inefficient repeated retrieval actions.
[0154] In the embodiments of this disclosure, by performing multiple rounds of retrieval on multiple candidate sample segments of a sample video based on the sample question to obtain an output segment set, the retrieval module can progressively perform retrieval actions in the complete video space, narrowing down the scope layer by layer and finally selecting evidence. This simulates and learns the real evidence localization process, gaining the ability to locate evidence in complex long videos. By adjusting the model parameters of the intermediate retrieval module based on the third difference between the target sample segment and the output segment set, as well as the resource consumption information corresponding to the retrieval trajectory of the multiple rounds of retrieval, until a predetermined termination condition is met, the retrieval module can simultaneously consider the accuracy of evidence localization and the resource consumption of retrieval during training. This achieves a balance between correctly finding evidence and minimizing video viewing, improving the accuracy of evidence localization and retrieval efficiency.
[0155] The above are merely exemplary embodiments, but are not limited thereto. Other model training methods known in the art may also be included, as long as they can enhance the evidentiary binding force and interpretability of the answer.
[0156] Figure 8 A flowchart illustrating a video question-and-answer method according to an embodiment of the present disclosure is shown.
[0157] like Figure 8 As shown, the video question-and-answer method 800 may include operations S810~S820.
[0158] In operation S810, in response to receiving a question to be processed for the target video, the target video and the question to be processed are input into the retrieval module in the trained model to obtain the target segment. The target video includes multiple candidate segments, and the trained model is obtained using a model training method.
[0159] When operating the S820, the target question and target fragment are input into the answer module of the trained model to obtain the target answer.
[0160] The target video is the long video input during the application phase; that is, the video object from which the user poses a question. For example, the target video could be a 30-minute meeting recording or surveillance video uploaded by a user, from which the user hopes to obtain the answer to a certain question. The problem to be processed is the natural language question posed by the user to the target video during the application phase.
[0161] The trained model is a self-evolving long video understanding model trained using the aforementioned model training method. It comprises two functional modules: a retrieval module and an answering module. The retrieval module, or Retriever, is used to locate evidence segments relevant to the question being addressed among multiple candidate segments in the target video. The answering module, or Answerer, is used to answer the question based on the target segments output by the retrieval module.
[0162] In the application phase, the retrieval module receives the target video and the question to be processed, and outputs the located target segment. The target segment is the evidence segment related to the question to be processed, located and output by the retrieval module from the candidate segments. In the application phase, the answering module receives the question to be processed and the target segment, and outputs the target answer.
[0163] In the embodiments of this disclosure, in response to receiving a question to be processed regarding a target video, the target video and the question to be processed are input into the retrieval module of a trained model to obtain a target segment. This enables the model to automatically locate evidence segments related to the question to be processed from multiple candidate segments, achieving automatic and accurate evidence location, thereby improving reasoning efficiency and reducing resource consumption. Furthermore, by inputting the target question and the target segment into the answer module of the trained model to obtain a target answer, the answer module can provide an answer based on the clearly located target segment as explicit evidence. This ensures the answer is grounded in specific video evidence, thereby improving the accuracy, interpretability, and credibility of the target answer.
[0164] Figure 9 The illustration shows an example diagram of a video question-and-answer process according to an embodiment of the present disclosure.
[0165] like Figure 9 As shown, in the embodiment of video question answering, the trained model Mend, including the trained retrieval module M4 and the trained answering module M5, is used as an example for illustration.
[0166] In response to receiving a question 920 to be processed for the target video 910, the target video 910 and the question 920 to be processed are input into the retrieval module M4 in the trained model to obtain the target segment 930; the target question 920 and the target segment 930 are input into the answer module M5 in the trained model to obtain the target answer 940.
[0167] The above are merely exemplary embodiments, but are not limited thereto. Other video question-answering methods known in the art may also be included, as long as they can improve the accuracy, interpretability, and credibility of the target answer.
[0168] Figure 10 A block diagram of a sample generation apparatus according to an embodiment of the present disclosure is shown schematically.
[0169] like Figure 10 As shown, the sample generation device 1000 may include a first determining module 1010, a second determining module 1020, and a sample generation module 1030.
[0170] The first determining module 1010 is used to determine the target sample segment for the current training round from multiple candidate sample segments of the sample video based on the guidance information for the current training round, wherein the guidance information indicates the capability dimension that the model obtained in the previous training round needs to be adjusted in the current training round.
[0171] The second determining module 1020 is used to determine interference sample segments from multiple candidate sample segments based on the sample question and sample answer obtained based on the target sample segment, wherein the interference sample segment semantically matches the target sample segment but does not semantically match the sample answer.
[0172] The sample generation module 1030 is used to take the sample question, sample answer, target sample fragment and interference sample fragment as training samples for the current round of training.
[0173] According to embodiments of this disclosure, the guidance information for the current training round can be obtained through a partitioning module, a processing module, and a third determination module.
[0174] The partitioning module is used to partition the validation sample set according to multiple capability dimensions, resulting in sub-validation sample sets for each capability dimension.
[0175] The processing module is used to process each sub-validation sample set by calling the model obtained from the previous round of training and the reference model, and obtain the evaluation results of multiple capability dimensions respectively. The parameter size of the reference model is the same as the parameter size of the model obtained from the previous round of training.
[0176] The third determination module is used to determine guidance information based on the assessment results of multiple capability dimensions.
[0177] According to embodiments of this disclosure, the second determining module 1020 may include a first generating submodule and a filtering submodule.
[0178] The first generation submodule is used to generate sample questions and sample answers based on the first semantic information of the target sample fragment.
[0179] The filtering submodule is used to filter multiple candidate sample segments to obtain interference sample segments with the goal of reducing at least one of the following: reducing the first difference between the first semantic information and the second semantic information of the candidate sample segments, increasing the second difference between the sample question and the second semantic information, and increasing the third difference between the sample answer and the second semantic information.
[0180] According to embodiments of this disclosure, the sample generation module 1030 may include an evaluation submodule and a second generation submodule.
[0181] The evaluation submodule is used to perform quality assessment on sample questions, sample answers, target sample fragments, and interference sample fragments, and obtain the quality assessment results; and
[0182] The second generation submodule is used to, in response to the passing of the quality assessment result, use the sample question, sample answer, target sample fragment, and interference sample fragment as training samples for the current round of training.
[0183] According to embodiments of this disclosure, the evaluation submodule may include at least one of a first obtaining unit, a second obtaining unit, a third obtaining unit, and a fourth obtaining unit.
[0184] The first acquisition unit is used to input the sample question and the target sample fragment into the reference model to obtain the first probability of the reference model outputting the sample answer.
[0185] The second obtaining unit is used to input the sample question into the reference model and obtain the second probability of the reference model outputting the sample answer.
[0186] The third acquisition unit is used to input the sample question and interference sample fragments into the reference model to obtain the third probability of the reference model outputting the sample answer.
[0187] The fourth acquisition unit is used to input the sample problem into the reference model to determine the degree of temporal overlap between the output fragment obtained by the reference model and the target sample fragment.
[0188] Figure 11 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown schematically.
[0189] like Figure 11 As shown, the model training device 1100 may include an acquisition module 1110, a first training module 1120, and a second training module 1130.
[0190] The acquisition module 1110 is used to acquire training samples for the current training round. The training samples are obtained using a sample generation device and include sample questions, sample answers, target sample fragments, and interference sample fragments.
[0191] The first training module 1120 is used to train the retrieval module in the model obtained from the previous training round using training samples, so as to obtain the trained retrieval module.
[0192] The second training module 1130 is used to train the answer module in the model obtained in the previous round of training using sample questions, evidence fragments and sample answers, while keeping the model parameters of the trained retrieval module unchanged, so as to obtain the model obtained in the current round of training.
[0193] According to embodiments of this disclosure, the first training module 1120 may include a first training submodule and a second training submodule.
[0194] The first training submodule is used to perform the first stage of training on the retrieval module based on the training samples to obtain the intermediate retrieval module. The first stage of training is used to improve the retrieval module's ability to identify the correlation between the sample question and the candidate sample fragment.
[0195] The second training submodule is used to perform a second stage of training on the intermediate retrieval module to obtain a trained retrieval module. The second stage of training is used to improve the ability of the intermediate retrieval module to locate evidence fragments in the sample video.
[0196] According to embodiments of this disclosure, the first training submodule may include a determining unit and a first adjusting unit.
[0197] A determination unit is used to determine the contrast loss using the sample problem and the target sample segment as positive samples and the sample problem and the interference sample segment as negative samples. The contrast loss is used to constrain the first difference between the sample problem and the target sample segment to be less than the second difference between the sample problem and the sample segment.
[0198] The first adjustment unit is used to adjust the model parameters of the retrieval module based on the contrast loss, with the goal of reducing the first difference and expanding the second difference, until the predetermined termination condition is met, thus obtaining the intermediate retrieval module.
[0199] According to embodiments of this disclosure, the first training submodule may include a retrieval unit and a second adjustment unit.
[0200] The retrieval unit is used to perform multiple rounds of retrieval on multiple candidate sample segments of the sample video according to the sample question, and obtain the output segment set.
[0201] The second adjustment unit is used to adjust the model parameters of the intermediate retrieval module based on the third difference between the target sample fragment and the output fragment set, and the resource consumption information corresponding to the retrieval trajectory of multiple rounds of retrieval, until the predetermined termination condition is met, and thus obtain the trained retrieval module.
[0202] Figure 12 A block diagram of a video question-and-answer device according to an embodiment of the present disclosure is shown schematically.
[0203] like Figure 12 As shown, the video question-and-answer device 1200 may include a first question-and-answer module 1210 and a second question-and-answer module 1220.
[0204] The first question-answering module 1210 is used to respond to receiving a question to be processed for the target video, inputting the target video and the question to be processed into the retrieval module in the trained model to obtain the target segment. The target video includes multiple candidate segments, and the trained model is obtained using a model training device.
[0205] The second question-answering module 1220 is used to input the target question and target fragment into the answering module of the trained model to obtain the target answer.
[0206] Figure 13A schematic diagram illustrating the structure of a large-model-based intelligent agent according to an embodiment of the present disclosure is shown.
[0207] In embodiments of this disclosure, the von Neumann architecture in modern computer theory is inspired, such as... Figure 13 As shown, the AI agent 1300 may include five core modules: input module 1310, processing module 1320 and output module 1330.
[0208] Input module 1310 is responsible for receiving or sensing information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment), and converting it into a format that the AI agent 1300 can understand and process. Input module 1310 is the primary link for the AI agent 1300 to interact with the outside world, enabling the AI agent 1300 to efficiently and accurately acquire necessary "sensory" information from the outside world and respond to this information. In the example, input module 1310 can input the original sample set, training sample set, and content to be recognized, as described above.
[0209] In embodiments of this disclosure, the processing module 1320 may include a control module 1321, a storage module 1322, and a computation module 1323. The processing module 1320 is configured to determine a target task based on the input information received by the input module 1310, determine a target large model based on the target task, and obtain output information by calling the corresponding method executed by the target large model.
[0210] The control module 1321 is the core support for the AI agent 1300's ability to handle complex tasks. The control module 1321 can execute the sample generation method, model training method, and video question answering method described above.
[0211] In the example, the control module 1321 will continuously interact with the storage module 1322, the arithmetic module 1323, and / or the output module 1330 during operation. However, it should be noted that in the embodiments of this disclosure, the control module 1321 initiates communication with the storage module 1322, the arithmetic module 1323, and / or the output module 1330 as a single initiator, and there is no communication coupling between the storage module 1322, the arithmetic module 1323, and the output module 1330.
[0212] In the example, the performance of the control module 1321 is closely related to the large model on which the AI agent 1300 is based. To fully leverage the capabilities of the large model, the internal structure of the control module 1321 can be designed to be highly configurable and scalable to handle various types of tasks and requirements in real-world scenarios.
[0213] The storage module 1322 is responsible for remembering the generated training samples. The training samples mentioned above can be included in the storage module 1322.
[0214] In the example of sample generation, after receiving guidance information for the current training round, AI agent 1300 can execute the sample generation process, obtain training samples, and feed them back to control module 1321. Then, control module 1321 can pass the feedback training samples to output module 1330.
[0215] In the model training example, after receiving training samples, the AI agent 1300 can execute the model training process, obtain the model trained in the current round, and feed back the model parameters of the model trained in the current round to the control module 1321. Then, the control module 1321 can pass the model parameters of the model trained in the current round to the output module 1330.
[0216] In the video question-and-answer example, after receiving a question to be processed for the target video, the AI agent 1300 can perform the video question-and-answer process, obtain the target answer, and feed the target answer back to the control module 1321. Then, the control module 1321 can pass the feedback target answer to the output module 1330.
[0217] The computation module 1323 can be viewed as a predefined tool library. Tools used to determine boot information, as described above, can be included in the computation module 1323.
[0218] In the example, when the AI agent 1300 needs to determine guidance information, it can invoke relevant tools from the computing module 1323 and feed them back to the control module 1321. The control module 1321 can then use the fed-back tools to process the relevant information. It is understandable that although large models have excellent language understanding and generation capabilities, like humans, the tasks they can solve without any tools are very limited. When the AI agent 1300 is given the ability to invoke tools, it can perform tasks such as determining guidance information using tools specifically designed for this purpose.
[0219] The output module 1330 can output the training samples described above, the model parameters of the model obtained by training in the current round, and the target answer.
[0220] The AI agent 1300 according to the embodiments of this disclosure can simply and effectively improve the level of intelligence, as well as enhance flexibility and versatility.
[0221] Figure 14The diagram schematically illustrates an electronic device suitable for implementing sample generation, model training, and video question answering methods according to embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0222] like Figure 14 As shown, device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1402 or a computer program loaded from storage unit 1408 into random access memory (RAM) 1403. The RAM 1403 may also store various programs and data required for the operation of device 1400. The computing unit 1401, ROM 1402, and RAM 1403 are interconnected via bus 1404. Input / output (I / O) interface 1405 is also connected to bus 1404.
[0223] Multiple components in device 1400 are connected to I / O interface 1405, including: input unit 1406, such as a keyboard, mouse, etc.; output unit 1407, such as various types of displays, speakers, etc.; storage unit 1408, such as a disk, optical disk, etc.; and communication unit 1409, such as a network card, modem, wireless transceiver, etc. Communication unit 1409 allows device 1400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0224] The computing unit 1401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1401 performs the various methods and processes described above, such as sample generation, model training, and video question answering methods. For example, in some embodiments, the sample generation, model training, and video question answering methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1400 via ROM 1402 and / or communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of the sample generation, model training, and video question answering methods described above can be performed. Alternatively, in other embodiments, computing unit 1401 may be configured in any other suitable manner (e.g., by means of firmware) to perform sample generation, model training, and video question answering methods.
[0225] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0226] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0227] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0228] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0229] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0230] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.
[0231] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0232] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A sample generation method, comprising: Based on the guidance information used for the current training round, a target sample segment is determined from multiple candidate sample segments of the sample video for the current training round, wherein the guidance information indicates the capability dimension that the model obtained in the previous training round needs to be adjusted in the current training round. Based on the sample question and sample answer obtained from the target sample fragment, interference sample fragments are determined from a plurality of candidate sample fragments, wherein the interference sample fragments semantically match the target sample fragment but semantically do not match the sample answer; and The sample question, the sample answer, the target sample fragment, and the interference sample fragment are used as training samples for the current training round.
2. The method according to claim 1, wherein, The guidance information used for the current training round is obtained in the following way: The verification sample set is divided according to multiple capability dimensions to obtain multiple sub-verification sample sets for each of the capability dimensions. By calling the model and reference model obtained from the previous training round, each of the sub-validation sample sets is processed to obtain evaluation results for each of the multiple capability dimensions, wherein the parameter size of the reference model is the parameter size of the model obtained from the previous training round; and The guidance information is determined based on the evaluation results of each of the multiple capability dimensions.
3. The method according to claim 1, wherein, The step of determining interfering sample segments from a plurality of candidate sample segments based on sample questions and sample answers obtained based on the target sample segment includes: Based on the first semantic information of the target sample fragment, generate the sample question and the sample answer; and With the objective of reducing at least one of the following: reducing the first difference between the first semantic information and the second semantic information of the candidate sample fragment, increasing the second difference between the sample question and the second semantic information, and increasing the third difference between the sample answer and the second semantic information, multiple candidate sample fragments are screened to obtain the interference sample fragment.
4. The method according to any one of claims 1 to 3, wherein, The step of using the sample question, the sample answer, the target sample fragment, and the interference sample fragment as training samples for the current training round includes: The quality of the sample question, the sample answer, the target sample fragment, and the interference sample fragment is evaluated to obtain the quality evaluation result; and In response to the passing of the quality assessment result, the sample question, the sample answer, the target sample fragment, and the interference sample fragment are used as training samples for the current round of training.
5. The method according to claim 4, wherein, The quality assessment of the sample question, the sample answer, the target sample fragment, and the interference sample fragment includes at least one of the following: The sample question and the target sample fragment are input into the reference model to obtain the first probability that the reference model outputs the sample answer. The sample question is input into the reference model to obtain a second probability that the reference model outputs the sample answer; The sample question and the interference sample fragments are input into the reference model to obtain the third probability of the reference model outputting the sample answer; The sample problem is input into the reference model to determine the degree of temporal overlap between the output fragment obtained by the reference model and the target sample fragment.
6. A model training method, comprising: Obtain training samples for the current training round, wherein the training samples are obtained using the method of any one of claims 1 to 5, and the training samples include sample questions, sample answers, target sample fragments, and interference sample fragments; The retrieval module in the model obtained from the previous training round is trained using the training samples to obtain the trained retrieval module; and While keeping the model parameters of the trained retrieval module unchanged, the answer module in the model obtained in the previous round of training is trained using the sample question, the evidence fragment, and the sample answer to obtain the model obtained in the current round of training.
7. The method according to claim 6, wherein, The step of training the retrieval module in the model obtained from the previous training round using the training samples to obtain the trained retrieval module includes: Based on the training samples, the retrieval module undergoes a first-stage training to obtain an intermediate retrieval module. This first-stage training is used to improve the retrieval module's ability to identify the correlation between the sample question and candidate sample fragments. The intermediate retrieval module is trained in a second stage to obtain the trained retrieval module, wherein the second stage training is used to improve the ability of the intermediate retrieval module to locate the evidence fragment in the sample video.
8. The method according to claim 7, wherein, The step of training the retrieval module in the first stage based on the training samples to obtain the intermediate retrieval module includes: Using the sample problem and the target sample segment as positive samples, and the interfering sample problem and the sample segment as negative samples, a contrast loss is determined, wherein the contrast loss is used to constrain a first difference between the sample problem and the target sample segment to be less than a second difference between the sample problem and the sample segment; and Based on the contrast loss, with the goal of reducing the first difference and expanding the second difference, the model parameters of the retrieval module are adjusted until a predetermined termination condition is met, thus obtaining an intermediate retrieval module.
9. The method according to claim 7 or 8, wherein, The second stage of training the intermediate retrieval module to obtain the trained retrieval module includes: Based on the sample problem, multiple rounds of retrieval are performed on multiple candidate sample segments of the sample video to obtain an output segment set; and Based on the third difference between the target sample fragment and the output fragment set, and the resource consumption information corresponding to the retrieval trajectory of multiple rounds of retrieval, the model parameters of the intermediate retrieval module are adjusted until the predetermined termination condition is met, thus obtaining the trained retrieval module.
10. A video question-answering method, comprising: In response to receiving a question to be processed for a target video, the target video and the question to be processed are input into a retrieval module in a trained model to obtain a target segment, wherein the target video includes multiple candidate segments, and the trained model is obtained using the method of any one of claims 6 to 9; and The target question and the target fragment are input into the answer module of the trained model to obtain the target answer.
11. A sample generation apparatus, comprising: The first determining module is used to determine the target sample segment for the current training round from multiple candidate sample segments of the sample video based on the guidance information for the current training round, wherein the guidance information indicates the capability dimension that the model obtained in the previous training round needs to be adjusted in the current training round. The second determining module is configured to determine interfering sample segments from a plurality of candidate sample segments based on the sample question and sample answer obtained based on the target sample segment, wherein the interfering sample segments semantically match the target sample segment but semantically do not match the sample answer; and The sample generation module is used to use the sample question, the sample answer, the target sample fragment, and the interference sample fragment as training samples for the current round of training.
12. A model training device, comprising: The acquisition module is used to acquire training samples for the current training round, wherein the training samples are obtained using the apparatus of claim 11, and the training samples include sample questions, sample answers, target sample fragments, and interference sample fragments; The first training module is used to train the retrieval module in the model obtained from the previous training round using the training samples, thereby obtaining the trained retrieval module; and The second training module is used to train the response module in the model obtained in the previous round of training using the sample question, the evidence fragment, and the sample answer, while keeping the model parameters of the trained retrieval module unchanged, so as to obtain the model obtained in the current round of training.
13. A video question-and-answer device, comprising: A first question-answering module, configured to, in response to receiving a question to be processed for a target video, input the target video and the question to be processed into a retrieval module in a trained model to obtain a target segment, wherein the target video includes multiple candidate segments, and the trained model is obtained using the apparatus of claim 12; and The second question-and-answer module is used to input the target question and the target fragment into the answer module of the trained model to obtain the target answer.
14. An intelligent agent based on a large model, comprising: The input module is used to receive input information; The processing module is configured to determine a target task based on the input information received by the input module, determine a target large model based on the target task, and execute the method described in claims 1 to 10 by calling the target large model to obtain output information; as well as An output module is used to output the output information obtained by the processing module.
15. An electronic device comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 10.
16. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 10.
17. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 10.
