A video understanding adaptive evidence selection method, system and electronic device

Abstract

The application discloses a video understanding adaptive evidence selection method and system and electronic equipment, which matches a question vector with each video segment vector respectively, constructs a rough retrieval candidate pool based on the video segments matched with the question vector, guarantees candidate evidence recall, and then judges whether the candidate segment is truly helpful for answering by a multi-modal verification model, effectively filters similar interference segments of the same scene, the same character or the same object, alleviates the inconsistency problem of "relevance-answer utility", and more importantly, adopts positive samples and weighted negative samples for comparative learning, so that the multi-modal encoder can distinguish highly similar but different answer utility segments, thereby realizing distinguishing coding in the coding step, making the matching degree of correct segments higher than that of interference segments, further avoiding the selection of interference segments which are highly related in semantics with the question text but do not contain the decisive evidence required for answering the question, and improving the accuracy of evidence selection and video understanding.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, multimodal information processing, video understanding and visual language modeling, and more specifically, relates to an adaptive evidence selection method, system and electronic device for video understanding. Background Technology

[0002] With the development of visual language models and multimodal large models, video question answering systems are now able to output answers based on video content and natural language questions. However, in long video scenarios, the evidence needed for the answer often appears only within a very short time interval of the entire video. For example, in a video lasting tens of minutes or even hours, the action, object, subtitle, or voice that truly determines the answer may only occupy a few seconds. Due to the limitations of the context window, GPU memory, and computational overhead of visual language models, the system can typically only extract a limited number of frames from long videos to input into the model. Therefore, "selecting which time segments and which frames" becomes crucial to the performance of long video question answering.

[0003] The first type of method commonly used in existing technologies is uniform frame sampling or fixed-interval frame sampling. This type of method is simple to implement, problem-independent, and model-independent, and has certain usability in short videos or videos with abundant evidence. However, in long videos with sparse evidence, uniform frame sampling is prone to missing key moments; even if some relevant frames are extracted, the system will waste a lot of its resources on background segments that are irrelevant to the problem because the number of frames is approximately equal for each time period.

[0004] The second common approach in existing technologies is selection based on the semantic similarity between the question and video frames or segments. This type of method typically maps the question text and candidate frames / segments to the same vector space and selects the Top-K candidates based on similarity. While this approach has some query-awareness compared to uniform frame sampling, its core drawback is that question relevance does not equate to answer utility. Long videos contain numerous "similar distracting segments" with the same scene, characters, objects, or actions. These segments are semantically highly relevant to the question but do not contain the decisive evidence needed to answer it. If selection is based solely on similarity, the limited frame budget is easily allocated to these distracting segments, thus crowding out genuine evidence frames.

[0005] The third type of method commonly found in existing technologies is end-to-end training of frame selectors or frame extraction strategies based on reinforcement learning. These methods attempt to learn to select more useful frames through question-and-answer rewards or labeled data, but they usually have the following shortcomings: First, frame-level selection lacks local temporal context, and isolated frames are difficult to express action processes, speech semantics, and event transitions; second, the training signal is strongly coupled with the specific answer model or dataset, resulting in insufficient stability when transferred to different models, different video domains, and different frame budgets; and third, the training cost is high, making it difficult to frequently adapt to real-world systems.

[0006] Existing technologies also include methods for compressing long videos into fewer visual tokens, summary text, or global video representations. These methods can alleviate input length limitations, but compression itself cannot determine which time periods are most worth retaining, and in particular, it cannot effectively distinguish between segments that "appear relevant but do not answer the question" and segments that "truly support the answer."

[0007] In summary, existing technologies suffer from at least the following problems: First, under a fixed frame budget, uniform frame extraction struggles to cover sparse, critical evidence; second, similarity-based retrieval methods are susceptible to interference from similar interfering segments within the same video, making it impossible to directly determine whether candidate segments are useful for the answer; third, existing methods typically lack an independent evidence utility verification stage, resulting in a gap between retrieval recall and final question-answer accuracy; fourth, existing frame allocation strategies often treat candidate segments equally, failing to dynamically allocate frame budgets based on evidence credibility; and fifth, when multiple questions reuse the same video in long videos, there is a lack of cacheable, reusable, and scalable evidence indexing and verification frameworks. Therefore, there is an urgent need to provide a coarse-to-fine evidence selection scheme for long-video question answering that explicitly distinguishes between "relevance" and "answer utility." Summary of the Invention

[0008] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a video understanding adaptive evidence selection method, system, and electronic device, thereby solving the technical problem that existing technologies struggle to accurately extract video segments that truly support the answer, and that video understanding accuracy is poor when context windows, video memory, and computational overhead are all limited.

[0009] To achieve the above objectives, according to a first aspect of the present invention, a video understanding adaptive evidence selection method is provided, comprising: Receive a video understanding request and extract the video to be understood and the question text from the video understanding request; The video to be understood is divided into multiple video segments according to a preset duration, and each video segment is encoded based on a trained multimodal encoder to obtain the segment vector of the corresponding video segment. Extract the question vector from the question text, and match the question vector with the segment vector of each video segment. Construct a coarse search candidate pool based on the video segments that match the question vector. The answer utility score of each candidate video segment in the coarse search candidate pool is evaluated using a multimodal verification model; wherein, the higher the answer utility score of any candidate video segment, the greater the likelihood that the candidate video segment contains the evidence required for the question text; Based on the matching degree between the segment vector of each candidate video segment and the question vector, and the answer utility score of each candidate video segment, the target video segment containing the evidence required for the question text is determined. The multimodal encoder is trained based on the model loss calculated from positive and negative samples. The model loss is the average of the sample losses of each positive sample. The sample loss of any positive sample is the negative logarithm of the ratio of the matching degree index between the segment vector output by the multimodal encoder for the video segment in the positive sample and the question vector of the question text in the positive sample to the sum of the matching degree index values ​​of all positive samples and weighted negative samples. The closer the video segment in any negative sample is to the time interval where the evidence required by the corresponding question text is located, the smaller the weight of the negative sample.

[0010] According to any of the above adaptive evidence selection methods for video understanding, the sample loss for any positive sample is: ; Where k is a positive sample, The sample loss for positive sample k, To compare the learning temperature coefficients, P represents the set of positive samples, and N represents the set of negative samples. The degree of matching between the segment vector output by the multimodal encoder for a video segment in positive sample k and the question vector of the question text in positive sample k. The degree of matching between the segment vector output by the multimodal encoder for a video segment in positive sample p and the question vector of the question text in positive sample p. The degree of matching between the segment vector output by the multimodal encoder for the video segment in negative sample j and the question vector of the question text in negative sample j. The weights are for negative sample j.

[0011] According to any of the above video understanding adaptive evidence selection methods, the weight of any negative sample is calculated based on the following formula: ; ; The time interval for the evidence required for the problematic text in this negative sample is [a, b]. and Let these be the start and end times of the video segment in the negative sample. For preset coefficients, This is the weight of the negative sample.

[0012] According to any of the above-mentioned adaptive evidence selection methods for video understanding, the step of evaluating the answer utility score of each candidate video segment in the coarse search candidate pool using a multimodal verification model specifically includes: For each candidate video segment in the coarse search candidate pool, a prompt word is constructed containing the corresponding candidate video segment, the question text, and the task requirement. The prompt word is then input into a multimodal validation model to obtain the original output of the multimodal validation model. The task requirement instructs the model to determine whether the input video segment contains the evidence required to answer the input question text. Based on the original output scores of affirmative category words and negative category words in the original output results of each candidate video segment, the answer utility score of the corresponding candidate video segment is calculated.

[0013] Based on any of the above video understanding adaptive evidence selection methods, the answer utility score for candidate video segment i is calculated using the following formula: ; ; Where Y and N are the sets of affirmative category lexical units and the sets of negative category lexical units, respectively; To affirm the original output score of the category term y, The original output score for the negation category term n; Let i be the utility score of the answer for candidate video segment i.

[0014] Based on any of the above-mentioned adaptive evidence selection methods for video understanding, and based on the matching degree between the segment vector of each candidate video segment and the question vector, as well as the answer utility score of each candidate video segment, the target video segment containing the evidence required for the question text is determined, specifically including: The fusion score for each candidate video segment is calculated based on the following formula: ; in, The Z-score normalization function, To preset weights, Let i be the utility score of the answer for candidate video segment i. Let be the matching degree between the segment vector of candidate video segment i and the question vector. Let i be the fusion score of candidate video segment i; Based on the fusion scores of each candidate video segment, multiple target video segments are selected.

[0015] According to any of the above-mentioned video understanding adaptive evidence selection methods, after determining the target video segment containing the evidence required for the question text, the method further includes: Based on the video frame budget, the number of target video segments, and the minimum number of frames extracted from each target video segment, calculate the number of adjustable frames that satisfy the condition that the number of video frames extracted from each target video segment is greater than or equal to the minimum number of frames. Based on the fusion score of each target video segment, the allocation ratio of each target video segment is determined; wherein, the higher the fusion score of any target video segment, the higher the allocation ratio of that target video segment. Based on the allocation ratio of each target video segment and the number of adjustable frames, the number of additional frames allocated to each target video segment is determined, and the sum of the number of additional frames allocated to each target video segment and the minimum number of frames is determined as the final number of extracted frames for each target video segment. Based on the final extracted frame number of each target video segment, video frames are sampled from the corresponding target video segments. The video frames sampled from each target video segment are then concatenated and input together with the question text into the visual language response model to obtain the answer text output by the visual language response model.

[0016] Based on any of the above video understanding adaptive evidence selection methods, the allocation ratio of each target video segment is calculated using the following formula: ; in, The allocation ratio for target video segment i. Let i be the fusion score of the target video segment i. For the target set of video clips, Assign a temperature coefficient to the frame; The number of additional frames allocated to each target video segment is determined based on the following formula, so that the final number of extracted frames for each target video segment meets the preset condition: ; in, The number of extra frames allocated to target video segment i. For the adjustable number of frames, The maximum number of frames that can be allocated. The minimum number of frames, For use in filling in The integer terms of the rounded residual; The preset condition is that the sum of the final extracted frames of each target video segment is equal to the video frame budget, and the final extracted frame number of each target video segment is less than or equal to the maximum frame number.

[0017] According to a second aspect of the present invention, a video understanding adaptive evidence selection system is provided, comprising: The request parsing unit is used to receive video understanding requests and extract the video to be understood and the question text from the video understanding requests. The video encoding unit is used to divide the video to be understood into multiple video segments according to a preset duration, and to encode each video segment based on a trained multimodal encoder to obtain the segment vector of the corresponding video segment. A coarse retrieval unit is used to extract the question vector from the question text and match the question vector with the segment vector of each video segment. A coarse retrieval candidate pool is constructed based on the video segments that match the question vector. The answer utility evaluation unit is used to evaluate the answer utility score of each candidate video segment in the coarse search candidate pool using a multimodal verification model; wherein, the higher the answer utility score of any candidate video segment, the greater the likelihood that the candidate video segment contains the evidence required for the question text; An evidence selection unit is used to determine a target video segment containing the evidence required for the question text based on the matching degree between the segment vector of each candidate video segment and the question vector, as well as the answer utility score of each candidate video segment. The multimodal encoder is trained based on the model loss calculated from positive and negative samples. The model loss is the average of the sample losses of each positive sample. The sample loss of any positive sample is the negative logarithm of the ratio of the matching degree index between the segment vector output by the multimodal encoder for the video segment in the positive sample and the question vector of the question text in the positive sample to the sum of the matching degree index values ​​of all positive samples and weighted negative samples. The closer the video segment in any negative sample is to the time interval where the evidence required by the corresponding question text is located, the smaller the weight of the negative sample.

[0018] According to a third aspect of the present invention, an electronic device is provided, comprising: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0019] According to a fourth aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to perform the method as described in the first aspect.

[0020] According to a fifth aspect of the invention, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the method as described in the first aspect.

[0021] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: This approach decouples coarse retrieval from fragment answer utility verification. First, it matches the question vector with the fragment vectors of each video segment, constructing a coarse candidate pool based on the matched video segments to ensure recall of candidate evidence. Then, a multimodal verification model determines whether candidate segments truly contribute to the answer, effectively filtering similar interfering segments from the same scene, person, or object, explicitly mitigating the inconsistency between relevance and answer utility. More importantly, it employs comparative learning using positive and weighted negative samples, enabling the multimodal encoder to distinguish highly similar segments with different answer utilities. This achieves discriminative encoding at the encoding stage, ensuring that the matching degree of correct segments is higher than that of interfering segments. This further avoids selecting interfering segments that are semantically highly relevant to the question text but do not contain the decisive evidence needed to answer the question. This improves the accuracy of evidence selection and video understanding even with limited context windows, GPU memory, and computational overhead. Furthermore, this scheme can be used with different visual language answer models without requiring fine-tuning of the answer model, making it easy to insert as an external evidence selection module into existing systems, offering high flexibility and scalability.

[0022] Furthermore, by using fusion scores to guide frame budget allocation, more frames can be obtained by efficiently using fragments, while fewer frames can be obtained or fragments can be discarded by inefficiently using fragments. This can avoid the budget waste caused by uniform frame extraction and improve the coverage of key evidence under a fixed frame budget. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the adaptive evidence selection method for video understanding provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram illustrating the selection and comparison of video understanding evidence provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0026] This invention provides an adaptive evidence selection method for video understanding, such as... Figure 1 As shown, it includes: S1, Receive a video understanding request, and extract the video to be understood and the question text from the video understanding request; S2, the video to be understood is divided into multiple video segments according to a preset duration, and each video segment is encoded based on a trained multimodal encoder to obtain the segment vector of the corresponding video segment; S3, extract the question vector from the question text, and match the question vector with the segment vector of each video segment respectively, and construct a coarse search candidate pool based on the video segments that match the question vector; S4. Utilize a multimodal verification model to evaluate the answer utility score of each candidate video segment in the coarse search candidate pool; wherein, the higher the answer utility score of any candidate video segment, the greater the likelihood that the candidate video segment contains the evidence required for the question text; S5. Based on the matching degree between the segment vector of each candidate video segment and the question vector, and the answer utility score of each candidate video segment, determine the target video segment containing the evidence required for the question text. The multimodal encoder is trained based on the model loss calculated from positive and negative samples. The model loss is the average of the sample losses of each positive sample. The sample loss of any positive sample is the negative logarithm of the ratio of the matching degree index between the segment vector output by the multimodal encoder for the video segment in the positive sample and the question vector of the question text in the positive sample to the sum of the matching degree index values ​​of all positive samples and weighted negative samples. The closer the video segment in any negative sample is to the time interval where the evidence required by the corresponding question text is located, the smaller the weight of the negative sample.

[0027] Here, the received video understanding request includes the video to be understood and the question text. In some scenarios, it may also include several candidate answer options corresponding to the question text. The video to be understood can be a long video. Subsequently, the video to be understood is divided into multiple video segments according to a preset duration Δt. The preset duration Δt can be from 1 second to 60 seconds, preferably 15 seconds, or it can be dynamically determined based on camera transitions, voice pauses, subtitle segments, or event boundaries. For segments at the end of the video whose duration is less than Δt, they can be merged into the previous video segment when their duration is below a threshold, or they can be retained as independent segments. This embodiment of the invention does not specifically limit this.

[0028] For each video segment, visual frames and audio, along with subtitles or text obtained from automatic speech recognition of the audio, are extracted. A pre-trained multimodal encoder encodes these visual frames, subtitles, or text to obtain a segment vector for that video segment. In some embodiments, the multimodal encoder may include a visual encoder, an audio encoder, a language model backbone network, a position-weighted pooling module, and a projection layer to provide understanding and encoding capabilities for multimodal data. The encoded video segment and its timestamp can be cached offline to support the reuse of multiple questions from the same video. Furthermore, for the aforementioned question text, a question vector is extracted using a language model. Subsequently, the question vector is matched with the segment vectors of each video segment to construct a coarse search candidate pool based on the video segments whose segment vectors match the question vector. Specifically, the Kr video segments with the highest matching degree between their segment vectors and question vectors can be selected to construct the coarse search candidate pool. Kr can be adaptively set according to video length, frame budget, or system latency requirements, preferably ranging from 5 to 100, and more preferably from 20 to 50. This phase focuses on recall, ensuring that as many potential evidence fragments as possible enter the coarse search candidate pool.

[0029] In some embodiments, the question vector of the aforementioned question text q and any video clip fragment vector When performing matching, the problem vector and fragment vector can be normalized first based on the following formula to obtain the normalized problem vector. and normalized fragment vector : ; Subsequently, the cosine similarity between the normalized problem vector and the normalized segment vector is calculated as the degree of matching between the two.

[0030] After constructing the coarse search candidate pool, since there may still be a large number of "similar interference segments" in the video clips selected solely based on their similarity to the question text—segments with the same scene, characters, objects, or actions—these segments are semantically highly relevant to the question but do not contain the decisive evidence needed to answer it. Therefore, to improve the accuracy of subsequent video understanding, this embodiment of the invention also utilizes a multimodal validation model to evaluate the answer utility score of each candidate video clip in the coarse search candidate pool. The higher the answer utility score of any candidate video clip, the greater the likelihood that the candidate video clip contains the evidence needed to answer the question text. Here, the multimodal validation model can be any large multimodal model capable of receiving video clips and text questions, such as a visual language model, an audio-visual language model, or a distilled small multimodal validator; this embodiment of the invention does not specifically limit this.

[0031] In some embodiments, to obtain the answer utility score for each candidate video segment, a cue word containing the corresponding candidate video segment, the aforementioned question text, and the task requirement is constructed for each candidate video segment. This cue word is then input into a multimodal validation model to obtain the original output of the multimodal validation model. The task requirement instructs the model to determine whether the input video segment contains the evidence needed to answer the input question text. The original output is the original output score (logit) for each corresponding word before softmax calculation. Here, the multimodal validation model does not require a direct final answer but rather outputs a segment-level usefulness judgment. For example, the task requirement could be: "Determine whether this video segment contains the information necessary to answer the question. If helpful, answer Yes; otherwise, answer No." Subsequently, based on the original output scores of affirmative category words (e.g., Yes, yes, and True) and negative category words (e.g., No, no, and False) in the original output of each candidate video segment, the answer utility score for the corresponding candidate video segment is calculated. Here, the higher the original output score of positive category words, or the lower the original output score of negative category words, the higher the answer utility score of the corresponding candidate video segment.

[0032] In other embodiments, the answer utility score for candidate video segment i can be calculated based on the following formula: ; ; Where Y and N are the sets of affirmative category lexical units and the sets of negative category lexical units, respectively; To affirm the original output score of the category term y, The original output score for the negation category term n; is the answer utility score of candidate video segment i.

[0033] Next, by comprehensively considering the matching degree between the segment vector of each candidate video segment and the question vector as well as the answer utility score of each candidate video segment, video segments that are not only highly semantically relevant to the question text but also more likely to contain decisive evidence required for answering the question are screened out, which can improve the accuracy of evidence selection and the accuracy of subsequent video understanding.

[0034] In some embodiments, the fusion score of each candidate video segment may be calculated based on the following formula: ; wherein, is the Z-score normalization function; is a preset weight, , The larger the value is, the more the system emphasizes the judgment of the multi-modal verification model on answer utility; The smaller the value is, the more the system emphasizes semantic relevance. In some embodiments, when the evidence in a long video is sparse and there are many interfering segments, increase , and in scenarios of short videos or dense evidence, decrease ; is the answer utility score of candidate video segment i, is the matching degree between the segment vector of candidate video segment i and the aforementioned question vector, is the fusion score of candidate video segment i.

[0035] Subsequently, based on the fusion scores of the respective candidate video segments, Ks candidate video segments with the highest fusion scores are screened out as target video segments. Wherein Ks<Kr, the preferred range is 2 to 30, and the more preferred range is 5 to 12.

[0036] In some embodiments, in scenarios that require cross-segment understanding, for segments with a fusion score higher than a threshold, several context segments can be extended to their adjacent time segments; for segments with time overlap or content repetition, deduplication may be performed according to the time intersection-over-union ratio, vector similarity or shot repetition degree.

[0037] As can be seen, the selection of target video segments is based on the matching degree between the segment vectors of each candidate video segment in the coarse search candidate pool and the question vector of the question text. Specifically, the matching degree between the segment vector of the video segment and the question vector of the question text directly affects the calculation of the fusion score. Only video segments with a high matching degree are likely to obtain a higher fusion score and thus be selected as target video segments. Therefore, in order to further avoid selecting interference segments that are highly relevant to the question text in semantics but do not contain the decisive evidence needed to answer the question, efforts are made at the multimodal encoder. Through a special training method, the multimodal encoder has a high ability to distinguish between the interference segments and the correct segments that truly contain the decisive evidence needed to answer the question. Thus, discriminative encoding is achieved in the encoding step. This discriminativeness is further reflected in the matching degree between the corresponding segment vector and the question vector, so that the matching degree of the correct segment is higher than that of the interference segment.

[0038] Specifically, during the training of the multimodal encoder, a model loss is calculated based on positive and negative samples, and the parameters of the multimodal encoder are adjusted using this model loss. Positive samples include the question text and a video clip containing evidence needed to answer the question text; correspondingly, negative samples include the question text and a video clip that does not contain evidence needed to answer the question text.

[0039] In some embodiments, multiple training videos can be obtained, and for each training video, a question can be obtained from that training video. Corresponding evidence time interval This interval can be derived from manually labeled data, timestamps from existing video question-answering datasets, weakly supervised alignment results, or large model-assisted labeling results. The training video is segmented into segments with fixed or dynamic durations. Audio is extracted from each segment, and automatic speech recognition is performed to obtain the corresponding speech-to-text. Next, a visual language model is used to combine the visual content of the segments with the speech-to-text to generate dense segment descriptions. These descriptions include details that aid in retrieval and differentiation, such as people, objects, colors, actions, subtitles, scene changes, and audio content. Subsequently, positive and negative samples are constructed. Specifically, for each segment... Calculate its relationship with the evidence interval. Overlap .when If the number of positive samples exceeds a preset threshold, the segment is considered a positive sample; if the number of positive samples is too large, the previous segment is selected based on the overlap duration or evidence coverage ratio. There are one positive sample. Additionally, negative samples are constructed from segments that do not overlap with the evidence time interval. Preferably, segments close to the evidence interval and semantically similar are considered hard negative samples, and segments far from the evidence interval are considered moderate negative samples. Alternatively, cross-video segments can be introduced as weak negative samples. This design enables the model to learn to distinguish similar but invalid interference evidence within the same video.

[0040] Here, the model loss is the average of the sample losses of each positive sample, and the sample loss of any positive sample is the negative logarithm of the ratio of the matching index between the segment vector output by the multimodal encoder for that video segment and the question vector of the question text in that positive sample to the sum of the matching indexes of all positive samples and weighted negative samples. Here, the closer the video segment in any negative sample is to the time interval of the evidence required by the corresponding question text, the smaller the weight of that negative sample. By setting positive samples and weighted negative samples for comparative learning, the multimodal encoder can distinguish between interference segments and correct segments during training, and then perform differential encoding, so that correct segments achieve a higher matching degree compared to interference segments. Simultaneously, for segments that are temporally close to the correct segments, since they may contain context or some relevant information, setting smaller or softened weights for them can also result in a higher matching degree for these segments, thus allowing them to be filtered into the coarse search candidate pool and avoiding the omission of evidence.

[0041] In some embodiments, the sample loss for any positive sample is: ; Where k is a positive sample, The sample loss for positive sample k, To compare the learning temperature coefficients, P represents the set of positive samples, and N represents the set of negative samples. This represents the matching degree between the segment vector output by the multimodal encoder for a video segment in positive sample k and the question vector of the question text in positive sample k. This represents the matching degree between the segment vector output by the multimodal encoder for a video segment in positive sample p and the question vector of the question text in positive sample p. This represents the matching degree between the segment vector output by the multimodal encoder for a video segment in negative sample j and the question vector of the question text in negative sample j. The weights are for negative sample j.

[0042] In other embodiments, the weight of any negative sample is calculated based on the following formula: ; ; In this negative sample, the evidence required for the question text is located in the time interval [a, b] of the original video. and Let these be the start and end times of the video segment in the negative sample. For preset coefficients, This represents the weight of the negative sample. It should be noted that... Bucketing functions, linear functions, or other learnable functions can also be used. For example, when using a bucketing function, if... When less than a fragment length Take the smaller weight. When it is between one and three segment lengths Take a moderate weight. When the length is greater than three segments Take the larger weight.

[0043] After selecting Ks target video segments, considering the limitations of the context window, memory, and computational overhead of the visual language model, these Ks target video segments can be sampled, and a number of frames can be extracted for subsequent video understanding. To further improve the accuracy of video understanding, in some embodiments, more video frames will be extracted from the target video segments that are more likely to contain the evidence needed to answer the above-mentioned question text. Specifically, this is based on the video frame budget B, the number of target video segments Ks, and the minimum number of frames extracted from each target video segment. Calculate the number of adjustable frames R that satisfies the condition that the number of video frames extracted from each target video segment is greater than or equal to the minimum number of frames. .

[0044] Based on the fusion score of each target video segment, the allocation ratio of each target video segment is determined; the higher the fusion score of any target video segment, the higher the allocation ratio of that target video segment. Subsequently, based on the allocation ratio of each target video segment and the number of available frames, the number of additional frames allocated to each target video segment is determined, and the sum of the number of additional frames allocated to each target video segment and the minimum number of frames is determined as the final number of extracted frames for each target video segment.

[0045] In some embodiments, the allocation ratio of each target video segment is calculated based on the following formula: ; in, The allocation ratio for target video segment i. Let i be the fusion score of the target video segment i. For the target set of video clips, Assign a temperature coefficient to the frame; The number of additional frames allocated to each target video segment is determined based on the following formula, so that the final number of extracted frames for each target video segment meets the preset condition: ; in, The number of extra frames allocated to target video segment i. For the adjustable number of frames, The maximum number of frames that can be allocated. The minimum number of frames, For use in filling in The integer terms of the rounded residual; The preset condition is that the sum of the final extracted frames of each target video segment is equal to the video frame budget, and the final extracted frame number of each target video segment is less than or equal to the maximum frame number.

[0046] Finally, based on the final extracted frame count of each target video segment, video frame sampling is performed on the corresponding target video segment. The video frames sampled from each target video segment are then concatenated and input together with the aforementioned question text into the visual language response model to obtain the answer text output by the visual language response model. It can be seen that, as... Figure 2 As shown, compared with the existing uniform frame extraction method or the frame extraction method based on semantic relevance to the problem, the embodiments of the present invention can effectively eliminate interference items, thereby improving the accuracy of video understanding.

[0047] In summary, the method provided by this invention decouples coarse retrieval from fragment answer utility verification. First, it matches the question vector with the fragment vectors of each video segment, constructing a coarse candidate pool based on the video segments matching the question vector to ensure recall of candidate evidence. Then, a multimodal verification model determines whether the candidate fragments truly contribute to the answer, effectively filtering similar interfering fragments from the same scene, person, or object, explicitly mitigating the inconsistency between "relevance" and "answer utility." More importantly, it uses positive samples and weighted negative samples for comparative learning, enabling the multimodal encoder to distinguish highly similar fragments with different answer utilities. This achieves discriminative encoding at the encoding step, ensuring that the matching degree of correct fragments is higher than that of interfering fragments. This further avoids selecting interfering fragments that are semantically highly relevant to the question text but do not contain the decisive evidence needed to answer the question. This improves the accuracy of evidence selection and video understanding even with limited context windows, memory, and computational overhead. Furthermore, this scheme can be used with different visual language answer models without requiring fine-tuning of the answer model, making it easy to insert as an external evidence selection module into existing systems, offering high flexibility and scalability.

[0048] Furthermore, by using fusion scores to guide frame budget allocation, more frames can be obtained by efficiently using fragments, while fewer frames can be obtained or fragments can be discarded by inefficiently using fragments. This can avoid the budget waste caused by uniform frame extraction and improve the coverage of key evidence under a fixed frame budget.

[0049] The video understanding adaptive evidence selection system provided by the present invention is described below. The video understanding adaptive evidence selection system described below and the video understanding adaptive evidence selection method described above can be referred to in correspondence.

[0050] This invention provides a video understanding adaptive evidence selection system, comprising: The request parsing unit is used to receive video understanding requests and extract the video to be understood and the question text from the video understanding requests. The video encoding unit is used to divide the video to be understood into multiple video segments according to a preset duration, and to encode each video segment based on a trained multimodal encoder to obtain the segment vector of the corresponding video segment. The coarse retrieval unit is used to extract the question vector of the question text, and match the question vector with the segment vector of each video segment, and construct a coarse retrieval candidate pool based on the video segments that match the question vector; The answer utility evaluation unit is used to evaluate the answer utility score of each candidate video segment in the coarse search candidate pool using a multimodal verification model; wherein, the higher the answer utility score of any candidate video segment, the greater the likelihood that the candidate video segment contains the evidence required for the question text; An evidence selection unit is used to determine a target video segment containing the evidence required for the question text based on the matching degree between the segment vector of each candidate video segment and the question vector, as well as the answer utility score of each candidate video segment. The multimodal encoder is trained based on the model loss calculated from positive and negative samples. The model loss is the average of the sample losses of each positive sample. The sample loss of any positive sample is the negative logarithm of the ratio of the matching degree index between the segment vector output by the multimodal encoder for the video segment in the positive sample and the question vector of the question text in the positive sample to the sum of the matching degree index values ​​of all positive samples and weighted negative samples. The closer the video segment in any negative sample is to the time interval where the evidence required by the corresponding question text is located, the smaller the weight of the negative sample.

[0051] This invention provides an electronic device, including: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0052] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0053] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0054] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video understanding adaptive evidence selection method, characterized in that, include: Receive a video understanding request and extract the video to be understood and the question text from the video understanding request; The video to be understood is divided into multiple video segments according to a preset duration, and each video segment is encoded based on a trained multimodal encoder to obtain the segment vector of the corresponding video segment. Extract the question vector from the question text, and match the question vector with the segment vector of each video segment. Construct a coarse search candidate pool based on the video segments that match the question vector. The answer utility score of each candidate video segment in the coarse search candidate pool is evaluated using a multimodal verification model; wherein, the higher the answer utility score of any candidate video segment, the greater the likelihood that the candidate video segment contains the evidence required for the question text; Based on the matching degree between the segment vector of each candidate video segment and the question vector, and the answer utility score of each candidate video segment, the target video segment containing the evidence required for the question text is determined. The multimodal encoder is trained based on the model loss calculated from positive and negative samples. The model loss is the average of the sample losses of each positive sample. The sample loss of any positive sample is the negative logarithm of the ratio of the matching degree index between the segment vector output by the multimodal encoder for the video segment in the positive sample and the question vector of the question text in the positive sample to the sum of the matching degree index values ​​of all positive samples and weighted negative samples. The closer the video segment in any negative sample is to the time interval where the evidence required by the corresponding question text is located, the smaller the weight of the negative sample.

2. The video understanding adaptive evidence selection method as described in claim 1, characterized in that, The sample loss for any positive sample is: ； Where k is a positive sample, The sample loss for positive sample k, To compare the learning temperature coefficients, P represents the set of positive samples, and N represents the set of negative samples. The degree of matching between the segment vector output by the multimodal encoder for a video segment in positive sample k and the question vector of the question text in positive sample k. The degree of matching between the segment vector output by the multimodal encoder for a video segment in positive sample p and the question vector of the question text in positive sample p. The degree of matching between the segment vector output by the multimodal encoder for the video segment in negative sample j and the question vector of the question text in negative sample j. The weights are for negative sample j.

3. The video understanding adaptive evidence selection method as described in claim 2, characterized in that, The weight of any negative sample is calculated based on the following formula: ； ； The time interval for the evidence required for the problematic text in this negative sample is [a, b]. and Let these be the start and end times of the video segment in the negative sample. For preset coefficients, This represents the weight of the negative sample.

4. The video understanding adaptive evidence selection method as described in any one of claims 1 to 3, characterized in that, The evaluation of the answer utility score of each candidate video segment in the coarse search candidate pool using a multimodal validation model specifically includes: For each candidate video segment in the coarse search candidate pool, a prompt word is constructed containing the corresponding candidate video segment, the question text, and the task requirement. The prompt word is then input into a multimodal validation model to obtain the original output of the multimodal validation model. The task requirement instructs the model to determine whether the input video segment contains the evidence required to answer the input question text. Based on the original output scores of affirmative category words and negative category words in the original output results of each candidate video segment, the answer utility score of the corresponding candidate video segment is calculated.

5. The video understanding adaptive evidence selection method as described in claim 4, characterized in that, The utility score of the answer for candidate video segment i is calculated based on the following formula: ； ； Where Y and N are the sets of affirmative category lexical units and the sets of negative category lexical units, respectively; To affirm the original output score of the category term y, The original output score for the negation category term n; Let i be the utility score of the answer for candidate video segment i.

6. The video understanding adaptive evidence selection method as described in any one of claims 1 to 3, characterized in that, Based on the matching degree between the segment vectors of each candidate video segment and the question vector, and the answer utility score of each candidate video segment, the target video segment containing the evidence required for the question text is determined, specifically including: The fusion score for each candidate video segment is calculated based on the following formula: ； in, For Z-score normalization function, To preset weights, Let i be the utility score of the answer for candidate video segment i. Let be the matching degree between the segment vector of candidate video segment i and the question vector. The fusion score of candidate video segment i; Based on the fusion scores of each candidate video segment, multiple target video segments are selected.

7. The video understanding adaptive evidence selection method as described in claim 6, characterized in that, The process of determining the target video segment containing the evidence required for the question text further includes: Based on the video frame budget, the number of target video segments, and the minimum number of frames extracted from each target video segment, calculate the number of adjustable frames that satisfy the condition that the number of video frames extracted from each target video segment is greater than or equal to the minimum number of frames. Based on the fusion score of each target video segment, the allocation ratio of each target video segment is determined; wherein, the higher the fusion score of any target video segment, the higher the allocation ratio of that target video segment. Based on the allocation ratio of each target video segment and the number of adjustable frames, the number of additional frames allocated to each target video segment is determined, and the sum of the number of additional frames allocated to each target video segment and the minimum number of frames is determined as the final number of extracted frames for each target video segment. Based on the final extracted frame number of each target video segment, video frames are sampled from the corresponding target video segments. The video frames sampled from each target video segment are then concatenated and input together with the question text into the visual language response model to obtain the answer text output by the visual language response model.

8. The video understanding adaptive evidence selection method as described in claim 7, characterized in that, The allocation ratio of each target video segment is calculated based on the following formula: ； in, The allocation ratio for target video segment i. Let i be the fusion score of the target video segment i. For the target set of video clips, Assign a temperature coefficient to the frame; The number of additional frames allocated to each target video segment is determined based on the following formula, so that the final number of extracted frames for each target video segment meets the preset condition: ； in, The number of extra frames allocated to target video segment i. For the adjustable number of frames, The maximum number of frames that can be allocated. The minimum number of frames, For use in filling in The integer terms of the rounded residual; The preset condition is that the sum of the final extracted frames of each target video segment is equal to the video frame budget, and the final extracted frame number of each target video segment is less than or equal to the maximum frame number.

9. A video understanding adaptive evidence selection system, characterized in that, include: The request parsing unit is used to receive video understanding requests and extract the video to be understood and the question text from the video understanding requests. The video encoding unit is used to divide the video to be understood into multiple video segments according to a preset duration, and to encode each video segment based on a trained multimodal encoder to obtain the segment vector of the corresponding video segment. A coarse retrieval unit is used to extract the question vector from the question text and match the question vector with the segment vector of each video segment. A coarse retrieval candidate pool is constructed based on the video segments that match the question vector. The answer utility evaluation unit is used to evaluate the answer utility score of each candidate video segment in the coarse search candidate pool using a multimodal verification model; wherein, the higher the answer utility score of any candidate video segment, the greater the likelihood that the candidate video segment contains the evidence required for the question text; An evidence selection unit is used to determine a target video segment containing the evidence required for the question text based on the matching degree between the segment vector of each candidate video segment and the question vector, as well as the answer utility score of each candidate video segment. The multimodal encoder is trained based on the model loss calculated from positive and negative samples. The model loss is the average of the sample losses of each positive sample. The sample loss of any positive sample is the negative logarithm of the ratio of the matching degree index between the segment vector output by the multimodal encoder for the video segment in the positive sample and the question vector of the question text in the positive sample to the sum of the matching degree index values ​​of all positive samples and weighted negative samples. The closer the video segment in any negative sample is to the time interval where the evidence required by the corresponding question text is located, the smaller the weight of the negative sample.

10. An electronic device, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the video understanding adaptive evidence selection method as described in any one of claims 1-8.

Images