A video understanding method and system based on multi-modal evidence chain

Abstract

The application relates to a video understanding method and system based on a multimodal evidence chain, and the method comprises the following steps: acquiring user input question text, an option set and a target video; using the question text and the option set to form a complementary analysis angle set; performing frame-by-frame matching on the target video based on a preset angle feature mapping library to obtain a key timestamp set corresponding to different analysis angles; extracting a corresponding key video frame set from the target video; scoring each key video frame in the key video frame set to construct an angle-frame mapping table between different analysis angles and corresponding visual evidence frames; acquiring to-be-reasoned text that needs visual evidence, associating the to-be-reasoned text with the angle-frame mapping table to construct a multimodal evidence chain; and obtaining a video reasoning result of the question text based on the multimodal evidence chain. The video understanding method and system realize high-precision and strong-explainable video understanding and reduce the dependence of video understanding on large-scale labeled data.

Description

Technical Field

[0001] This invention belongs to the field of multimodal intelligent understanding technology, and in particular relates to a video understanding method and system based on multimodal evidence chains. Background Technology

[0002] With the rapid development of short video platforms, intelligent monitoring, and autonomous driving, the demand for intelligent understanding of video content is becoming increasingly urgent.

[0003] In practical applications, users often need to deeply interpret video content and obtain accurate answers. Traditional methods rely heavily on large-scale, finely labeled data, and the reasoning process lacks interpretability, making it difficult to trace the evidence path "from video to answer".

[0004] Therefore, there is an urgent need for a multimodal video understanding solution that requires no training or loss function, can be deployed locally, and has a chain of evidence. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a video understanding method and system based on a multimodal evidence chain.

[0006] The technical solution adopted in this invention is:

[0007] Firstly, a video understanding method based on a multimodal evidence chain is provided, characterized by comprising:

[0008] Obtain the user's input question text, option set, and target video;

[0009] Use the question text and the set of options to form a set of complementary analytical perspectives;

[0010] Based on a pre-set angle feature mapping library, the target video is matched frame by frame to obtain the set of key timestamps corresponding to different analysis angles in the complementary analysis angle set.

[0011] Extract the corresponding set of key video frames from the target video based on the set of key timestamps;

[0012] Each key video frame in the key video frame set is scored, and multiple key video frames are selected as visual evidence frames according to the scores from high to low, thus constructing an angle-frame mapping table between different analysis angles and corresponding visual evidence frames.

[0013] Obtain the text to be inferred that requires visual corroboration, associate the text to be inferred with the angle-frame mapping table, and construct a multimodal evidence chain;

[0014] The video reasoning results of the question text are obtained based on the multimodal evidence chain.

[0015] Furthermore, a complementary set of analytical perspectives is formed using the question text and the set of options, including:

[0016] The question text Q is encoded into a question semantic vector using a semantic encoder in a pre-built large language model. Problem semantic vector The expression is: ;

[0017] Based on the question semantic vector Obtain the probability of angle categories from different analytical perspectives Angle category probability The expression is:

[0018] ;

[0019] in, Represents the set of analytical perspectives. , This represents the k-th analytical perspective; To preset the semantic vector weight values ​​for the problem, Fixed parameters for the preset question semantics;

[0020] Encode the option set C into option semantic vectors. option semantic vector The expression is: ;

[0021] Calculate the similarity vector between any two options i and j in option set C. Similarity vector The expression is:

[0022] ;

[0023] in, , , , This represents the option semantic vector of the i-th option. The option semantic vector representing the j-th option;

[0024] Based on similarity vector Differential expression Differential expression The expression is:

[0025] ;

[0026] in, The preset option semantic vector weight values, Specifically, it is in matrix form. The semantic parameters are fixed for the preset options. Specifically, it is in vector form;

[0027] Based on the preset complementary fusion rules, the angle category probability is... Differential expression By integrating these elements, a set of complementary analytical perspectives can be obtained. complementary analysis perspectives The expression is:

[0028] ;

[0029] in, To pre-define the complementary fusion function, This is the preset filtering threshold.

[0030] Furthermore, based on a pre-set angle feature mapping library, frame-by-frame matching of the target video is performed to obtain a set of key timestamps for different analysis angles in the complementary analysis angle set, including:

[0031] Through a pre-trained angle feature encoder From the perspective of analysis Encode the corresponding visual feature vectors. , , Represent real numbers, Represents visual feature vectors The dimension;

[0032] Construct a pre-defined angle feature mapping library The visual feature dictionary is stored in the preset angle feature mapping library. Visual feature dictionary The expression is:

[0033] ;

[0034] The general visual feature model is used to extract features from each video frame in the target video to obtain the frame features of each video frame.

[0035] visual feature dictionary Matching with each frame feature yields a matching degree sequence. , , This represents the matching degree value between the visual features of the k-th analysis angle and the frame features of the t-th video frame. m is the total number of video frames in the target video;

[0036] The expression is:

[0037] ;

[0038] in, The frame characteristics of the t-th video frame, representing the timestamp Time;

[0039] According to the descending order of matching degree value and matching threshold The top N video frames are selected as keyframes, and a set of key timestamps corresponding to the k-th analysis angle is generated based on the timestamps (Time) of the top N video frames. Key timestamp set The expression is:

[0040] ;

[0041] This indicates that the t-th video frame in the target video is a keyframe.

[0042] Furthermore, each key video frame in the key video frame set is scored, and multiple key video frames are selected as visual evidence frames according to their scores from high to low. An angle-frame mapping table is constructed between different analysis angles and their corresponding visual evidence frames, including:

[0043] Based on a pre-set multi-dimensional scoring mechanism, each key video frame in the key video frame set is scored to obtain the score value of each key video frame under the corresponding analysis angle.

[0044] Following the descending order of score values ​​under the same analytical perspective, the Top M keyframes corresponding to each analytical perspective are selected as visual evidence frames; M is less than N.

[0045] An angle-frame mapping table is constructed based on the visual evidence frames and scoring values ​​corresponding to different analysis angles.

[0046] Furthermore, the text to be inferred, which requires visual corroboration, is obtained, and then associated with an angle-frame mapping table to construct a multimodal evidence chain, including:

[0047] Obtain the text to be inferred that requires visual corroboration;

[0048] Perform textual analysis on the text to be reasoned and determine the target analytical angle;

[0049] Extract the target visual evidence frames related to the target analysis angle from the angle-frame mapping table;

[0050] A multimodal evidence chain is constructed based on the text to be reasoned and the target visual evidence frame.

[0051] Furthermore, the methods also include:

[0052] Image description text is generated after image recognition of the target visual evidence frame;

[0053] Add the image description text as evidence to the multimodal evidence chain.

[0054] Furthermore, based on the multimodal evidence chain, the video reasoning results of the question text are obtained, including:

[0055] Evidence features are obtained based on multimodal evidence chains. These evidence features are then matched with different options in the option set, and the option with the highest matching degree is selected as the answer to the question.

[0056] The average score of all visual evidence frames corresponding to different analysis angles in the multimodal evidence chain is obtained, and the evidence strength value is obtained by pre-weighted calculation.

[0057] Confidence level is derived from the strength of evidence;

[0058] The video inference results are obtained based on the answers to the questions and the confidence levels.

[0059] Secondly, a video understanding system based on a multimodal evidence chain is provided, including:

[0060] The acquisition module is used to acquire the user's input question text, option set, and target video;

[0061] The analysis perspective generation module is used to generate a complementary set of analysis perspectives using the question text and the set of options.

[0062] The timestamp acquisition module is used to perform frame-by-frame matching of the target video based on a preset angle feature mapping library to obtain the set of key timestamps corresponding to different analysis angles in the complementary analysis angle set.

[0063] The video frame extraction module is used to extract the corresponding set of key video frames from the target video based on the set of key timestamps.

[0064] The visual evidence acquisition module is used to score each key video frame in the key video frame set, and select multiple key video frames as visual evidence frames according to the scores from high to low, and construct an angle-frame mapping table between different analysis angles and corresponding visual evidence frames.

[0065] The evidence chain construction module is used to obtain the text to be reasoned that requires visual corroboration, associate the text to be reasoned with the angle-frame mapping table, and construct a multimodal evidence chain.

[0066] The video reasoning module is used to obtain video reasoning results for the question text based on a multimodal evidence chain.

[0067] The beneficial effects achieved by this invention are as follows:

[0068] The process involves: acquiring user-input question text, a set of options, and a target video; forming a complementary analysis angle set using the question text and option set; performing frame-by-frame matching on the target video based on a pre-defined angle feature mapping library to obtain a set of key timestamps corresponding to different analysis angles in the complementary analysis angle set; extracting a set of corresponding key video frames from the target video based on the key timestamp set; scoring each key video frame in the key video frame set and selecting multiple key video frames as visual evidence frames according to their scores from high to low, thus constructing an angle-frame mapping table between different analysis angles and their corresponding visual evidence frames; acquiring the text to be reasoned that requires visual corroboration and associating it with the angle-frame mapping table to construct a multimodal evidence chain; and obtaining the video reasoning result of the question text based on the multimodal evidence chain. This multi-stage processing framework—"analysis angle generation - key timestamp location - key video frame extraction - visual evidence frame selection - multimodal evidence chain construction - video reasoning"—achieves high-precision and highly interpretable video understanding, reducing the dependence of video understanding on large-scale labeled data. Attached Figure Description

[0069] Figure 1 This is a flowchart of the video understanding method based on multimodal evidence chains of the present invention;

[0070] Figure 2 This is a flowchart illustrating the process of constructing the multimodal evidence chain of this invention;

[0071] Figure 3 This is a structural diagram of the video understanding system based on multimodal evidence chains according to the present invention. Detailed Implementation

[0072] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0073] like Figure 1 As shown, this embodiment of the invention provides a video understanding method based on a multimodal evidence chain, including:

[0074] 101. Obtain the user-input question text, option set, and target video;

[0075] In this embodiment, the main application scenarios can be intelligent video question answering, content review assistance, and human-machine collaborative annotation, etc., which require machine assistance for video understanding.

[0076] For example, intelligent video Q&A mainly provides accurate content query in fields such as education and security, such as "identifying the items carried by the people in the surveillance"; content review assistance mainly provides compliance verification for the media industry and generates a traceable evidence package of "violation pictures + text descriptions"; human-machine collaborative annotation mainly provides a pre-generated evidence chain for annotators to improve the efficiency and consistency of video annotation;

[0077] The question text is the video understanding question of the user for the target video, and the user needs to input it in the form of text or speech-to-text. For example, the question text can be "What are the people in the video doing?";

[0078] The option set is the possible question answers for the question text. For example, the option set includes multiple options "A. Cooking; B. Making a phone call; C. Walking; D. Reading a book".

[0079] 102. Use the question text and the option set to form a complementary analysis angle set;

[0080] Encode the question text Q into a question semantic vector through the semantic encoder in the pre-set large language model , the question semantic vector The expression of is: ;

[0081] According to the question semantic vector Obtain the angle category probabilities of different analysis angles ; The angle category probability The expression of is:

[0082] ;

[0083] Among them, Represents the analysis angle set, , Represents the k-th analysis angle; Is the preset question semantic vector weight value, Is the preset question semantic fixed parameter;

[0084] Encode the option set C into an option semantic vector , the option semantic vector The expression of is: ;

[0085] Calculate the similarity vector between any two options i and j in the option set C , the similarity vector The expression of is:

[0086] ;

[0087] Among them, , , , This represents the option semantic vector of the i-th option. The option semantic vector representing the j-th option;

[0088] Based on similarity vector Differential expression Differential expression The expression is:

[0089] ;

[0090] in, The preset option semantic vector weight values, Specifically, it is in matrix form. The semantic parameters are fixed for the preset options. Specifically, it is in vector form;

[0091] Based on the preset complementary fusion rules, the angle category probability is... Differential expression By integrating these elements, a set of complementary analytical perspectives can be obtained. complementary analysis perspectives The expression is:

[0092] ;

[0093] in, To pre-define the complementary fusion function, The preset filtering threshold;

[0094] By using precise prompt engineering to guide the analysis of the core issues, and designing non-overlapping analysis dimensions based on the differences in options, we ensure coverage of all scenario features such as "behavior-environment-interaction".

[0095] Based on the preset complementary fusion rules, the angle category probability is... Differential expression By integrating these elements, a set of complementary analytical perspectives can be obtained. Specific complementary analytical perspectives can include "character behavior cues," "environment and scene objects," and "interaction / sound cues," etc.

[0096] 103. Based on the preset angle feature mapping library, the target video is matched frame by frame to obtain the set of key timestamps corresponding to different analysis angles in the complementary analysis angle set;

[0097] Through a pre-trained angle feature encoder From the perspective of analysis Encode the corresponding visual feature vectors. , , Represent real numbers, Represents visual feature vectors The dimension;

[0098] Construct a pre-defined angle feature mapping library The visual feature dictionary is stored in the preset angle feature mapping library. Visual feature dictionary The expression is:

[0099] ;

[0100] Specifically, for different analytical perspectives, pre-designed feature matching rules can be used, such as matching "human movement change rate" from a behavioral perspective and matching "scene object integrity" from an environmental perspective.

[0101] The general visual feature model is used to extract features from each video frame in the target video to obtain the frame features of each video frame.

[0102] visual feature dictionary Matching with each frame feature yields a matching degree sequence. , , This represents the matching degree value between the visual features of the k-th analysis angle and the frame features of the t-th video frame. m is the total number of video frames in the target video;

[0103] The expression is:

[0104] ;

[0105] in, The frame characteristics of the t-th video frame, representing the timestamp Time;

[0106] According to the descending order of matching degree value and matching threshold The top N video frames are selected as keyframes, and a set of key timestamps corresponding to the k-th analysis angle is generated based on the timestamps (Time) of the top N video frames. Key timestamp set The expression is:

[0107] ;

[0108] This indicates that the t-th video frame in the target video is a keyframe.

[0109] This step filters out the most representative key timestamps from the target video to reduce invalid frame processing and improve processing efficiency.

[0110] 104. Extract the corresponding set of key video frames from the target video based on the set of key timestamps;

[0111] After obtaining the key timestamp set in step 103, the key video frames corresponding to the key timestamps can be extracted from the target video to form a key video frame set.

[0112] In addition, key video frames can be preprocessed to improve their clarity, which facilitates the selection of visual evidence in the future.

[0113] 105. Each key video frame in the key video frame set is scored, and multiple key video frames are selected as visual evidence frames according to the scores from high to low, and an angle-frame mapping table between different analysis angles and corresponding visual evidence frames is constructed.

[0114] In this embodiment, the key video frame set contains multiple key video frames. High-value video frames need to be selected as visual evidence. The specific process for constructing the angle-frame mapping table is as follows:

[0115] Based on a pre-set multi-dimensional scoring mechanism, each key video frame in the key video frame set is scored to obtain the score value of each key video frame under the corresponding analysis angle; the pre-set multi-dimensional scoring mechanism is specifically a scoring of three dimensions: "clarity (40%) + relevance (50%) + uniqueness (10%)".

[0116] Following the descending order of score values ​​under the same analytical perspective, the Top M keyframes corresponding to each analytical perspective are selected as visual evidence frames; M is less than N.

[0117] An angle-frame mapping table is constructed based on the visual evidence frames and scoring values ​​corresponding to different analysis angles.

[0118] 106. Obtain the text to be reasoned that requires visual corroboration, associate the text to be reasoned with the angle-frame mapping table, and construct a multimodal evidence chain.

[0119] In this embodiment, the process of constructing the multimodal chain of evidence is as follows: Figure 2 As shown, it includes the following steps:

[0120] 201. Obtain the text to be inferred that requires visual evidence;

[0121] When constructing a chain of evidence, it is necessary to identify the text to be inferred that requires visual corroboration in the initial text sentences. The text to be inferred generally accounts for 1 / 3 or at least 3 sentences of the total number of sentences; for example, the text to be inferred is "a person holding a spatula".

[0122] 202. Conduct textual analysis on the text to be reasoned and determine the target analytical angle;

[0123] By conducting textual analysis on the deduction text "the character holding a spatula," we can determine the target analysis angles, including behavioral clues and environmental objects.

[0124] 203. Extract the target visual evidence frames related to the target analysis angle from the angle-frame mapping table;

[0125] Assume the target visual evidence frame is a 0:12 video frame, and the content of the video frame shows the action of a spatula.

[0126] 204. A multimodal evidence chain is constructed based on the text to be reasoned and the target visual evidence frame;

[0127] 205. After performing image recognition on the target visual evidence frame, generate image description text;

[0128] 206. Add the image description text as evidence to the multimodal evidence chain.

[0129] The final multimodal evidence chain is specifically constructed as "person holding a spatula → 0:12 video frame → displaying spatula action", forming a multimodal evidence chain of "sentence - evidence image - evidence description".

[0130] 107. Video reasoning results of the problem text obtained based on multimodal evidence chains.

[0131] In this embodiment, evidence features are obtained based on a multimodal evidence chain. Taking the multimodal evidence chain in step 206 above as an example, the evidence feature can be "the action of a person holding a spatula".

[0132] The evidence feature "the action of a person holding a spatula" is matched with different options in the option set "A. cooking; B. making a phone call; C. walking; D. reading". The option "A. cooking" with the highest matching degree is selected as the answer to the question.

[0133] After reasoning out the answer to the question, it is also necessary to obtain the average score of all visual evidence frames corresponding to different analysis angles in the multimodal evidence chain, and perform a pre-weighted calculation to obtain the evidence strength value; combine the consistency of the evidence and the evidence strength value to obtain the confidence level; the confidence level score range is 0 to 10, the higher the confidence level value, the more credible the answer to the question, and obtain the video reasoning result based on the answer to the question and the confidence level.

[0134] The beneficial effects achieved by the embodiments of the present invention are as follows:

[0135] The process involves: acquiring user-input question text, a set of options, and a target video; forming a complementary analysis angle set using the question text and option set; performing frame-by-frame matching on the target video based on a pre-defined angle feature mapping library to obtain a set of key timestamps corresponding to different analysis angles in the complementary analysis angle set; extracting a set of corresponding key video frames from the target video based on the key timestamp set; scoring each key video frame in the key video frame set and selecting multiple key video frames as visual evidence frames according to their scores from high to low, thus constructing an angle-frame mapping table between different analysis angles and their corresponding visual evidence frames; acquiring the text to be reasoned that requires visual corroboration and associating it with the angle-frame mapping table to construct a multimodal evidence chain; and obtaining the video reasoning result of the question text based on the multimodal evidence chain. This multi-stage processing framework—"analysis angle generation - key timestamp location - key video frame extraction - visual evidence frame selection - multimodal evidence chain construction - video reasoning"—achieves high-precision and highly interpretable video understanding, reducing the dependence of video understanding on large-scale labeled data.

[0136] By using a multimodal evidence chain of "sentence-evidence image-evidence description", the entire chain of evidence traceability from the answer to the video is realized, effectively solving the "black box" problem of traditional models.

[0137] Cross-validation based on multimodal evidence significantly reduces the error rate compared to single-modal video understanding, and the quantified confidence level also significantly improves the reliability of the answers.

[0138] Based on the video understanding method based on multimodal evidence chains described in the above embodiments, the video understanding system based on multimodal evidence chains will be described below through embodiments.

[0139] like Figure 3 As shown, this embodiment of the invention provides a video understanding system based on a multimodal evidence chain, including:

[0140] The acquisition module 301 is used to acquire the user-input question text, option set, and target video;

[0141] The analysis perspective generation module 302 is used to form a complementary set of analysis perspectives using the question text and the set of options.

[0142] The timestamp acquisition module 303 is used to perform frame-by-frame matching of the target video based on a preset angle feature mapping library to obtain a set of key timestamps corresponding to different analysis angles in the complementary analysis angle set.

[0143] The video frame extraction module 304 is used to extract the corresponding key video frame set from the target video based on the key timestamp set.

[0144] The visual evidence acquisition module 305 is used to score each key video frame in the key video frame set, and select multiple key video frames as visual evidence frames according to the scores from high to low, and construct an angle-frame mapping table between different analysis angles and corresponding visual evidence frames.

[0145] The evidence chain construction module 306 is used to obtain the text to be reasoned that requires visual corroboration, associate the text to be reasoned with the angle-frame mapping table, and construct a multimodal evidence chain.

[0146] The video reasoning module 307 is used to obtain video reasoning results of the question text based on a multimodal evidence chain.

[0147] The beneficial effects achieved by the embodiments of the present invention are as follows:

[0148] The acquisition module 301 acquires the user-input question text, option set, and target video; the analysis angle generation module 302 uses the question text and option set to form a complementary analysis angle set; the timestamp acquisition module 303 performs frame-by-frame matching on the target video based on a preset angle feature mapping library to obtain a set of key timestamps corresponding to different analysis angles in the complementary analysis angle set; the video frame extraction module 304 extracts the corresponding set of key video frames from the target video based on the set of key timestamps; the visual evidence acquisition module 305 scores each key video frame in the set of key video frames and selects multiple key video frames as visual evidence frames according to the scores from high to low, constructing an angle-frame mapping table between different analysis angles and corresponding visual evidence frames; the evidence chain construction module 306 acquires the text to be reasoned that requires visual corroboration, associates the text to be reasoned with the angle-frame mapping table, and constructs a multimodal evidence chain; the video reasoning module 307 obtains the video reasoning result of the question text based on the multimodal evidence chain. By employing a multi-stage processing framework of "analysis angle generation - key timestamp localization - key video frame extraction - visual evidence frame screening - multimodal evidence chain construction - video reasoning", high-precision and highly interpretable video understanding is achieved, reducing the dependence of video understanding on large-scale labeled data.

[0149] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A video understanding method based on a multimodal evidence chain, characterized in that, include: The system obtains the user-inputted question text, a set of options, and a target video, wherein the question text is the user's video comprehension question regarding the target video. Use the question text and the set of options to form a set of complementary analytical perspectives; The target video is matched frame by frame based on a preset angle feature mapping library to obtain a set of key timestamps corresponding to different analysis angles in the complementary analysis angle set. Extract the corresponding set of key video frames from the target video based on the set of key timestamps; Each key video frame in the set of key video frames is scored, and multiple key video frames are selected as visual evidence frames according to the scores from high to low, and an angle-frame mapping table between different analysis angles and the corresponding visual evidence frames is constructed. Obtain the text to be inferred that requires visual corroboration; Perform text analysis on the text to be reasoned to determine the target analysis angle; The target visual evidence frame related to the target analysis angle is extracted from the angle-frame mapping table; A multimodal evidence chain is constructed based on the text to be reasoned and the target visual evidence frame; The video reasoning result of the question text is obtained based on the multimodal evidence chain.

2. The video understanding method based on a multimodal evidence chain according to claim 1, characterized in that, The process of using the question text and the option set to form a complementary set of analytical perspectives includes: The question text Q is encoded into a question semantic vector using a semantic encoder in a pre-built large language model. The semantic vector of the problem The expression is: ; Based on the problem semantic vector Obtain the probability of angle categories from different analytical perspectives The angle category probability The expression is: ； Among them, the Represents the set of analytical perspectives. The This represents the k-th analytical angle; the... As a preset question semantic vector weight value, the Fixed parameters for the preset question semantics; The option set C is encoded into an option semantic vector. The option semantic vector The expression is: ; Calculate the similarity vector between any two options i and j in the option set C. The similarity vector The expression is: ； in, , , The The option semantic vector represents the i-th option. The option semantic vector representing the j-th option; Based on the similarity vector Differential expression The differential expression The expression is: ； Among them, the The preset option semantic vector weight values, the The semantic parameters are fixed for the preset options; Based on a preset complementary fusion rule, the angle category probability is... The difference expression By integrating these elements, a set of complementary analytical perspectives can be obtained. The set of complementary analysis angles The expression is: ； Among them, the For the preset complementary fusion function, the This is the preset filtering threshold.

3. The video understanding method based on a multimodal evidence chain according to claim 2, characterized in that, The frame-by-frame matching of the target video based on a preset angle feature mapping library yields a set of key timestamps for different analysis angles in the complementary analysis angle set, including: Through a pre-trained angle feature encoder From the perspective of analysis Encode the corresponding visual feature vectors. , The Representing real numbers, the stated Represents the visual feature vector The dimension; A preset angle feature mapping library is constructed, and the preset angle feature mapping library stores a visual feature dictionary. The visual feature dictionary The expression is: ； The general visual feature model is used to extract features from each video frame in the target video to obtain the frame features of each video frame. The visual feature dictionary Matching with each frame feature yields a matching degree sequence. , The This represents the matching degree value between the visual features of the k-th analysis angle and the frame features of the t-th video frame. Where m is the total number of video frames in the target video; The The expression is: ； Among them, the The frame characteristics of the t-th video frame, representing the timestamp Time; According to the descending order of matching degree value and matching threshold The top N video frames are selected as keyframes, and a set of key timestamps corresponding to the k-th analysis angle is generated based on the timestamps (Time) of the top N video frames. The key timestamp set The expression is: ； The The t-th video frame in the target video is designated as a keyframe.

4. The video understanding method based on a multimodal evidence chain according to claim 3, characterized in that, The process of scoring each key video frame in the key video frame set and selecting multiple key video frames as visual evidence frames according to their scores from high to low, and constructing an angle-frame mapping table between different analysis angles and the corresponding visual evidence frames, includes: Based on a preset multi-dimensional scoring mechanism, each key video frame in the set of key video frames is scored to obtain the score value of each key video frame under the corresponding analysis angle. According to the descending order of the score values ​​under the same analytical perspective, the Top M keyframes corresponding to each analytical perspective are selected as visual evidence frames; where M is less than N. An angle-frame mapping table is constructed based on the visual evidence frames and scoring values ​​corresponding to different analysis angles.

5. The video understanding method based on a multimodal evidence chain according to claim 4, characterized in that, The method further includes: Image description text is generated after image recognition is performed on the target visual evidence frame; The image description text is added as evidence to the multimodal evidence chain.

6. The video understanding method based on a multimodal evidence chain according to claim 5, characterized in that, The video reasoning result obtained based on the multimodal evidence chain for the question text includes: Based on the multimodal evidence chain, evidence features are obtained, and the evidence features are matched with different options in the option set. The option with the highest matching degree is selected as the answer to the question. The average score of all visual evidence frames corresponding to different analysis angles in the multimodal evidence chain is obtained, and the evidence strength value is obtained by performing a pre-set weighted calculation. The confidence level is determined based on the strength of the evidence. The video inference result is obtained based on the answer to the question and the confidence level.

7. A video understanding system based on a multimodal evidence chain, characterized in that, include: The acquisition module is used to acquire the question text, option set, and target video input by the user, wherein the question text is the user's video comprehension question regarding the target video; An analysis perspective generation module is used to form a complementary set of analysis perspectives using the question text and the set of options. The timestamp acquisition module is used to perform frame-by-frame matching of the target video based on a preset angle feature mapping library to obtain a set of key timestamps corresponding to different analysis angles in the complementary analysis angle set. The video frame extraction module is used to extract a corresponding set of key video frames from the target video based on the set of key timestamps. The visual evidence acquisition module is used to score each key video frame in the key video frame set, and select multiple key video frames as visual evidence frames according to the scores from high to low, and construct an angle-frame mapping table between different analysis angles and the corresponding visual evidence frames. The evidence chain construction module is used to acquire the text to be reasoned that requires visual corroboration; perform text analysis on the text to be reasoned to determine the target analysis angle; and extract the target visual evidence frame related to the target analysis angle from the angle-frame mapping table. A multimodal evidence chain is constructed based on the text to be reasoned and the target visual evidence frame; The video reasoning module is used to obtain the video reasoning result of the question text based on the multimodal evidence chain.

Images