Large model tracing reason reasoning method for complex video
By enhancing the visual abductive reasoning capabilities of multimodal large models through cross-modal causal contrastive learning and lightweight adapters, the problem of insufficient causal relationship modeling in complex videos in existing models is solved, and higher abductive reasoning performance is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-10
AI Technical Summary
Existing multimodal large models lack the ability to model invisible causal relationships in complex videos. Furthermore, existing models primarily focus on knowledge representation and the expression of symbolic behavior rather than advancing the abductive reasoning process, resulting in a significant gap between their abductive reasoning capabilities and those of humans.
We employ cross-modal causal contrastive learning and lightweight adapters to enhance multimodal large models. Candidate hypotheses are generated through a large language model, and end-to-end training is performed by combining visual cross-attention, temporal convolution, and feedforward network adapters to enhance the abductive reasoning ability of the visual modality.
It improves the abductive reasoning ability of multimodal large models in complex videos, reduces the reasoning space through language and visual enhancement, provides prior knowledge, achieves higher causal correlation and image thinking ability, and demonstrates good performance.
Smart Images
Figure CN121835898A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence and computer vision, in particular to a training method for video understanding and inference of a multi-modal large model. BACKGROUND
[0002] Visual Abductive Reasoning (VAR) aims to give the most reasonable explanation for a series of incomplete observations. Abductive reasoning is an important feature of human cognition, which enables individuals to generate hypotheses by combining language and image thinking in the case of information missing.
[0003] Early solutions are image-based abductive reasoning methods that use models such as CLIP to align the semantics of images and text. However, these methods are limited by their inability to capture dynamic spatio-temporal relationships in video data.
[0004] In recent years, video-based abductive reasoning methods such as REASONER and UPD-Trans have improved early solutions by introducing temporal dependency modeling and combining knowledge of video sequences and text annotations. However, these models still face challenges when dealing with complex real-world scenarios, especially those involving causal relationships across multiple modalities. Models like KN-VLM and MAR have explored integrating symbolic reasoning and multi-modal representation, but they mainly focus on improving knowledge representation and symbolic behavior expression rather than advancing the abductive reasoning process.
[0005] Existing multi-modal large models have made significant progress in tasks such as video question answering and video caption generation, such as Video-LLaVA and Video-ChatGPT models. However, these models mainly deepen the fine-grained understanding of known video content through technical means, lacking modeling of invisible causal relationships in videos. Existing research has shown that the abductive reasoning capabilities of current multi-modal large models still lag far behind humans.
[0006] Existing visual abductive methods are all small-scale models dedicated to language reasoning. SUMMARY
[0007] In view of this, the present application provides a large model abductive reasoning method for complex videos, which enhances the abductive reasoning capabilities of multi-modal large models from both language and vision aspects.
[0008] To achieve the above purpose, the technical scheme of the large model abductive reasoning method for complex videos provided by the present application includes the following steps: Step one: For the observed video, construct a visual abductive reasoning task.
[0009] Step two: the reasoner first generates candidate hypotheses using a large language model without visual input, and then selects the hypothesis with the highest causal correlation through cross-modal causal contrastive learning. The output layer features are obtained using a multi-modal large model.
[0010] Step three: the imagination model is obtained by adding three types of lightweight adapters to the existing text-to-image diffusion model. The model takes the output layer features of the reasoner and the observed video as conditions, and provides visual supervision by calculating diffusion loss during training to achieve causal enhancement in the visual modality.
[0011] Step four: end-to-end training of the imagination model and the reasoner to provide enhancement in the visual modality.
[0012] Further, the visual causal reasoning task is as follows: the observed video is a video sequence containing T events , represents a set of T-1 observed events, and H represents an unobserved explanatory event.
[0013] The goal of the visual causal reasoning task is to infer the most likely language explanation for the unknown event H based on the observed events in .
[0014] Further, the reasoner includes a multi-modal large model, a large language model, and a causal contrastive learning module.
[0015] Step 2 includes the following processes: First, a pre-trained multi-modal large model is used to generate video captions for each video segment in the observed event set , obtaining a caption sequence , where is the description corresponding to .
[0016] Then, the large language model is prompted to infer reasonable unknown events, obtaining candidate hypotheses. Through L calls to the large language model, a set of L diverse candidate hypotheses is obtained, denoted as , where each represents a candidate hypothesis for the unknown event.
[0017] The causal contrastive learning module is used to calculate the causal correlation score of each candidate hypothesis with the observed video, and the top hypotheses with the highest causal correlation are selected and input into the multi-modal large model to obtain the output layer features , which are passed into the imagination model as the conditions for imagination.
[0018] Further, the causal contrastive learning module comprises a visual encoder Φ V and a text encoder Φ T , V encodes the observed initial segment and the final segment into visual embeddings and , T encodes the positive and negative samples into text embeddings and .
[0019] For each generated candidate hypothesis , it is projected into the joint space by the text encoder Φ T , and the causal relevance score of the candidate hypothesis with respect to the observed video is calculated:
[0020] wherein = Φ T ( ) represents the features of .
[0021] Then, the candidate hypotheses are sorted according to the causal relevance scores, and the top candidate hypotheses with the highest causal relevance are selected.
[0022] Further, the imagination module comprises a visual cross-attention adapter V-Adapter, a temporal convolution adapter T-Adapter, and a feedforward network adapter F-Adapter.
[0023] The visual cross-attention adapter V-Adapter, whose key-value sequence input is an efficient local-global hybrid representation that is used to capture the visual semantics related to the linguistic explanation ; for the local representation, the image and text encoders of CLIP are used to obtain the features of all N frames in , and the features of , and then the similarity scores are calculated:
[0024] wherein represents the similarity score of the i-th frame with ; H most likely linguistic explanation ; the cosine similarity between the features.
[0025] The video frames are sorted according to the similarity scores, and the top k high-score frames are connected to form the local representation For global representation, the weighted average of the similarity scores is computed, resulting in the global representation
[0026] Finally, the local and global representations are concatenated as the visual condition, denoted as ; finally, the V-Adapter is represented as follows:
[0027] where x is the query sequence input of the V-Adapter, Q is the query matrix, is the value matrix in the cross-attention computation of the V-Adapter, is the projection matrix in the visual cross-attention computation.
[0028] The V-Adapter is integrated in parallel with the original text cross-attention in each U-Net block while freezing the text attention parameters; the outputs from the parallel cross-attention branches are added, and the process is described as follows:
[0029] where , is the key matrix in the text cross-attention computation, is the value matrix in the text cross-attention computation, is the projection matrix in the text cross-attention computation, and the text cross-attention function.
[0030] The temporal convolution adapter T-Adapter is used to model the temporal dependencies between frames and is attached after the spatial convolution layer in each U-Net block, adopting a fully convolutional design, and the temporal convolution adapter function is defined as:
[0031] where and represent the upper and lower projection layers, respectively, both of which are three-dimensional convolutions. The feedforward network adapter F-Adapter is used to retain the integrity of the original feedforward network while enhancing the spatial representation, the F-Adapter is introduced into the feedforward network layer as a parallel block, ensuring that the pre-trained feedforward network layer remains unchanged when adapting to video spatial features, the feedforward network adapter comprises two fully connected layers with GELU activation functions, which are expressed as:
[0032] Wherein, and are the upward projection layer and the downward projection layer respectively.
[0033] Further, step four: end-to-end training is performed on the imagination and inference, specifically, a two-stage training mode is adopted; in this mode, each module is first independently trained, and then jointly end-to-end optimized.
[0034] Further, the two-stage training mode, in the first stage, for the inference, the multimodal large model is fine-tuned by LoRA under the standard cross-entropy loss to generate reasonable hypotheses from incomplete observations; for the imagination, the weight of the diffusion model is frozen, and only the latent diffusion loss is used to update the parameters of the adapter; at the same time, the minimum signal-to-noise ratio Min SNR weighting strategy is applied, which adaptively reweights the loss at each diffusion time step to accelerate the convergence of the diffusion model.
[0035] In the second stage, the inference and the imagination are jointly optimized in an end-to-end manner, and the total loss is defined as:
[0036] Wherein, is the coefficient for balancing the two terms.
[0037] Beneficial effects: 1. The large model trace inference method for complex videos provided by the application enhances the trace inference ability of the multimodal large model from the aspects of language and vision. From the language aspect, candidate explanations with high causal correlation are proposed to narrow down the inference space and provide prior knowledge for the large model. From the visual aspect, the visual imagination module is designed by referring to the process of human cognitive reasoning to provide image thinking ability for language trace. The application is a pioneering work for solving the visual trace inference task by using large model technology, and it performs well in the standard benchmark test, continuously outperforms existing small models and large models, and sets a new advanced level.
[0038] 2. From the perspective of language, a causal contrastive learning model is designed to mine high-quality text hypotheses, narrow down the reasoning space, and provide key prior knowledge for large models.
[0039] 3. From the perspective of vision, the present application is the first to explicitly incorporate image thinking ability to improve abductive reasoning. BRIEF DESCRIPTION OF DRAWINGS
[0040] Figure 1 The network architecture of the present application is shown in the figure. Figure 2 The cross-modal causal contrastive learning schematic diagram of the present application is shown in the figure. DETAILED DESCRIPTION
[0041] The present application will be described in detail below with reference to the accompanying drawings and examples.
[0042] The complex video-oriented large model abductive reasoning method provided by the embodiments of the present application comprises the following steps: Step 1: For the observed video, a visual abductive reasoning task is constructed.
[0043] First, the visual abductive reasoning task targeted by the present application is described. Given a video sequence containing T events The events have logical relevance and are arranged in chronological order. represents a set of T-1 observed premise events, and H represents an unobserved explanatory event. H can appear anywhere in The goal of the visual abductive reasoning task is to infer the most likely linguistic explanation for the unknown event H based on the observed events in The video sequence can be obtained from the existing dataset "VAR".
[0044] For this task, the present application is inspired by the way humans integrate language and image thinking in abductive reasoning, and introduces a joint architecture to enhance the abductive reasoning ability of the multi-modal large model. The system consists of two main modules: a reasoner and an imaginer. The reasoner first generates candidate hypotheses using a large language model without visual input, and then selects hypotheses with high causal relevance through cross-modal causal contrastive learning. The imaginer is a diffusion model with a lightweight adapter, which takes the output embedding of the reasoner and the observed video as conditions. It is trained end-to-end with the reasoner to provide enhancement in the visual modality.
[0045] Step 2: The reasoner first generates candidate hypotheses using a large language model without visual input, and then selects hypotheses with the highest causal relevance through cross-modal causal contrastive learning, and obtains output layer features using a multi-modal large model.
[0046] The inference engine includes a multimodal large model, a large language model, and a causal contrastive learning module; A multimodal large model is used for candidate hypothesis generation. The difficulty of abductive reasoning lies in the existence of a large and complex space of plausible explanations. To address this problem, this invention first employs a pre-trained multimodal large model (an arbitrary multimodal large model) as the set of observed events. Each video segment generates a video subtitle, resulting in a subtitle sequence. Each of them correspond The description (T-1 captions corresponding to the observed premise events) is then used. Next, GPT-4o-mini (an arbitrary large language model) is given the prompt: "You are an event completion expert, please infer the most reasonable event at the [MASK] position," prompting it to infer the reasonable unknown event. To ensure diversity and reduce redundancy, GPT-4o-mini is called multiple times (L times) for each data point using a relatively high sampling temperature (e.g., 1.4). This results in a set of L diverse candidate hypotheses, denoted as […]. Each of them This represents a linguistic explanation (candidate hypothesis) for an unknown event.
[0047] exist The hypotheses may contain low-quality or illusory candidates, which can hinder effective reasoning in large models. Therefore, this invention designs a hypothesis selection module based on contrastive learning, namely the causal contrastive learning module. Unlike the classical contrastive learning approach that only establishes superficial similarities between visual and textual modalities, this causal contrastive learning module is specifically designed to capture causal correlations.
[0048] In contrastive learning, a sufficient number of high-quality negative samples are crucial. Although the true explanation... While positive samples can be directly used as positive samples, constructing diverse and meaningful negative samples remains a challenge. This invention uses GPT-4o-mini to generate negative samples. The prompt begins with a task description: "There is a contrastive learning task aimed at matching missing video captions to a series of observed videos." Then, positive samples are provided to GPT. Subtitles for the observation video The instructions state: "Negative samples should be semantically different from positive samples, but still consistent with the logical context of the observed captions." GPT-4o-mini is called multiple times for each data point to obtain M negative samples.
[0049] Given any input It can be naturally divided into three consecutive segments: the initial segment Process fragments and the final segment H could correspond to any of these, and the following explanation will use H as an example of a process segment. The observations before and after H represent the initial and final segments, respectively. The causal contrastive learning module proposed in this invention includes a visual encoder Φ. V and a text encoder Φ T , Φ V The observed initial and final segments are encoded into visual embeddings. and Φ T Encode positive and negative samples into text embeddings. and .
[0050] During the training phase, the model uses NT-Xent loss optimization, which maximizes the causal correlation between observed videos and positive samples while minimizing the correlation with negative samples. The formula is as follows:
[0051] in, It is the temperature coefficient. It is the first Embedding of a negative sample, It is the cosine similarity between features. The loss is NT-Xent, where M is the total number of negative samples; During the inference phase, for each generated candidate hypothesis We use the text encoder Φ T Projected into the joint space, its causal correlation score with respect to the observed video is calculated:
[0052] in, =Φ T ( )express The embedding is then performed. The candidate hypotheses are then sorted according to their scores, and a top-priority hypothesis is selected for downstream inference. The hypothesis with the highest causal relevance.
[0053] Then, the candidate hypotheses were ranked according to their causal relevance scores, and the top hypotheses were selected. The candidate hypothesis with the highest causal correlation is input into the multimodal large model to obtain the output layer features. .
[0054] Will Multiple events are stitched together to form a single video. For unknown events H, placeholder frames with random pixels are used to fill in the gaps. To help the multimodal large model understand the location of H, video frames are numbered (video frames belonging to the same event are given the same number). These numbers indicate the temporal order of the events, helping the model understand the temporal order of different events. The labeled video is then input into the multimodal large model.
[0055] Output layer features of the multimodal large model of the inference engine It is passed into the imaginator as a condition for imagination.
[0056] Step 3: The Imagineer efficiently extends the existing text-based image diffusion model into a video generation model by adding three types of lightweight adapters. This model uses the output layer features of the inferencer and the observed video as conditions, providing visual supervision during training by calculating the diffusion loss, thus achieving abductive enhancement of the visual modality.
[0057] In this embodiment of the invention, the imaginator includes a visual cross-attention adapter (V-Adapter), a temporal convolution adapter (T-Adapter), and a feedforward network adapter (F-Adapter).
[0058] Visual Cross-Attention Adapter (V-Adapter) In the original Stable Diffusion, the U-Net attention module performs self-attention only on a single frame, ignoring information from other frames. To fully utilize... Based on visual cues observed in text (which may contain valuable information relevant to text interpretation), this invention designs a V-Adapter. Direct use... Computing all frames in a single dataset is computationally expensive and prone to noise due to redundancy. To address this issue, this invention proposes an efficient local-global hybrid representation that can capture data related to text interpretation. Related fine-grained and overall visual semantics. For local representations, CLIP's image and text encoders are used to obtain... Features of all N frames ,as well as Features Then calculate the similarity score:
[0059] in Indicates the i-th frame and Similarity score; H is the most likely linguistic interpretation ; Cosine similarity between features.
[0060] The video frames are sorted according to their similarity scores. The top k high-scoring frames in sorted i are concatenated to form a local representation. For the global representation, based on similarity scores... calculate The weighted average is used to obtain the global representation. Finally, the local and global representations are concatenated as a visual condition, denoted as... Ultimately, the V-Adapter can be represented by the following formula:
[0061] Where x is the input of the V-Adapter, Q is the query matrix. This is the value matrix used in the cross-attention calculation of the V-Adapter. / / It is the projection matrix in visual cross-attention calculation.
[0062] The V-Adapter is integrated in parallel with the original text cross-attention in each U-Net block, while freezing the text attention parameters. This allows the model to pay attention to both visual and textual cues simultaneously. The outputs from the parallel cross-attention branches are then summed. This process is described below:
[0063] in, , It is the key matrix in text cross-attention calculation. It is the value matrix in text cross-attention calculation. , It is the projection matrix in text cross-attention calculation. Text cross-attention function.
[0064] Temporal Convolution Adapter (T-Adapter) This adapter is designed to model inter-frame temporal dependencies and is appended to the spatial convolutional layer in each U-Net block. It uses depthwise separable 3D convolutions in the low-dimensional space of the projection, which helps reduce the complexity of temporal modeling. To maintain structural consistency and further improve temporal modeling, this invention employs a fully convolutional design, defined as:
[0065] in, and These represent the upper projection layer and the lower projection layer, respectively, both of which are 3D convolutions; F-Adapter (Feedforward Network Adapter) The F-Adapter enhances spatial representation while preserving the integrity of the original feedforward network. It is introduced as a parallel block into the feedforward network layers, ensuring that the pre-trained feedforward network layers remain invariant when adapting to video spatial features. The feedforward network adapter consists of two fully connected layers with GELU activation functions, which can be expressed as:
[0066] in, and These are the upward projection layer and the downward projection layer, respectively.
[0067] Step 4: Perform end-to-end training on the imaginator and inferencer to provide enhancements in the visual modality.
[0068] This invention employs a two-stage training model. In this model, each module is first trained independently, and then jointly optimized end-to-end. In the first stage, for the inference engine, the multimodal large model uses standard cross-entropy loss. The LoRA algorithm is then fine-tuned to enable it to generate reasonable hypotheses from incomplete observations. For the Imagineer, the weights of the diffusion model are frozen, and only the latent diffusion loss is used. This updates the adapter parameters. Simultaneously, a minimum signal-to-noise ratio (Min SNR) weighting strategy is applied, which adaptively reweights the loss at each diffusion time step to accelerate the convergence of the diffusion model. In the second stage, the inferencer and imaginer are jointly tuned end-to-end. The total loss is defined as:
[0069] in, It is the coefficient that balances these two items.
[0070] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A large-scale abductive reasoning method for complex videos, characterized in that, Includes the following steps: Step 1: Construct a visual abductive reasoning task based on the observed videos; Step 2: The inference engine first uses a large language model without visual input to generate candidate hypotheses, and then uses cross-modal causal contrastive learning to filter out the hypotheses with the highest causal relevance, and uses a multimodal large model to obtain output layer features; Step 3: The Imagineer efficiently extends the existing text-based image diffusion model into a video generation model by adding three types of lightweight adapters. This model uses the output layer features of the inferencer and the observed video as conditions, and provides visual supervision by calculating the diffusion loss during training to achieve abductive enhancement of the visual modality. Step 4: Perform end-to-end training on the imaginator and reasoner to provide enhancements in both language and visual modalities.
2. The large-model abductive reasoning method for complex videos as described in claim 1, characterized in that, The visual abductive reasoning task specifically involves the observation video being a video sequence containing T events. , Let T-1 be the set of observed events, and H be the set of unobserved explanatory events. The goal of the visual abductive reasoning task is based on From the observed events, infer the most probable linguistic interpretation of the unknown event H. .
3. The large-model abductive reasoning method for complex videos as described in claim 2, characterized in that, The inference engine includes a multimodal large model, a large language model, and a causal contrastive learning module; Step 2 specifically includes the following process: First, a pre-trained multimodal large model is used for the set of observed events. Each video segment generates a video subtitle, resulting in a subtitle sequence. ,in For the corresponding Description; Next, prompt words are given to the large language model to infer reasonable unknown events and obtain candidate hypotheses. Through L calls to the large language model, a set of L diverse candidate hypotheses is finally obtained, denoted as . Each of them This represents a candidate hypothesis about an unknown event; The causal comparison learning module is used to calculate the causal correlation score of each candidate hypothesis with respect to the observed video, and select the top candidate hypothesis. The hypothesis with the highest causal correlation is input into a multimodal large model to obtain the output layer features. , It is passed into the imaginator as a condition for imagination.
4. The large-model abductive reasoning method for complex videos as described in claim 3, characterized in that, The causal comparison learning module includes a visual encoder Φ V and a text encoder Φ T , Φ V The observed initial and final segments are encoded into visual embeddings. and , Φ T Encode positive and negative samples into text embeddings. and ; For each generated candidate hypothesis Φ T Projected into the joint space, candidate hypotheses are computed. Regarding the causal correlation score of the observed video: in, =Φ T ( )express Features; Then, the candidate hypotheses were ranked according to their causal relevance scores, and the top hypotheses were selected. The candidate hypothesis with the highest causal relevance.
5. A large-model abductive reasoning method for complex videos as described in any one of claims 1 to 4, characterized in that, The imaginator includes a visual cross-attention adapter (V-Adapter), a temporal convolution adapter (T-Adapter), and a feedforward network adapter (F-Adapter); The visual cross-attention adapter (V-Adapter) takes a key-value sequence as input, which is an efficient local-global hybrid representation used for capturing and interpreting language. Related visual semantics; for local representations, CLIP image and text encoders are used to obtain... Features of all N frames ,as well as Features Then calculate the similarity score: in Indicates the i-th frame and Similarity score; H is the most likely linguistic interpretation ; Cosine similarity between features; The video frames are sorted according to their similarity scores, and the top k high-scoring frames are concatenated to form a local representation. For the global representation, based on similarity scores calculate The weighted average is used to obtain the global representation. ; Finally, the local and global representations are concatenated as a visual condition, denoted as... Ultimately, the V-Adapter can be represented by the following formula: Where x is the query sequence input of the V-Adapter, Q is the query matrix. This is the value matrix used in the cross-attention calculation of the V-Adapter. , , It is the projection matrix in visual cross-attention calculation; The V-Adapter is integrated in parallel with the original text cross-attention in each U-Net block, while freezing the text attention parameters; the outputs from the parallel cross-attention branches are summed, as described below: in, , It is the key matrix in text cross-attention calculation. It is the value matrix in text cross-attention calculation. , It is the projection matrix in text cross-attention calculation. Text cross-attention function; The temporal convolutional adapter (T-Adapter) is used to model inter-frame temporal dependencies and is appended after the spatial convolutional layer in each U-Net block. It employs a fully convolutional design, and the temporal convolutional adapter function... Defined as: in, and These represent the upper projection layer and the lower projection layer, respectively, both of which are 3D convolutions; The feedforward network adapter F-Adapter enhances spatial representation while preserving the integrity of the original feedforward network. The F-Adapter is introduced as a parallel block into the feedforward network layer, ensuring that the pre-trained feedforward network layer remains unchanged when adapting to video spatial features. The feedforward network adapter contains two fully connected layers with GELU activation functions, described as follows: in, and These are the upward projection layer and the downward projection layer, respectively.
6. The large-model abductive reasoning method for complex videos as described in claim 1, characterized in that, Step four: Perform end-to-end training for the imaginator and the inferencer. Specifically, a two-stage training mode is adopted. In this mode, each module is first trained independently, and then joint end-to-end optimization is performed.
7. The large-model abductive reasoning method for complex videos as described in claim 6, characterized in that, In the two-stage training mode, in the first stage, for the inferencer, the multimodal large model uses standard cross-entropy loss. The LoRA algorithm is then fine-tuned to generate reasonable hypotheses from incomplete observations; for the Imagineer, the weights of the diffusion model are frozen, and only the latent diffusion loss is used. To update the adapter parameters; at the same time, a minimum signal-to-noise ratio (Min SNR) weighting strategy is applied, which adaptively reweights the loss at each diffusion time step to accelerate the convergence of the diffusion model; In the second phase, the inference engine and the imaginator are jointly tuned end-to-end, with the total loss... Defined as: in, It is the coefficient that balances these two items.