Video understanding method, device and equipment based on physical motion and appearance decoupling

Abstract

The invention provides a video understanding method, device and equipment based on physical motion and appearance decoupling, relates to the technical field of artificial intelligence, and aims to realize accurate understanding of videos. The method comprises the following steps: performing feature extraction on each video frame of a video frame sequence to obtain a feature sequence; a double-branch encoder is used to decouple the feature sequence to obtain an appearance feature sequence and a preliminary motion feature sequence, the preliminary motion feature sequence represents the motion feature of each video frame of the video frame sequence, and the double-branch encoder comprises an appearance encoder and a motion encoder; utilizing a neural differential equation modeler to perform continuous time motion trail modeling between frames according to the preliminary motion feature sequence to obtain a motion feature sequence; and inputting the motion feature sequence, the appearance feature sequence and the user prompt statement into a large language model for cross-modal information fusion and reasoning to obtain answer content corresponding to the user prompt statement.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video understanding method, apparatus and device based on the decoupling of physical motion and appearance. Background Technology

[0002] Video Large Language Models (VLMs) have demonstrated outstanding performance in multimodal understanding and generation tasks, attracting widespread attention in the field of visual language. Existing models generally employ a still-frame-based encoding strategy: decomposing the video into a sequence of static images, extracting single-frame visual features using a pre-trained visual encoder, and then fine-tuning the alignment between visual features and language modalities through instructions. This strategy enables the model to learn strong correlations between images and text on large-scale multimodal data, demonstrating significant effectiveness in appearance-driven tasks.

[0003] However, these methods still rely on semantic matching and retrieval mechanisms, focusing more on finding similar patterns from historical data rather than possessing true physical reasoning capabilities. Especially when understanding video content with significant physical dynamics in the real world, relying solely on feature alignment of static frames is far from sufficient. For example, in distinguishing whether an object is accelerating or decelerating, two video clips may appear almost identical in still frames, but their underlying physical states are completely different. Humans can infer correct results by judging physical signals such as speed changes, but existing video language models, lacking modeling of motion processes, often struggle with this type of reasoning. Therefore, existing video language models still have shortcomings in video understanding, and how to achieve accurate video understanding is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a video understanding method, apparatus and device based on the decoupling of physical motion and appearance, so as to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of this application discloses a video understanding method based on the decoupling of physical motion and appearance, the method comprising: Feature extraction is performed on each video frame in the video frame sequence to obtain a feature sequence; The feature sequence is decoupled using a dual-branch encoder to obtain an appearance feature sequence and a preliminary motion feature sequence. The preliminary motion feature sequence characterizes the motion features of each video frame in the video frame sequence. The dual-branch encoder includes an appearance encoder and a motion encoder. Using a neural differential equation modeler, based on the preliminary motion feature sequence, continuous-time motion trajectory modeling is performed between frames to obtain a motion feature sequence, which characterizes the dynamic change trajectory of the continuous time corresponding to the video frame sequence. The motion feature sequence, the appearance feature sequence, and the user prompt statement are input into a large language model for cross-modal information fusion and inference to obtain the answer content corresponding to the user prompt statement.

[0006] Optionally, the motion feature sequence, the appearance feature sequence, and the user prompt statement are input into a large language model for cross-modal information fusion and inference to obtain the response content corresponding to the user prompt statement, including: The motion feature sequence is linearly mapped and projected into a motion token sequence, and the appearance feature sequence is linearly mapped and projected into an appearance token sequence; Convert the user prompt statement into a sequence of prompt tokens; The motion token sequence, the appearance token sequence, and the prompt word token sequence are embedded into a natural language prompt template to obtain a token combination sequence; The token combination sequence is input into the large language model for cross-modal information fusion and inference to obtain the answer content corresponding to the user prompt statement.

[0007] Optionally, the natural language prompt template includes a first embedding position, a second embedding position, and a third embedding position; the motion token sequence, the appearance token sequence, and the prompt word token sequence are embedded into the natural language prompt template to obtain a token combination sequence, including: The motion token sequence is embedded in the first embedding position, the appearance token sequence is embedded in the second embedding position, and the prompt word token sequence is embedded in the third embedding position to obtain the token combination sequence.

[0008] Optionally, the feature sequence is decoupled using a dual-branch encoder to obtain an appearance feature sequence and a preliminary motion feature sequence, including: The feature sequence is sequentially input into the appearance encoder of the dual-branch encoder, and the appearance encoder extracts the appearance features to obtain the appearance feature sequence; wherein, the appearance features include texture information, shape information and structural information in the scene; The feature sequence is input into the motion encoder of the dual-branch encoder. The motion encoder, based on the temporal self-attention mechanism of Transformer, extracts temporal information from the features of each frame of the feature sequence to obtain the preliminary motion feature sequence.

[0009] Optionally, the neural differential equation modeler is trained with the goal of minimizing the error between the predicted motion feature sequence and the actual motion feature sequence.

[0010] Optionally, the video understanding method based on the decoupling of physical motion and appearance is implemented based on a large video language model, which includes at least an appearance encoder, a motion encoder, a neural differential equation modeler, and a large language model. The large video language model is trained according to the following steps. Acquire sample data, which includes: sample video frame sequences, sample user prompts, and actual response content; The sample feature sequence is decoupled using a dual-branch encoder to obtain a sample appearance feature sequence and a sample preliminary motion feature sequence. The sample preliminary motion feature sequence characterizes the motion features of each video frame in the sample video frame sequence. The neural differential equation modeler is used to model the continuous time motion trajectory between frames based on the preliminary motion feature sequence of the sample, to obtain the observed motion feature sequence and the predicted motion feature sequence. The observed motion feature sequence represents the dynamic change trajectory of the continuous time corresponding to the sample video frame sequence, and the predicted motion feature sequence represents the predicted dynamic change trajectory of the future continuous time corresponding to the sample video frame sequence. The observed motion feature sequence, the sample appearance feature sequence, and the sample user prompt statement are input into the large language model for cross-modal information fusion and inference to obtain the sample answer content corresponding to the sample user prompt statement; Based on the observed motion feature sequence, the predicted motion feature sequence, the sample appearance feature sequence, the sample answer content, and the actual answer content, the total loss value is calculated, and the model parameters of the video big language model are updated based on the total loss value.

[0011] Optionally, the neural differential equation modeler includes a state modeling network and a differentiable numerical solver; using the neural differential equation modeler to perform continuous-time motion trajectory modeling between frames based on the preliminary motion feature sequence of the sample, the observed motion feature sequence and the predicted motion feature sequence are obtained, including: The state modeling network is used to model the derivative of the state in the time dimension. The initial motion feature sequence of the sample is used as the initial state, and the derivative of the hidden state with respect to time is used as the neural network parameter to obtain the observed motion feature sequence. The derivative of the state in the time dimension is modeled by a state modeling network. The observed motion feature sequence is used as the initial state, and the predicted motion feature sequence is obtained by extrapolation through a differentiable numerical solver.

[0012] Optionally, the total loss value is calculated based on the observed motion feature sequence, the predicted motion feature sequence, the sample appearance feature sequence, the sample response content, and the actual response content, including: Calculate the physical consistency loss value based on the observed motion feature sequence and the predicted motion feature sequence; Calculate the mutual information loss value based on the observed motion feature sequence and the sample appearance feature sequence; Calculate the supervised learning loss value based on the sample answers and the actual answers; The total loss value is obtained based on the physical consistency loss value, the mutual information loss value, and the supervised learning loss value.

[0013] A second aspect of this application discloses a video understanding device based on the decoupling of physical motion and appearance, the device comprising: The extraction module is used to extract features from each video frame in the video frame sequence to obtain a feature sequence; The decoupling module uses a dual-branch encoder to decouple the feature sequence to obtain an appearance feature sequence and a preliminary motion feature sequence. The preliminary motion feature sequence characterizes the motion features of each video frame in the video frame sequence. The dual-branch encoder includes an appearance encoder and a motion encoder. The modeling module is used to model the continuous-time motion trajectory between frames based on the preliminary motion feature sequence using a neural differential equation modeler, thereby obtaining a motion feature sequence, which characterizes the dynamic change trajectory of the continuous time corresponding to the video frame sequence. The reasoning module is used to input the motion feature sequence, the appearance feature sequence, and the user prompt statement into a large language model for cross-modal information fusion and reasoning to obtain the answer content corresponding to the user prompt statement.

[0014] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the video understanding method based on physical motion and appearance decoupling described in the first aspect of this application.

[0015] A fourth aspect of this application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the video understanding method based on physical motion and appearance decoupling described in the first aspect of this application.

[0016] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the video understanding method based on physical motion and appearance decoupling described in the first aspect of this application.

[0017] The embodiments of this application have the following advantages: In this embodiment, by acquiring the feature sequence corresponding to the video frame sequence and decoupling the feature sequence using a dual-branch encoder, the visual appearance features and physical motion information in the video frame sequence are decoupled, thereby obtaining a structured and interpretable multimodal representation (i.e., appearance feature sequence and preliminary motion feature sequence). Furthermore, a neural differential equation modeler is used to model the continuous-time motion trajectory between frames based on the preliminary motion feature sequence, modeling the physical motion process as a differential process in continuous time, and obtaining a differentiable physical representation (i.e., motion feature sequence). Finally, the motion feature sequence, the appearance feature sequence, and the user prompt statement are input into a large language model for cross-modal information fusion and inference, enabling the large language model to understand the continuous motion trajectory in the video and achieve accurate understanding of the video by combining visual appearance information. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the steps of a video understanding method based on the decoupling of physical motion and appearance, as provided in an embodiment of this application. Figure 2 This is a schematic diagram of a video large language model provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating an application scenario of a video large language model provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a video understanding device based on the decoupling of physical motion and appearance provided in an embodiment of this application; Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] In related technologies, video-based large language models are mainly built upon image-language models. This involves decomposing videos into static image sequences, extracting image features using a pre-trained visual encoder, and then fine-tuning them with instructions to achieve alignment with language. This approach has achieved some success in tasks such as video question answering and video summarizing. However, fundamentally, these methods still rely on pattern matching at the visual semantic level and lack the ability to understand the real physical dynamics within the video. Especially in tasks involving object motion, interaction, and reasoning about physical laws, existing models often perform poorly.

[0022] The current limitations of large video models in physical understanding stem primarily from two challenges: First, motion and appearance features are highly coupled. The motion information of objects in videos is often intertwined with visual changes such as lighting, texture, and perspective, making it difficult for models to accurately separate pure physical motion trajectories. Existing methods often rely on inter-frame differences to approximate motion, but this often only reflects visual changes rather than the actual continuous dynamic process. Second, there is a lack of physical supervision data. Unlike tasks such as image classification, which have rich annotations, it is difficult to obtain accurate annotations for physical attributes such as acceleration and force in videos, severely limiting the model's ability to model physical laws and its generalization capabilities.

[0023] To enhance video understanding capabilities, related technologies have attempted to introduce explicit physical modeling methods, such as physical reasoning models based on neural implicit modeling, recursive physical units, or expert hybrid networks. These methods have shown potential in small-scale physical scenarios. However, these methods generally suffer from weak generalization ability and difficulty in scaling to complex video tasks. Meanwhile, while existing large models possess cross-modal modeling and strong generalization capabilities, they lack prior knowledge of the physical world's structure, resulting in significant shortcomings in areas such as judging physical plausibility and predicting motion.

[0024] In summary, the relevant technologies mainly suffer from the following shortcomings: 1) Lack of explicit physical modeling capabilities, making it impossible to perform realistic physical reasoning. Current mainstream video language models mostly rely on image-level semantic matching, failing to model the motion trajectory, force processes, and other continuous physical dynamics of the real world. Therefore, they are ill-suited for tasks involving physical understanding, such as object interaction and causal reasoning. 2) High coupling between object motion and appearance changes, resulting in unclear physical representation. Existing models often mix motion information in videos with appearance changes such as lighting and texture, approximating motion based solely on inter-frame differences, making it difficult to form a structured, reasonable physical trajectory representation. 3) Lack of physical supervision data, limiting training. Unlike tasks such as image classification, physical attributes such as acceleration and force direction in videos are often difficult to label, leading to a lack of effective supervision during model training and limiting its performance and generalization ability in physical modeling tasks.

[0025] To overcome the limitations of related technologies, this application provides a video understanding method based on the decoupling of physical motion and appearance. By decoupling the physical motion information and visual appearance features in the video and introducing a differentiable physical modeling mechanism, the method can explicitly capture and fit the potential physical laws in the video, thereby achieving accurate understanding of the video based on multimodal representations (such as appearance feature sequences and motion feature sequences).

[0026] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a video understanding method based on the decoupling of physical motion and appearance, as provided in an embodiment of this application. Figure 1 As shown, the video understanding method based on the decoupling of physical motion and appearance may include steps S110 to S140: Step S110: Extract features from each video frame of the video frame sequence to obtain a feature sequence.

[0027] In this embodiment of the application, the video frame sequence includes multiple video frames, and the video frame sequence can be represented as follows: Where T represents the total number of frames in the video frame sequence. This represents the t-th video frame. The feature sequence includes features from multiple frames, with each feature corresponding to a feature of one video frame.

[0028] Specifically, through a shared visual backbone network (e.g., Feature extraction is performed on each video frame in the video frame sequence to obtain a feature sequence, which is used to determine the consistency of the underlying visual features and the efficiency of parameter sharing. For example, the t-th feature in the feature sequence... It can be represented as .

[0029] Step S120: Decouple the feature sequence using a dual-branch encoder to obtain an appearance feature sequence and a preliminary motion feature sequence. The preliminary motion feature sequence characterizes the motion features of each video frame in the video frame sequence. The dual-branch encoder includes an appearance encoder and a motion encoder.

[0030] Specifically, the appearance encoder of the dual-branch encoder is used to extract appearance features from the feature sequence to obtain an appearance feature sequence. Then, the motion encoder of the dual-branch encoder is used to extract motion features from the feature sequence to obtain a preliminary motion feature sequence. Thus, the visual appearance features and physical motion information in the video frame sequence are decoupled based on the dual-branch encoder.

[0031] The appearance feature sequence includes multiple appearance features arranged in temporal order, with each appearance feature corresponding to the static appearance information of a video frame. The appearance encoder is a lightweight neural network capable of appearance feature extraction (appearance feature modeling). A dual-branch appearance encoder extracts appearance features from the feature sequence to model static appearance feature information based on the feature sequence (i.e., shared features).

[0032] The preliminary motion feature sequence includes multiple preliminary motion features arranged in temporal order, that is, it includes the motion features (physical motion information) corresponding to each video frame. Therefore, the preliminary motion feature sequence represents the motion features of each video frame in the video frame sequence, that is, the preliminary motion feature sequence corresponds to a preliminary motion trajectory. A motion encoder is a lightweight neural network with the ability to extract physical motion information (physical motion information modeling capability). A motion encoder using a dual-branch encoder extracts motion features from the feature sequence to model dynamic physical motion information based on the feature sequence (i.e., shared features).

[0033] In an optional embodiment, the feature sequence is decoupled using a dual-branch encoder to obtain an appearance feature sequence and a preliminary motion feature sequence, including steps S120-1 and S120-2: Step S120-1: The feature sequence is sequentially input into the appearance encoder of the dual-branch encoder. The appearance encoder extracts the appearance features to obtain the appearance feature sequence. The appearance features include texture information, shape information and structural information in the scene.

[0034] Thus, the appearance encoder processes each feature in the feature sequence sequentially to capture visual cues such as texture, shape, and structure information in the scene, thereby maintaining consistency between video frames. For example, the appearance features in the appearance feature sequence can be represented as:

[0035] in, This represents the t-th appearance feature in the appearance feature sequence. Indicates the appearance encoder. This represents the t-th feature in the feature sequence.

[0036] Step S120-2: Input the feature sequence into the motion encoder of the dual-branch encoder. The motion encoder extracts temporal information from the features of each frame of the feature sequence based on the temporal self-attention mechanism of Transformer to obtain the preliminary motion feature sequence.

[0037] Thus, the motion encoder, based on the Transformer's temporal self-attention mechanism, extracts temporal information from the features of each frame in the feature sequence, perceiving changes in features over time to enhance inter-frame dependency modeling capabilities and achieve sensitive capture of continuous motion. For example, the initial motion feature sequence can be represented as:

[0038] in, Indicates a motion encoder. This represents the motion characteristics of the t-th frame.

[0039] Through the above steps S110 to S120, the visual appearance features and physical motion information in the video frame sequence are decoupled, thereby obtaining a structured and interpretable multimodal representation, namely the appearance feature sequence and the preliminary motion feature sequence.

[0040] Step S130: Using a neural differential equation modeler, the continuous-time motion trajectory between frames is modeled based on the preliminary motion feature sequence to obtain a motion feature sequence, which represents the dynamic change trajectory of the continuous time corresponding to the video frame sequence.

[0041] In this embodiment, the preliminary motion feature sequence includes motion features corresponding to each video frame, and the preliminary motion feature sequence is a preliminary motion trajectory. The motion feature sequence can be understood as a dynamic motion trajectory observed based on the video frame sequence. Compared with the preliminary motion feature sequence, the motion feature sequence is a dynamic change trajectory simulated in a continuous time manner, that is, the motion feature sequence is a more accurate motion trajectory.

[0042] To obtain the dynamic trajectory (motion feature sequence) in continuous time, a Neural Ordinary Differential Equation (Neural ODE) modeler is introduced. This model the physical motion process as a differential process in continuous time, simulating the dynamic trajectory of an object in a continuous-time manner, thus obtaining the motion feature sequence. Based on the motion feature sequence, it is possible to more effectively capture and fit the underlying physical laws in the video.

[0043] Specifically, the neural differential equation modeler includes a state modeling network (learnable neural network) and a differentiable numerical solver. Based on the initial motion feature sequence, the neural differential equation modeler models the continuous temporal motion trajectory between frames to obtain the motion feature sequence. Specifically, this includes: modeling the derivative of the state in the time dimension using the state modeling network; using the initial motion feature sequence as the initial state; and parameterizing the derivative of the hidden state with respect to time as a neural network to obtain the motion feature sequence. The differentiable numerical solver employs a fourth-order Runge-Kutta (RK4) algorithm to ensure the stability and accuracy of the motion feature sequence.

[0044] Step S140: Input the motion feature sequence, the appearance feature sequence, and the user prompt statement into a large language model for cross-modal information fusion and inference to obtain the answer content corresponding to the user prompt statement.

[0045] In this embodiment, the user prompt statement is a statement describing natural semantics. The motion feature sequence, appearance feature sequence, and user prompt statement are input into the large language model for processing, so that the large language model can process two types of modal information, namely physical motion (motion feature sequence) and appearance (appearance feature sequence), to achieve cross-modal information fusion and joint reasoning.

[0046] In an optional embodiment, the motion feature sequence, the appearance feature sequence, and the user prompt statement are input into a large language model for cross-modal information fusion and inference to obtain the answer content corresponding to the user prompt statement, including sub-steps S140-1 to S140-4: Step S140-1: Linearly map and project the motion feature sequence into a motion token sequence, and linearly map and project the appearance feature sequence into an appearance token sequence; Step S140-2: Convert the user prompt statement into a sequence of prompt tokens; Step S140-3: Embed the motion token sequence, the appearance token sequence, and the prompt word token sequence into the natural language prompt template to obtain a token combination sequence; Step S140-4: Input the token combination sequence into the large language model for cross-modal information fusion and reasoning to obtain the answer content corresponding to the user prompt statement.

[0047] In this embodiment of the application, in order to achieve effective fusion between the obtained information (motion feature sequence and appearance feature sequence) and the large language model, a specific feature injection mechanism is adopted to input the motion feature sequence and appearance feature sequence into the large language model.

[0048] Specifically, firstly, in step S140-1, the motion feature sequence and appearance feature sequence are linearly mapped onto tokens acceptable to the large language model. That is, the motion feature sequence and appearance feature sequence are projected into the token embedding space of the large language model, respectively, to obtain the motion token sequence and appearance token sequence. For example, the motion token sequence and appearance token sequence can be represented as:

[0049] in, Represents a sequence of motion tokens. This represents a linear transformation of motion characteristics. Represents a sequence of motion features. Represents the appearance token sequence. This represents a linear transformation of appearance features. This represents a sequence of appearance features.

[0050] Next, in step S140-2, the user prompt statement is converted into a token acceptable to the large language model. Then, in step S140-3, the token sequence is embedded into the natural language prompt template at a specific location to obtain a token combination sequence, which enables the large language model to process the two modal features uniformly based on the token combination sequence.

[0051] Furthermore, the natural language prompt template includes a first embedding position, a second embedding position, and a third embedding position; step S140-3, "embedding the motion token sequence, the appearance token sequence, and the prompt word token sequence into the natural language prompt template to obtain a token combination sequence," specifically includes: embedding the motion token sequence into the first embedding position, embedding the appearance token sequence into the second embedding position, and embedding the prompt word token sequence into the third embedding position to obtain the token combination sequence.

[0052] For example, a natural language prompt template could be "This is an appearance feature". <appearance>This is a characteristic of motion. <motion>"Can you describe this video?" In this template, "motion features" corresponds to the first embedding position, "appearance features" corresponds to the second embedding position, and the remaining natural language descriptions correspond to the third embedding position.

[0053] In practical implementation, to maintain the modal independence and semantic controllability of decoupled features, a slot-based prompt structure is introduced into the natural language prompt template. This is achieved by embedding specific anchor tokens (e.g., ...) into the prompt word token sequence. <motion>and <appearance>), the appearance token sequence and motion token sequence Each token is embedded into a natural language prompt template to obtain a token combination sequence.

[0054] For example, a sequence of token combinations It can be represented as:

[0055] in, , , , , , Both refer to the prompt word "token" in the prompt word sequence. These are the insertion positions for the appearance token sequence and the motion token sequence, respectively.

[0056] Finally, in step S140-4, the token combination sequence is input into the large language model. The token combination sequence guides the large language model to process the two types of modal information, namely motion and appearance, to achieve cross-modal information fusion and joint reasoning.

[0057] The technical solution of this application embodiment obtains the feature sequence corresponding to the video frame sequence and decouples the feature sequence using a dual-branch encoder, thereby decoupling the visual appearance features and physical motion information in the video frame sequence to obtain a structured and interpretable multimodal representation (i.e., appearance feature sequence and preliminary motion feature sequence). Furthermore, a neural differential equation modeler is used to model the continuous-time motion trajectory between frames based on the preliminary motion feature sequence, modeling the physical motion process as a differential process in continuous time to obtain a differentiable physical representation (i.e., motion feature sequence). Finally, the motion feature sequence, the appearance feature sequence, and the user prompt statement are input into a large language model for cross-modal information fusion and inference, enabling the large language model to understand the continuous motion trajectory in the video and achieve accurate understanding of the video by combining visual appearance information.

[0058] In one alternative embodiment, the neural differential equation modeler is trained with the goal of minimizing the error between the predicted motion feature sequence and the actual motion feature sequence.

[0059] In this embodiment of the application, considering the lack of physical annotation data in the training data, a self-supervised mechanism is introduced during the training process. The training objective is to minimize the error (e.g., mean square error) between the predicted motion feature sequence and the real motion feature sequence, so as to enhance the neural differential equation modeler's adherence to physical laws.

[0060] The true motion feature sequence can be the observed motion feature sequence obtained from the sample video sequence, that is, the dynamic change trajectory of the continuous time corresponding to the sample video frame sequence; the predicted motion feature sequence can be the predicted dynamic change trajectory of the future continuous time corresponding to the sample video frame sequence.

[0061] Thus, the neural differential equation modeler trained by this method can simulate the dynamic trajectory of an object in a continuous-time manner, improving the video understanding system's ability to model physical laws. Furthermore, it ensures that the neural differential equation modeler can predict the evolution path of a certain number of consecutive moments in the future, starting from the motion characteristics at any time step, thereby achieving explicit capture and fitting of the potential physical laws in the video.

[0062] In one optional embodiment, the video understanding method based on the decoupling of physical motion and appearance is implemented based on a large video language model, such as... Figure 2 As shown, the video large language model includes at least a dual-branch encoder, a neural differential equation modeler, and a large language model, wherein the dual-branch encoder includes an appearance encoder and a motion encoder.

[0063] The dual-branch encoder comprises an appearance encoder (appearance branch), a motion encoder, and a neural differential equation modeler (motion branch). Together, these branches form the motion-appearance decoupling module. The appearance branch extracts static appearance features, while the motion branch extracts temporally continuous motion features. Furthermore, the video large language model includes a token mapping and integration module. This module projects the motion and appearance feature sequences output from the motion-appearance decoupling module into the large language model's token space and fuses them using a lightweight LoRA fine-tuning mechanism. This allows the model to maintain its original multimodal capabilities while also possessing the ability to model physical dynamics.

[0064] Specifically, the video large language model is trained according to the following steps S210 to S270; Step S210: Obtain sample data, which includes: sample video frame sequences, sample user prompts, and actual response content.

[0065] Specifically, in order to achieve accurate training of the video large language model, a large-scale multi-task instruction fine-tuning dataset (i.e., training dataset) is constructed. The sample data in the training dataset covers tasks such as video dialogue, video description, and video question answering. The data sources include VideoChat, Video-ChatGPT, Ego4D, YouCook2, ActivityNet-QA, and NextQA.

[0066] Meanwhile, in order to effectively evaluate the physical reasoning ability of the video large language model, the training dataset was also constructed based on the synthetic dataset PhyBench of the physical simulation platform. The sample data in the training dataset covers multiple classic physical scenarios such as acceleration, deceleration, rebound, uniform speed, and parabolic motion.

[0067] Step S220: Extract features from each video frame of the sample video frame sequence to obtain a sample feature sequence.

[0068] Specifically, through a shared visual backbone network (e.g., Feature extraction is performed on each video frame of the sample video frame sequence to obtain the sample feature sequence, so as to determine the consistency of the underlying visual features and the efficiency of parameter sharing.

[0069] Step S230: Decouple the sample feature sequence using a dual-branch encoder to obtain a sample appearance feature sequence and a sample preliminary motion feature sequence, wherein the sample preliminary motion feature sequence characterizes the motion features of each video frame in the sample video frame sequence; Specifically, the appearance encoder of the dual-branch encoder is used to extract appearance features from the sample feature sequence to obtain the sample appearance feature sequence. In addition, the motion encoder of the dual-branch encoder is used to extract motion features from the sample feature sequence to obtain the sample preliminary motion feature sequence. Thus, the visual appearance features and physical motion information in the sample video frame sequence are decoupled based on the dual-branch encoder.

[0070] The sample appearance feature sequence includes multiple appearance features arranged in temporal order, with each appearance feature corresponding to the static appearance information of a video frame. Each feature of the sample feature sequence is sequentially input into the appearance encoder, which extracts appearance features from each feature to obtain the sample appearance feature sequence. The sample appearance features include texture information, shape information, and structural information in the scene.

[0071] The preliminary motion feature sequence of the samples includes multiple preliminary motion features arranged in temporal order, that is, it includes the motion features (physical motion information) corresponding to each sample video frame. Therefore, the preliminary motion feature sequence of the samples represents the motion features of each video frame in the sample video frame sequence, that is, the preliminary motion feature sequence of the samples corresponds to a preliminary motion trajectory. The sample feature sequence is input into the motion encoder, which, based on the temporal self-attention mechanism of the Transformer, extracts temporal information from the features of each frame in the sample feature sequence to obtain the preliminary motion feature sequence of the samples.

[0072] Step S240: Using the neural differential equation modeler, based on the preliminary motion feature sequence of the sample, perform continuous time motion trajectory modeling between frames to obtain the observed motion feature sequence and the predicted motion feature sequence. The observed motion feature sequence represents the dynamic change trajectory of the continuous time corresponding to the sample video frame sequence, and the predicted motion feature sequence represents the predicted dynamic change trajectory of the future continuous time corresponding to the sample video frame sequence.

[0073] The observed preliminary motion feature sequence includes the motion features corresponding to each video frame, and the sample preliminary motion feature sequence is a preliminary motion trajectory. The observed motion feature sequence can be understood as the dynamic motion trajectory observed from the sample video frame sequence based on the current network parameters. Compared with the sample preliminary motion feature sequence, the observed motion feature sequence is a dynamic change trajectory simulated in a continuous time manner, that is, the observed motion feature sequence is a more accurate motion trajectory.

[0074] Specifically, the neural differential equation modeler includes a state modeling network and a differentiable numerical solver; step S240 specifically includes sub-steps S240-1 to S240-2: Step S250-1: Model the derivative of the state in the time dimension using the state modeling network. Take the initial motion feature sequence of the sample as the initial state and parameterize the derivative of the hidden state with respect to time as the neural network to obtain the observed motion feature sequence.

[0075] Step S250-2: Model the derivative of the state in the time dimension using a state modeling network, take the observed motion feature sequence as the initial state, and extrapolate it using a differentiable numerical solver to obtain the predicted motion feature sequence.

[0076] In this embodiment, the state modeling network is a learnable neural network that simulates the motion features of preliminary samples in the time dimension based on the current network parameter state. Using the preliminary sample motion features at each time step as the initial state, and parameterizing the derivative of the hidden state with respect to time as the neural network parameter, the observed motion feature sequence can be obtained. Furthermore, based on the observed motion feature sequence, the predicted motion feature sequence for future time steps can be calculated through extrapolation by a neural differential equation modeler. For example, the predicted motion features in the predicted motion feature sequence... It can be represented as:

[0077] in, This indicates a differentiable numerical solver, which employs a fourth-order Runge-Kutta (RK4) algorithm to ensure the stability and accuracy of trajectory prediction. Represents a state-modeling network. Indicates a future moment. This represents the motion characteristics at time t in the observed motion characteristic sequence.

[0078] Step S250: Input the observed motion feature sequence, the sample appearance feature sequence, and the sample user prompt statement into the large language model for cross-modal information fusion and inference to obtain the sample answer content corresponding to the sample user prompt statement.

[0079] Specifically, the observed motion feature sequence can be linearly mapped and projected into a sample motion token sequence through the Token mapping and integration module, and the sample appearance feature sequence can be linearly mapped and projected into a sample appearance token sequence; the sample user prompt statement can be converted into a sample prompt word token sequence; then the sample motion token sequence, sample appearance token sequence, and sample prompt word token sequence are embedded into the natural language prompt template to obtain a sample token combination sequence; finally, the sample token combination sequence is input into the large language model for cross-modal information fusion and inference to obtain the sample answer content corresponding to the sample user prompt statement.

[0080] Step S260: Calculate the total loss value based on the observed motion feature sequence, the predicted motion feature sequence, the sample appearance feature sequence, the sample answer content, and the real answer content, and update the model parameters of the video big language model based on the total loss value.

[0081] In this embodiment of the application, in order to endow the video large model with physical modeling capabilities, enabling it to understand continuous motion trajectories in the video, predict physical behavior results, and perform multimodal semantic reasoning, the training objectives include three aspects, namely, the total loss value includes physical consistency loss value, mutual information loss value, and supervised learning loss value.

[0082] Specifically, step S260 may include sub-steps S260-1 to S260-4: Step S260-1: Calculate the physical consistency loss value based on the observed motion feature sequence and the predicted motion feature sequence.

[0083] In this embodiment, considering the lack of physically labeled data in the training data, a self-supervised mechanism is introduced during training. Training is achieved by minimizing the error (e.g., mean squared error) between the predicted motion feature sequence and the observed motion feature sequence, thereby enhancing the neural differential equation modeler's adherence to physical laws. For example, the physical consistency loss value... It can be represented as:

[0084] in, Represents the future at time t. Predicted motion characteristics at any given time Represents the future at time t. The true value of the motion feature at time t is the motion feature corresponding to time t in the observed motion feature sequence.

[0085] The physical consistency loss value enables the neural differential equation modeler to simulate the dynamic trajectory of an object in a continuous time manner, improving the video understanding system's ability to model physical laws. Furthermore, it ensures that the neural differential equation modeler can predict the evolution path of a certain number of consecutive moments in the future, starting from the motion characteristics at any time step, thus achieving explicit capture and fitting of the potential physical laws in the video.

[0086] Step S260-2: Calculate the mutual information loss value based on the observed motion feature sequence and the sample appearance feature sequence.

[0087] The mutual information loss value can be understood as the mutual information minimization loss based on the Hilbert-Schmidt independence criterion (HSIC), which is used to enhance the feature separation between motion features and appearance features. Since physical information and appearance information are encoded using two different encoders (appearance encoder and motion encoder), in order to ensure that physical information and appearance information are decoupled, the similarity information between the observed motion feature sequence and the sample appearance feature sequence must be minimized.

[0088] Step S260-3: Calculate the supervised learning loss value based on the sample response content and the real response content.

[0089] Specifically, to enhance the model's language understanding ability and achieve cross-modal alignment, some parameters of the language model are fine-tuned based on the LoRA mechanism, and the supervised learning loss is calculated based on the sample answer content and the real answer content, using the standard supervised learning loss as the training objective.

[0090] Step S260-4: Obtain the total loss value based on the physical consistency loss value, the mutual information loss value, and the supervised learning loss value.

[0091] In this embodiment, the three loss values ​​mentioned above are combined to construct a total loss value (i.e., a comprehensive training objective), enabling the model to be optimized simultaneously in terms of physical modeling, representation decoupling, and language alignment. Thus, by training the model using this method, the video large language model is endowed with physical modeling capabilities, enabling it to understand continuous motion trajectories in videos, predict physical behavior outcomes, and perform multimodal semantic reasoning.

[0092] Furthermore, the video large language model in this embodiment emphasizes the rationality and synergy of the decoupling strategy. The appearance encoder and motion encoder of the dual-branch encoder adopt different modeling paths in terms of structure and function. The appearance encoder emphasizes spatial stability, while the motion encoder emphasizes temporal dynamics. In terms of supervision signals, the motion branch is subject to the forced trajectory constraint of the neural differential equation modeler, while the appearance branch is subject to the regularization control of the information minimization objective, thereby suppressing dynamic interference. This design of structural decoupling and asymmetric supervision effectively improves the interpretability and modularity of the video large language model, enabling each sub-module to be independently adapted or combined for application in different task scenarios.

[0093] For example, Figure 3 This is a schematic diagram illustrating an application scenario of a video big language model provided in this application embodiment. This video big language model can realize the video understanding method based on the decoupling of physical motion and appearance described in the above embodiment. For example, for the user prompt "What type of physical motion is the object in the video doing? A. Uniform linear motion; B. Parabolic motion; C. Rebound; D. Accelerating linear motion; E. Decelerating linear motion", the video big language model outputs the answer "D. Accelerating linear motion". For the user prompt "Why do the driver and passengers lean forward in the car?", the video big language model outputs the answer "Because the car is braking and decelerating, and the passengers' bodies continue to move forward due to inertia".

[0094] The technical solution of this application embodiment endows the video large language model with physical modeling capabilities, thereby achieving video understanding and reasoning with physical consistency, causal rationality, and motion interpretability. Compared with existing video large language models, the video large language model of this application embodiment has the following advantages: First, most existing video language models treat motion information as part of visual changes and lack the ability to explicitly model the actual physical motion trajectory of objects. However, the technical solution of this application introduces the decoupling of motion encoder and appearance encoder, as well as a neural differential equation modeler, to effectively separate and model the physical motion process in continuous time, thereby achieving a structured understanding of video dynamics.

[0095] Secondly, existing video language models mostly rely on large-scale language models to perform coarse-grained fusion of video features, which cannot make explicit inferences about physical laws and dynamic trajectories. However, the technical solution of this application embodiment enables the model to continuously simulate and predict dynamic behaviors such as acceleration and velocity changes through a self-supervised physical trajectory learning mechanism.

[0096] Finally, the technical solution of this application embodiment achieves efficient docking between the external physical modeling module and the language model while keeping the large language model structure frozen. It has excellent cross-modal compatibility and low-cost adaptation capability, which is significantly better than the existing methods that require large-scale training or modification of the model architecture.

[0097] This application also provides a video understanding device based on the decoupling of physical motion and appearance, referring to... Figure 4 As shown, Figure 4 This is a schematic diagram of a video understanding device based on the decoupling of physical motion and appearance provided in an embodiment of this application. The device includes: The extraction module 410 is used to extract features from each video frame of the video frame sequence to obtain a feature sequence; The decoupling module 420 uses a dual-branch encoder to decouple the feature sequence to obtain an appearance feature sequence and a preliminary motion feature sequence. The preliminary motion feature sequence characterizes the motion features of each video frame in the video frame sequence. The dual-branch encoder includes an appearance encoder and a motion encoder. Modeling module 430 is used to use a neural differential equation modeler to model the continuous time motion trajectory between frames based on the preliminary motion feature sequence, so as to obtain a motion feature sequence, wherein the motion feature sequence characterizes the dynamic change trajectory of the continuous time corresponding to the video frame sequence. The reasoning module 440 is used to input the motion feature sequence, the appearance feature sequence, and the user prompt statement into a large language model for cross-modal information fusion and reasoning to obtain the answer content corresponding to the user prompt statement.

[0098] It is understood that the video understanding device based on physical motion and appearance decoupling in the embodiments of this application can realize the video understanding method based on physical motion and appearance decoupling in the above embodiments. The video understanding device based on physical motion and appearance decoupling and the video understanding method based on physical motion and appearance decoupling have the same advantages over the prior art, and will not be repeated here.

[0099] This application also provides an electronic device, see embodiments thereof. Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 5 As shown, the electronic device 500 includes a memory 510 and a processor 520. The memory 510 and the processor 520 are connected via a bus. The memory 510 stores a computer program that can run on the processor 520 to implement the steps of the video understanding method based on physical motion and appearance decoupling described in the embodiments of this application.

[0100] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the video understanding method based on physical motion and appearance decoupling described in this application.

[0101] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video understanding method based on physical motion and appearance decoupling described in this application.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0103] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0104] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0106] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0107] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0108] The above provides a detailed description of a video understanding method, apparatus, and device based on the decoupling of physical motion and appearance provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / appearance> < / motion> < / motion> < / appearance>

Claims

1. A video understanding method based on the decoupling of physical motion and appearance, characterized in that, include: Feature extraction is performed on each video frame in the video frame sequence to obtain a feature sequence; The feature sequence is decoupled using a dual-branch encoder to obtain an appearance feature sequence and a preliminary motion feature sequence. The preliminary motion feature sequence characterizes the motion features of each video frame in the video frame sequence. The dual-branch encoder includes an appearance encoder and a motion encoder. Using a neural differential equation modeler, based on the preliminary motion feature sequence, continuous-time motion trajectory modeling is performed between frames to obtain a motion feature sequence, which characterizes the dynamic change trajectory of the continuous time corresponding to the video frame sequence. The motion feature sequence, the appearance feature sequence, and the user prompt statement are input into a large language model for cross-modal information fusion and inference to obtain the answer content corresponding to the user prompt statement.

2. The method according to claim 1, characterized in that, The motion feature sequence, the appearance feature sequence, and the user prompt statement are input into a large language model for cross-modal information fusion and inference to obtain the response content corresponding to the user prompt statement, including: The motion feature sequence is linearly mapped and projected into a motion token sequence, and the appearance feature sequence is linearly mapped and projected into an appearance token sequence; Convert the user prompt statement into a sequence of prompt tokens; The motion token sequence, the appearance token sequence, and the prompt word token sequence are embedded into a natural language prompt template to obtain a token combination sequence; The token combination sequence is input into the large language model for cross-modal information fusion and inference to obtain the answer content corresponding to the user prompt statement.

3. The method according to claim 2, characterized in that, The natural language prompt template includes a first embedding position, a second embedding position, and a third embedding position; the motion token sequence, the appearance token sequence, and the prompt word token sequence are embedded into the natural language prompt template to obtain a token combination sequence, including: The motion token sequence is embedded in the first embedding position, the appearance token sequence is embedded in the second embedding position, and the prompt word token sequence is embedded in the third embedding position to obtain the token combination sequence.

4. The method according to claim 1, characterized in that, The feature sequence is decoupled using a dual-branch encoder to obtain an appearance feature sequence and a preliminary motion feature sequence, including: The feature sequence is sequentially input into the appearance encoder of the dual-branch encoder, and the appearance encoder extracts the appearance features to obtain the appearance feature sequence; wherein, the appearance features include texture information, shape information and structural information in the scene; The feature sequence is input into the motion encoder of the dual-branch encoder. The motion encoder, based on the temporal self-attention mechanism of Transformer, extracts temporal information from the features of each frame of the feature sequence to obtain the preliminary motion feature sequence.

5. The method according to any one of claims 1-4, characterized in that, The neural differential equation modeler is trained with the goal of minimizing the error between the predicted motion feature sequence and the actual motion feature sequence.

6. The method according to claim 5, characterized in that, The video understanding method based on the decoupling of physical motion and appearance is implemented based on a large video language model. The large video language model includes at least an appearance encoder, a motion encoder, a neural differential equation modeler, and a large language model. The large video language model is trained according to the following steps. Acquire sample data, which includes: sample video frame sequences, sample user prompts, and actual response content; Feature extraction is performed on each video frame of the sample video frame sequence to obtain the sample feature sequence; The sample feature sequence is decoupled using a dual-branch encoder to obtain a sample appearance feature sequence and a sample preliminary motion feature sequence. The sample preliminary motion feature sequence characterizes the motion features of each video frame in the sample video frame sequence. The neural differential equation modeler is used to model the continuous time motion trajectory between frames based on the preliminary motion feature sequence of the sample, to obtain the observed motion feature sequence and the predicted motion feature sequence. The observed motion feature sequence represents the dynamic change trajectory of the continuous time corresponding to the sample video frame sequence, and the predicted motion feature sequence represents the predicted dynamic change trajectory of the future continuous time corresponding to the sample video frame sequence. The observed motion feature sequence, the sample appearance feature sequence, and the sample user prompt statement are input into the large language model for cross-modal information fusion and inference to obtain the sample answer content corresponding to the sample user prompt statement; Based on the observed motion feature sequence, the predicted motion feature sequence, the sample appearance feature sequence, the sample answer content, and the actual answer content, the total loss value is calculated, and the model parameters of the video big language model are updated based on the total loss value.

7. The method according to claim 6, characterized in that, The neural differential equation modeler includes a state modeling network and a differentiable numerical solver; using the neural differential equation modeler, based on the preliminary motion feature sequence of the sample, continuous-time motion trajectory modeling is performed between frames to obtain the observed motion feature sequence and the predicted motion feature sequence, including: The state modeling network is used to model the derivative of the state in the time dimension. The initial motion feature sequence of the sample is used as the initial state, and the derivative of the hidden state with respect to time is used as the neural network parameter to obtain the observed motion feature sequence. The derivative of the state in the time dimension is modeled by a state modeling network. The observed motion feature sequence is used as the initial state, and the predicted motion feature sequence is obtained by extrapolation through a differentiable numerical solver.

8. The method according to claim 6, characterized in that, Based on the observed motion feature sequence, the predicted motion feature sequence, the sample appearance feature sequence, the sample response content, and the actual response content, the total loss value is calculated, including: Calculate the physical consistency loss value based on the observed motion feature sequence and the predicted motion feature sequence; Calculate the mutual information loss value based on the observed motion feature sequence and the sample appearance feature sequence; Calculate the supervised learning loss value based on the sample answers and the actual answers; The total loss value is obtained based on the physical consistency loss value, the mutual information loss value, and the supervised learning loss value.

9. A video understanding device based on the decoupling of physical motion and appearance, characterized in that, include: The extraction module is used to extract features from each video frame in the video frame sequence to obtain a feature sequence; The decoupling module uses a dual-branch encoder to decouple the feature sequence to obtain an appearance feature sequence and a preliminary motion feature sequence. The preliminary motion feature sequence characterizes the motion features of each video frame in the video frame sequence. The dual-branch encoder includes an appearance encoder and a motion encoder. The modeling module is used to model the continuous-time motion trajectory between frames based on the preliminary motion feature sequence using a neural differential equation modeler, thereby obtaining a motion feature sequence, which characterizes the dynamic change trajectory of the continuous time corresponding to the video frame sequence. The reasoning module is used to input the motion feature sequence, the appearance feature sequence, and the user prompt statement into a large language model for cross-modal information fusion and reasoning to obtain the answer content corresponding to the user prompt statement.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the video understanding method based on the decoupling of physical motion and appearance as described in any one of claims 1-8.

Images